docs: validate baseline and add deterministic regression coverage
Session 3 baseline validation (no beta/support-weighted code added). - Two measured full-data original_protean conditions (X-IIoTID and 5G-NIDD, alpha=0.5, seed=0); frozen manifests reused and reload-verified. - Partition consistency, per-class/per-client inspection, round-0, loss audit, inference validation, reproducibility and a deterministic end-to-end test. - S_j participant rule remains an explicitly labelled REPRODUCTION ASSUMPTION; |S_j| < M quantified in 53/162 class-condition cells (32.7%), 15/18 conditions. - Adds docs/baseline/BASELINE_VALIDATION.md, tests/test_baseline_validation.py, scripts/validate_sj_participant_rule.py and small analysis artifacts. - Test suite: 46 passed. planningawal/ and raw datasets unchanged.
This commit is contained in:
willtanoe committed
1 parent
374651a0fe
commit
16b3add2cf
9 files changed
+24265
-200
No files matched your search
@@ -8,6 +8,7 @@ sweep was launched.
|
||||
|
||||
Companion documents:
|
||||
|
||||
- `docs/baseline/BASELINE_VALIDATION.md` — Session-3 validation and reproduction evidence.
|
||||
- `docs/baseline/PROTEAN_BASELINE_INVESTIGATION.md` — evidence for baseline semantics.
|
||||
- `docs/research/EXPERIMENT_PROTOCOL.md` — protocol and provenance tags.
|
||||
- `docs/research/IMPLEMENTATION_PLAN.md` — architecture.
|
||||
@@ -169,13 +170,19 @@ and hash, so manifests stay small.
|
||||
|
||||
## 11. Tests
|
||||
|
||||
`python -m pytest -q` -> **39 tests** (see run result in the handoff). Coverage includes
|
||||
`python -m pytest -q` -> **46 tests** (Session-3 count; the Session-2 handoff reported 34
|
||||
and an earlier draft of this file said 39, which was inaccurate). Coverage includes
|
||||
dataset config resolution, deterministic label mapping, deterministic/train-only
|
||||
preprocessing, deterministic stratified split, deterministic Dirichlet partition,
|
||||
manifest reload identity and corruption detection, rare/unseen semantics, local prototype
|
||||
means, baseline aggregation (including that `support_weighted` is not implemented), metrics
|
||||
on known fixtures, and a small end-to-end federated smoke test.
|
||||
|
||||
`tests/test_baseline_validation.py` (Session 3) adds round-0 handling, an independent
|
||||
replication of the documented loss formula, frozen-coefficient guards, equal-weight model
|
||||
averaging, inference/class-mapping checks, and a deterministic end-to-end regression test.
|
||||
See `docs/baseline/BASELINE_VALIDATION.md`.
|
||||
|
||||
## 12. Smoke test
|
||||
|
||||
```
|
||||
@@ -201,10 +208,18 @@ manifests -> `artifacts/partitions/`.
|
||||
## 14. Remaining assumptions and unresolved issues
|
||||
|
||||
1. **Participant rule** (`S_j`, equal weights) is a reproduction assumption (see §7).
|
||||
Session-3 validation quantified it: `|S_j| < M` in 53/162 class-condition cells
|
||||
(32.7 %) across 15/18 conditions, and the hypothetical zero-filled `1/M` reading is
|
||||
displaced by ~10–20 % of the prototype norm for the affected classes
|
||||
(`docs/baseline/BASELINE_VALIDATION.md` §D). It remains unresolved, not disproven.
|
||||
2. `lambda = 1.0`, optimiser and schedule are unverified against the paper.
|
||||
3. Prototype-alignment form and round-0 handling are inferred.
|
||||
4. Per-client evaluation partitioning (val/test assigned to clients via Dirichlet) is a
|
||||
project decision; the paper does not describe it.
|
||||
5. No full-scale run has been timed; compute feasibility of a real sweep is unmeasured.
|
||||
6. Whether the literal `1/M` reading differs from the `S_j` assumption is open and must
|
||||
be tested with a deliberate control in the verification phase.
|
||||
5. Full-scale compute is now measured: X-IIoTID (`alpha=0.5`, seed 0) 42:07 wall /
|
||||
6.21 GiB peak, 5G-NIDD 1:04:52 wall / 10.37 GiB peak, CPU-only (see
|
||||
`docs/baseline/BASELINE_VALIDATION.md` §B).
|
||||
6. Whether the literal `1/M` reading differs from the `S_j` assumption is open. It cannot
|
||||
be settled without original code (none exists); the only feasible treatments are
|
||||
documented interpretation choices, and the zero-fill variant was analysed as a
|
||||
hypothetical artifact only.
|
||||
@@ -0,0 +1,430 @@
|
||||
# Baseline Validation
|
||||
|
||||
Status: baseline validation and reproduction verification (Session 3, 2026-09-24).
|
||||
Scope: determine whether the frozen `original_protean` baseline behaves coherently,
|
||||
reproducibly and consistently with the best available PROTEAN evidence, and identify
|
||||
what is still a reproduction assumption. **No beta / support-weighted / `equal_available`
|
||||
functionality was added.**
|
||||
|
||||
Companion documents:
|
||||
|
||||
- `docs/baseline/BASELINE_IMPLEMENTATION.md` — what the baseline implements.
|
||||
- `docs/baseline/PROTEAN_BASELINE_INVESTIGATION.md` — paper/thesis evidence.
|
||||
- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen protocol.
|
||||
- `docs/handoff/SESSION_HANDOFF.md` — continuation point.
|
||||
|
||||
---
|
||||
|
||||
## A. Validation status
|
||||
|
||||
Three different questions must not be conflated:
|
||||
|
||||
1. **Implementation correctness** (does the code do what the frozen protocol says,
|
||||
deterministically and without leakage?). **Validated** for the checks below.
|
||||
2. **Reproduction evidence** (does the code reproduce original PROTEAN?). **Not
|
||||
established**, because no official PROTEAN implementation exists
|
||||
(`PROTEAN_BASELINE_INVESTIGATION.md` §2) and the paper is internally inconsistent
|
||||
about the prototype participant rule.
|
||||
3. **Unresolved assumptions** (what remains a documented reproduction assumption?).
|
||||
Listed in §G; the central one is the `S_j` participant/denominator rule (§D).
|
||||
|
||||
| Validation item | Result | Evidence |
|
||||
|---|---|---|
|
||||
| Full-data run completes | PASS | §B |
|
||||
| Partition consistency | PASS | §B, §D |
|
||||
| Global / per-client / per-class inspection | PASS (reviewed) | §C |
|
||||
| `S_j` ambiguity quantified | PASS (quantified, not resolved) | §D |
|
||||
| Round-0 + prototype-alignment entry | PASS | §E |
|
||||
| Loss implementation audit | PASS | §E |
|
||||
| Inference validation | PASS | §E |
|
||||
| Deterministic end-to-end regression test | PASS | §F |
|
||||
| Repeat-run reproducibility | PASS | §F |
|
||||
| Total test suite | PASS (46 tests) | §F |
|
||||
|
||||
---
|
||||
|
||||
## B. Full-data run
|
||||
|
||||
### B.1 Measured condition (primary)
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| dataset | `xiiotid` (X-IIoTID) |
|
||||
| alpha | 0.50 |
|
||||
| seed | 0 |
|
||||
| aggregation | `original_protean` |
|
||||
| num_clients | 10 |
|
||||
| rounds / local epochs / batch / lr / mu / lambda | 10 / 3 / 256 / 1e-3 / 0.1 / 1.0 |
|
||||
| device | CPU (`torch 2.14.0+cpu`, CUDA unavailable) |
|
||||
| run dir | `results/xiiotid__alpha0.5__seed0__original_protean/` |
|
||||
| manifest | `artifacts/partitions/xiiotid__alpha0.5__seed0.json` (reused, not regenerated) |
|
||||
| manifest content hash | `e81a6538ecefcda202aa7602310ff8d8c9529031ab9dcbec4ebbabc0e63b636c` |
|
||||
| preprocessing hash | `8580cee41c698ce43cf3269e4a2b3a01c92d6df22820f4004c0267478bd2579a` |
|
||||
| snapshot | `artifacts/analysis/xiiotid__alpha0.5__seed0__original_protean__prototype_snapshot.json` |
|
||||
|
||||
### B.2 Time and resource measurements (`/usr/bin/time -v`)
|
||||
|
||||
| Measure | Value |
|
||||
|---|---|
|
||||
| Wall-clock (incl. load + preprocessing + 10 rounds + eval) | **42:06.78** |
|
||||
| User CPU time | 18231.44 s |
|
||||
| System CPU time | 481.31 s |
|
||||
| CPU utilisation | 740 % |
|
||||
| Peak resident set size (host) | **6,509,448 KB ≈ 6.21 GiB** |
|
||||
| GPU memory | n/a (CPU-only) |
|
||||
| Swaps | 0 |
|
||||
| Exit status | 0 |
|
||||
|
||||
Separate component timings were not instrumented inside the runner. A calibration on a
|
||||
50,000-row subsample measured `raw read ≈ 0.9 s`, `prepare_dataset ≈ 1.3 s` and
|
||||
`≈ 0.163 ms` per sample-epoch (input_dim = 78 there), i.e. preprocessing is negligible
|
||||
next to training. No retries or client failures occurred; all 10 clients remained active
|
||||
in all 10 rounds (the smallest client, id 4, still had 18,521 training samples).
|
||||
|
||||
### B.3 Global test metrics
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| accuracy | 0.8904 |
|
||||
| macro-F1 | 0.7077 |
|
||||
| weighted-F1 | 0.9176 |
|
||||
| rare-class macro-F1 | 0.6096 |
|
||||
| rare-class recall | 0.8715 |
|
||||
| rare-class evaluation support | 25,454 |
|
||||
|
||||
Validation trajectory (history): accuracy rises to ~0.88 by round 3 and then plateaus
|
||||
(0.881–0.889); macro-F1 plateaus ~0.70; rare-class macro-F1 plateaus ~0.59–0.62. The
|
||||
curve is stable with mild round-to-round fluctuation, consistent with a converged (not
|
||||
diverging or collapsed) baseline.
|
||||
|
||||
### B.4 Second-dataset sanity run (5G-NIDD)
|
||||
|
||||
The optional second full-data condition was run to confirm the full-data path works on
|
||||
both datasets. Same frozen settings, `original_protean`, 10 clients / 10 rounds / 3 local
|
||||
epochs.
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| dataset | `5gnidd` (5G-NIDD) |
|
||||
| alpha / seed | 0.50 / 0 |
|
||||
| run dir | `results/5gnidd__alpha0.5__seed0__original_protean/` |
|
||||
| manifest content hash | `0732582cf4510089212717816d2151f5706c7b361123be32cc353bdf4b7f78e6` (reused, reload-verified) |
|
||||
| preprocessing hash | `ff8bc0e52967a2a081392282ae62ec86bc3f2b88648a9fa20de43fb0e7e2cc3a` |
|
||||
| wall-clock | **1:04:52** |
|
||||
| CPU / RSS | 739 % / **10,875,780 KB ≈ 10.37 GiB** |
|
||||
| accuracy | 0.7344 |
|
||||
| macro-F1 | 0.8369 |
|
||||
| weighted-F1 | 0.7088 |
|
||||
| rare-class macro-F1 | 0.8720 |
|
||||
| rare-class recall | 0.9743 |
|
||||
| exit status | 0 (10/10 rounds, no failures) |
|
||||
|
||||
Notable per-class behaviour: `Benign` (support 95,547) has precision 1.000 but recall
|
||||
only 0.380, while `UDPFlood` (support 91,468) has recall 1.000 and precision 0.616 — i.e.
|
||||
the nearest-prototype classifier over-predicts `UDPFlood` for many benign flows. This is
|
||||
why macro-F1 (0.837) exceeds weighted-F1 (0.709). As with X-IIoTID this is plausible
|
||||
nearest-prototype behaviour under non-IID data, not an obvious anomaly; 9/9 prototypes
|
||||
were present and no class had zero predictions. Per-client accuracy ranges 0.537–0.991.
|
||||
|
||||
Both full-data conditions therefore complete the frozen path end to end, reuse their
|
||||
frozen manifests, and produce finite, structurally explainable metrics.
|
||||
|
||||
---
|
||||
|
||||
## C. Per-client / per-class inspection
|
||||
|
||||
### C.1 Per-class test behaviour (X-IIoTID α=0.5 seed 0)
|
||||
|
||||
| class | support | precision | recall | F1 | note |
|
||||
|---|---|---|---|---|---|
|
||||
| C&C | 573 | 0.092 | 0.990 | 0.169 | very low precision; over-predicted |
|
||||
| Exfiltration | 4,427 | 1.000 | 0.952 | 0.975 | strong |
|
||||
| Exploitation | 227 | 0.019 | 0.581 | 0.037 | very low precision; over-predicted |
|
||||
| Lateral_movement | 6,319 | 0.964 | 0.727 | 0.829 | good precision, moderate recall |
|
||||
| Normal | 83,651 | 0.971 | 0.928 | 0.949 | strong |
|
||||
| RDOS | 28,252 | 0.992 | 0.997 | 0.995 | strong |
|
||||
| Reconnaissance | 25,474 | 0.992 | 0.627 | 0.768 | low recall |
|
||||
| Tampering | 1,024 | 0.563 | 0.992 | 0.718 | recall high, precision moderate |
|
||||
| Weaponization | 12,884 | 0.879 | 0.987 | 0.930 | strong |
|
||||
|
||||
The confusion matrix shows the dominant errors are structural, not random: 5,181 true
|
||||
`Normal` samples are predicted `C&C` (which is why `C&C` precision collapses), and 5,683
|
||||
true `Reconnaissance` samples are predicted `Exploitation`. The lowest-support classes
|
||||
(`C&C` n=573, `Exploitation` n=227) are the ones whose nearest-prototype precision
|
||||
collapses toward the large `Normal`/`Reconnaissance` prototypes. This is **plausible
|
||||
non-IID / nearest-prototype difficulty with scarce support**, not an obvious
|
||||
implementation anomaly: recalls are high, predictions are finite, no class has zero
|
||||
predictions, and no class is missing from the global prototype set (9/9 present).
|
||||
|
||||
Classification of the low rare-class F1: *insufficient support + prototype separation
|
||||
difficulty under non-IID data*, not a bug. It is **not** interpreted as improved or
|
||||
degraded method behaviour (no method comparison has been run).
|
||||
|
||||
### C.2 Per-client behaviour (X-IIoTID α=0.5 seed 0)
|
||||
|
||||
Per-client test metrics vary with each client's local eval partition:
|
||||
|
||||
- accuracy 0.691–0.986; macro-F1 0.552–0.840.
|
||||
- clients 4 (acc 0.691, rare macro-F1 0.135) and 7 (rare macro-F1 0.045) are the weakest.
|
||||
- clients 2 (rare macro-F1 0.876) and 5 (0.698) are the strongest.
|
||||
- 10/10 clients evaluated; no empty client eval set.
|
||||
|
||||
Per-client training structure (from the manifest):
|
||||
|
||||
| client | train samples | observed classes | unseen (attack) classes | rare classes (2 smallest positive attack support) |
|
||||
|---|---|---|---|---|
|
||||
| 0 | 94,564 | 7 | 2 | Weaponization, C&C |
|
||||
| 1 | 50,631 | 9 | 0 | Exploitation, Lateral_movement |
|
||||
| 2 | 52,476 | 9 | 0 | Tampering, C&C |
|
||||
| 3 | 69,609 | 9 | 0 | Exploitation, Lateral_movement |
|
||||
| 4 | 18,521 | 9 | 0 | C&C, Exploitation |
|
||||
| 5 | 41,886 | 9 | 0 | Tampering, C&C |
|
||||
| 6 | 45,085 | 7 | 2 | C&C, Tampering |
|
||||
| 7 | 84,396 | 9 | 0 | Exploitation, C&C |
|
||||
| 8 | 85,073 | 9 | 0 | C&C, Exfiltration |
|
||||
| 9 | 27,668 | 9 | 0 | Exploitation, Lateral_movement |
|
||||
|
||||
Only clients 0 and 6 are missing attack classes (`Exploitation` and `Tampering`/`Exfiltration`),
|
||||
i.e. under this middle-alpha condition most clients see all classes, but two do not.
|
||||
|
||||
---
|
||||
|
||||
## D. `S_j` participant-rule ambiguity analysis
|
||||
|
||||
### D.1 The question
|
||||
|
||||
Paper/thesis Eq. (2)/(5.5) writes `Cbar_j = (1/M) * sum_i C_ij` over all `M` clients,
|
||||
while the thesis also states prototypes are shared only for locally observed classes.
|
||||
The paper never defines how `C_ij` is treated when `N_ij = 0`. The baseline therefore
|
||||
assumes:
|
||||
|
||||
```
|
||||
S_j = { i | N_ij > 0 and a valid C_ij exists }
|
||||
Cbar_j = mean_{i in S_j} C_ij (equal 1/|S_j| weights)
|
||||
```
|
||||
|
||||
This remains a **REPRODUCTION ASSUMPTION**, not verified original behaviour.
|
||||
|
||||
### D.2 How often does `|S_j| < M` actually occur?
|
||||
|
||||
Across all 18 frozen manifests (`artifacts/analysis/sj_participant_rule_analysis.csv`):
|
||||
|
||||
- **15 of 18** `(dataset, alpha, seed)` conditions contain at least one class with `|S_j| < M`.
|
||||
- **53 of 162** class-condition cells (32.7 %) have `|S_j| < M`.
|
||||
- Affected classes are not only exotic ones: `Exploitation` (6 conditions), `C&C` (6),
|
||||
`ICMPFlood` (5), `SYNFlood` (5), `TCPConnectScan` (4), `Tampering` (4), and even the
|
||||
**benign** classes `Normal` (2) and `Benign` (implicit where listed among the 5G classes).
|
||||
- The effect is largest at `alpha = 0.25` (up to 7/9 classes missing a client) and can be
|
||||
zero at `alpha = 0.75` for some seeds.
|
||||
|
||||
Conclusion: the ambiguity is **material, not rare**. A literal `1/M` reading differs from
|
||||
the `S_j` reading for a large fraction of class-condition combinations.
|
||||
|
||||
### D.3 Direct prototype displacement (fixed `C_ij`, no retraining)
|
||||
|
||||
From the round-0 snapshot (fixed `C_ij`, `N_ij`) for the measured condition, comparing the
|
||||
current `S_j` mean against one hypothetical literal reading (**missing prototype replaced
|
||||
by the zero vector**, then divided by `M`):
|
||||
|
||||
| class | |S_j| | missing clients | ||Cbar_Sj − Cbar_literal1M,zero|| | ||Cbar_Sj|| |
|
||||
|---|---|---|---|---|
|
||||
| C&C | 10 | — | 0.000 | 5.819 |
|
||||
| Exfiltration | 9 | 6 | 0.724 | 7.243 |
|
||||
| Exploitation | 8 | 0, 6 | 1.131 | 5.654 |
|
||||
| Lateral_movement | 10 | — | 0.000 | 5.571 |
|
||||
| Normal | 10 | — | 0.000 | 6.042 |
|
||||
| RDOS | 10 | — | 0.000 | 8.705 |
|
||||
| Reconnaissance | 10 | — | 0.000 | 6.113 |
|
||||
| Tampering | 9 | 0 | 0.510 | 5.096 |
|
||||
| Weaponization | 10 | — | 0.000 | 6.166 |
|
||||
|
||||
For the three affected classes the hypothetical zero-filled `1/M` prototype is displaced
|
||||
by roughly 10–20 % of the prototype norm. Because inference is nearest-prototype in raw
|
||||
embedding space (not scale invariant), such displacement can change predictions for
|
||||
samples near class boundaries.
|
||||
|
||||
**Important caveats** (why this does not resolve the ambiguity):
|
||||
|
||||
- The zero-filled `1/M` prototype is only **one** hypothetical placeholder and is labelled
|
||||
hypothetical/analysis-only
|
||||
(`artifacts/analysis/sj_vs_literal_1m_hypothetical_xiiotid__alpha0.5__seed0__original_protean.csv`).
|
||||
Reproduced by the read-only script `scripts/validate_sj_participant_rule.py`.
|
||||
It was **not** used for training and is **not** an experiment mode.
|
||||
- Other equally-unspecified readings exist (retain previous round's prototype, skip the
|
||||
missing client but still divide by `M`, use the global prototype, etc.) and would give
|
||||
different answers.
|
||||
- The thesis statement "prototypes are shared only for locally observed classes" is the
|
||||
strongest available evidence for the `S_j` reading; the literal `1/M` equation is the
|
||||
strongest evidence for a denominator of `M`, but cannot be evaluated without an arbitrary
|
||||
placeholder.
|
||||
|
||||
### D.4 Defensibility of the current choice
|
||||
|
||||
The current `S_j` equal-weight rule remains **defensible as an explicitly documented
|
||||
reproduction assumption**: it is the only reading consistent with the thesis's stated
|
||||
sharing rule, and it avoids fabricating prototypes. It is **not** verified original
|
||||
behaviour, and the displacement analysis shows the choice is not immaterial. This project
|
||||
therefore continues to label `original_protean` a *documented interpretation*, and does
|
||||
**not** claim paper-reproduction success.
|
||||
|
||||
The `equal_available` control remains **unnecessary**: the baseline aggregation already
|
||||
uses equal weights over `S_j`, so it is arithmetically identical to the proposed `beta = 0`
|
||||
rule; introducing a separate mode would add no information. It stays deferred.
|
||||
|
||||
---
|
||||
|
||||
## E. Round-0, loss and inference validation
|
||||
|
||||
### E.1 Round-0 handling
|
||||
|
||||
Verified by `test_round0_sees_no_global_prototypes_and_round1_does`: during round 0 every
|
||||
client receives **no** global prototypes (empty), so the prototype-alignment term is
|
||||
skipped; after round-0 server aggregation the first global prototypes are broadcast back
|
||||
and used from round 1 onward. The round-0 snapshot stores the pre-aggregation client
|
||||
prototypes. Provenance: **INFERRED / PROJECT DECISION** (the paper's Algorithms say
|
||||
"initialize the global prototype set" but never define it).
|
||||
|
||||
### E.2 Loss implementation audit
|
||||
|
||||
The implemented local objective is (client.py):
|
||||
|
||||
```
|
||||
CE(logits, y) # nll_loss, mean over the batch
|
||||
+ lambda * ( sum_{matched samples x} ||phi(x) - Cbar_y||^2 / count_matched )
|
||||
+ (mu / 2) * sum_{params} ||w - w_global||^2
|
||||
```
|
||||
|
||||
Findings:
|
||||
|
||||
- **classification loss**: `F.nll_loss` on log-softmax outputs, default mean reduction.
|
||||
- **prototype alignment**: per **sample**, only for classes that currently have a global
|
||||
prototype; normalised by the number of matched samples in the batch (a mean), *not* a
|
||||
raw sum. Provenance **INFERRED** (FedProto-style reading of Eq. 3; the paper writes a
|
||||
class-mean-to-class-mean term).
|
||||
- **model alignment**: sum over parameter tensors of squared deviation from the round's
|
||||
global parameters, coefficient `mu/2`, `mu = 0.1` (**PAPER**).
|
||||
- **coefficients**: `lambda = 1.0`, `mu = 0.1`; guarded by
|
||||
`test_training_config_coefficients_are_frozen`.
|
||||
- **gradients flow** to the shared model parameters through all three terms (verified
|
||||
implicitly by parameter-change comparison).
|
||||
- **missing prototypes**: a label whose class has no global prototype contributes **no**
|
||||
alignment term and does **not** enter the normaliser.
|
||||
|
||||
`test_local_update_matches_documented_loss_formula` independently re-transcribes the
|
||||
documented formula and reproduces `local_update` parameters to `atol=1e-5`, pinning the
|
||||
reductions and coefficients. `test_prototype_alignment_skips_classes_without_a_global_prototype`
|
||||
confirms classes without a global prototype are skipped.
|
||||
|
||||
### E.3 Inference validation
|
||||
|
||||
- Rule: embedding `z = phi(x)`, prediction `argmin_j ||z - Cbar_j||^2` computed via the
|
||||
expanded-norm identity; class labels returned are the actual class ids
|
||||
(`test_nearest_prototype_predict_maps_noncontiguous_class_ids`).
|
||||
- `test_nearest_prototype_predict_matches_squared_l2_bruteforce` matches an independent
|
||||
brute-force squared-L2 argmin over 20 random points and 3 prototypes.
|
||||
- Empty global prototype sets raise an explicit `ValueError` from both
|
||||
`nearest_prototype_predict` and `server.predict_global`; global prototypes are never
|
||||
fabricated (`test_collect_global_omits_missing_class`).
|
||||
- `test_compute_local_prototypes_class_index_mapping` confirms local prototypes are keyed
|
||||
by true class id and only for observed classes.
|
||||
- Provenance: **PAPER** for the rule (thesis Eqs. 5.11–5.13); "classes considered = those
|
||||
with a global prototype" is **INFERRED**.
|
||||
|
||||
Minor note: the expanded-norm distance can be very slightly negative due to float
|
||||
rounding; `argmin` is unaffected and no sample observed a wrong nearest prototype under
|
||||
the brute-force cross-check.
|
||||
|
||||
---
|
||||
|
||||
## F. Determinism and reproducibility
|
||||
|
||||
### F.1 Repeat-run reproducibility (measured)
|
||||
|
||||
The same smoke condition (`xiiotid`, `alpha=0.5`, `seed=0`, 4,000 rows, 4 clients, 3 rounds,
|
||||
1 local epoch) was run twice into separate output roots. Diff of the two run trees:
|
||||
|
||||
- `history.json`, `test_global_metrics.json`, `test_confusion_matrix.json`,
|
||||
`test_per_class.csv`, `per_client.csv`, `train_support_Nij.csv`,
|
||||
`characterization.csv`, `rare_unseen.csv` and the prototype snapshot: **byte-identical**.
|
||||
- `manifest_content_hash` identical (`b5cf7ff...`); `preprocessing_hash` identical
|
||||
(`70fa831...`); `manifest_reload_ok = true` in both.
|
||||
- Only differences were run-specific metadata: the manifest `created_utc` timestamp and
|
||||
the explicit `manifest_path` (which pointed at the two different output roots). The
|
||||
content hash deliberately excludes `created_utc`/`python`, so partition identity is
|
||||
unchanged.
|
||||
|
||||
### F.2 Determinism source and tolerance
|
||||
|
||||
All randomness is isolated through `utils.seeding.derive_seed` (split, Dirichlet partition,
|
||||
model initialisation, client batch order). PyTorch dropout draws from the global torch RNG;
|
||||
the deterministic regression test reseeds it identically before each run, and two full
|
||||
`run_federated` invocations with the same seed produced **exactly equal** parameters,
|
||||
prototypes and history rows (`test_end_to_end_baseline_is_deterministic_and_learns_separable_fixture`).
|
||||
|
||||
Known nondeterminism source: none observed at the tested scales on this CPU backend. If a
|
||||
future accelerated backend introduces nondeterministic ops, the tolerance clause is the
|
||||
`atol=1e-5 / rtol=1e-4` used in the loss-replication test.
|
||||
|
||||
### F.3 Test suite
|
||||
|
||||
```
|
||||
.venv/bin/python -m pytest -q -> 46 passed
|
||||
```
|
||||
|
||||
34 pre-existing tests + 12 new baseline-validation tests
|
||||
(`tests/test_baseline_validation.py`).
|
||||
|
||||
---
|
||||
|
||||
## G. Remaining unresolved assumptions / blockers
|
||||
|
||||
1. **`S_j` participant/denominator rule** (§D) is a reproduction assumption, quantified but
|
||||
not resolved. No official code exists to settle it. Any between-method conclusion must
|
||||
carry this caveat.
|
||||
2. **`lambda = 1.0`** has no paper value; it is a PROJECT DECISION. Optimiser (Adam),
|
||||
learning rate (1e-3) and batch size (256) are also unspecified by the paper.
|
||||
3. **Prototype-alignment form** (per-sample embedding → class-mean global prototype) is an
|
||||
INFERENCE; the paper writes class-mean → class-mean.
|
||||
4. **Round-0 global-prototype initialisation** is INFERRED/PROJECT DECISION.
|
||||
5. **Model aggregation** is equal `1/M` (PAPER Eq. 2); the thesis intro's FedAvg-style
|
||||
"weighted average" wording conflicts with this and is not implemented (documented
|
||||
conflict, not silently chosen).
|
||||
6. **Per-client evaluation partitioning** (Dirichlet over the test split) is a PROJECT
|
||||
DECISION; the paper does not describe per-client test assignment.
|
||||
7. **Manifest provenance gap**: the Session-2 manifests store `preprocessing_hash = null`
|
||||
and `config_hash = null`, so they do not bind the preprocessing recipe directly. The
|
||||
split hash, partition hash and `N_ij` do reproduce exactly, and the current runner
|
||||
records the live preprocessing hash in run metadata; binding it into future manifests
|
||||
would close this gap.
|
||||
8. **Only the `(alpha=0.50, seed=0)` condition per dataset has been run**; the remaining
|
||||
`(alpha, seed)` grid is unmeasured. No sweep may be launched yet.
|
||||
9. Reproduction success is **not** claimed: two completed runs (one per dataset, one
|
||||
condition each) are consistent with a working baseline, not proof that original PROTEAN
|
||||
is reproduced.
|
||||
|
||||
---
|
||||
|
||||
## Appendix: provenance checklist for the frozen baseline
|
||||
|
||||
Classification: CODE (none exist), PAPER, INFERRED, PROJECT DECISION, UNRESOLVED.
|
||||
|
||||
| Item | Frozen value | Provenance |
|
||||
|---|---|---|
|
||||
| Dataset preprocessing | median-impute + standardise + one-hot, train-fitted, `__MISSING__` category, dup drop, `crypto-ransomware` dropped (X-IIoTID) | PROJECT DECISION (label drop: PAPER) |
|
||||
| Split | stratified 70/10/20, seed-derived | PAPER (80/20 test) + PROJECT DECISION (10 % validation, stratification) |
|
||||
| Label mapping | canonical labels, `Lateral_movement` fix, sorted class ids | PAPER (fix required) + PROJECT DECISION |
|
||||
| Model architecture | 2×Conv1D(64,128)+ReLU+pool+dropout(0.2,0.5), dense 128, log-softmax | PAPER (kernel 3/padding 1: PROJECT DECISION) |
|
||||
| Optimizer / lr / batch | Adam / 1e-3 / 256 | PROJECT DECISION |
|
||||
| Local epochs / rounds | 3 / 10 | PAPER |
|
||||
| mu | 0.1 | PAPER |
|
||||
| lambda | 1.0 | PROJECT DECISION |
|
||||
| Local prototype | end-of-round mean embedding per observed class | PAPER Eq. 1 |
|
||||
| Model aggregation | equal 1/M | PAPER Eq. 2/5.5 |
|
||||
| Prototype aggregation | equal mean over `S_j` | **REPRODUCTION ASSUMPTION** (PROJECT DECISION; paper unresolved) |
|
||||
| Round-0 handling | no prototypes, prototype term skipped | INFERRED / PROJECT DECISION |
|
||||
| Prototype alignment | per-sample, matched classes only, mean over matched | INFERRED |
|
||||
| Inference | nearest global prototype, squared L2, available classes | PAPER (available-class restriction INFERRED) |
|
||||
| Per-client eval partition | Dirichlet over test split | PROJECT DECISION |
|
||||
| Seed policy | `{0,1,2}`, derive_seed per purpose | PROJECT DECISION |
|
||||
|
||||
`planningawal/` was not modified during validation.
|
||||
Reference in new issue
Block a user