experiment: validate beta selection at alpha 0.25
This commit is contained in:
willtanoe committed
1 parent
6b97a0d38b
commit
ff68d380e2
22 files changed
+33471
-209
No files matched your search
@@ -0,0 +1,467 @@
|
||||
# Alpha-0.25 Beta Validation (Session 7)
|
||||
|
||||
Status: **three-seed paired protocol completed for a second non-IID condition**
|
||||
(X-IIoTID `alpha = 0.25`, seeds `{0,1,2}`) with pooled-across-seed validation-only beta
|
||||
selection, followed by a single final test evaluation of the pooled-selected beta. This is a
|
||||
**paired characterisation of a more heterogeneous non-IID setting**, not a sweep, not a
|
||||
hyper-parameter search over seeds, and not evidence of a global optimum.
|
||||
|
||||
Companion documents:
|
||||
|
||||
- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen protocol and selection hierarchy.
|
||||
- `docs/research/MULTISEED_BETA_VALIDATION.md` — Session-6 alpha=0.50 three-seed condition.
|
||||
- `docs/research/PAIRED_BETA_VALIDATION.md` — Session-5 single-seed condition.
|
||||
- `docs/research/SESSION6_AUDIT.md` — independent audit of the alpha=0.50 protocol.
|
||||
- `docs/research/BETA_AGGREGATION_IMPLEMENTATION.md` — beta mode implementation.
|
||||
- `docs/baseline/BASELINE_VALIDATION.md` — frozen baseline evidence.
|
||||
- `docs/handoff/SESSION_HANDOFF.md` — continuation point.
|
||||
|
||||
---
|
||||
|
||||
## A. Scope
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| dataset | `xiiotid` (X-IIoTID) |
|
||||
| alpha | 0.25 |
|
||||
| seeds | 0, 1, 2 |
|
||||
| aggregation | `original_protean` (per seed) + `support_weighted` for `beta ∈ {0, 0.25, 0.5, 0.75, 1}` |
|
||||
| runs | 18: 15 `support_weighted` + 3 `original_protean` |
|
||||
| new this session | all 18 runs (no alpha=0.25 run pre-existed) |
|
||||
| compute | sequential on CPU, one run at a time, ≈ 12.17 h wall-clock total |
|
||||
|
||||
No other alpha, dataset, seed, mode or beta was run. `equal_available` was not added.
|
||||
alpha=0.75 and 5G-NIDD were deliberately **not** run.
|
||||
|
||||
---
|
||||
|
||||
## B. Partition characterization (realized, before interpretation)
|
||||
|
||||
Manifests are reused, never regenerated. Each manifest was reloaded and independently
|
||||
reproduced from the raw dataset (`reproduce_train_assignments`: split hash, `N_ij` and
|
||||
partition hash all verified). All three seeds share train/validation/test sizes
|
||||
569,909 / 81,414 / 162,831 (10 clients, 9 classes, `Normal` benign).
|
||||
|
||||
| seed | manifest | content hash | partition hash | split hash | rng seed |
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | `artifacts/partitions/xiiotid__alpha0.25__seed0.json` | `839ba18e…` | `60faf299…` | `d0acc0c1…` | 1617371214 |
|
||||
| 1 | `artifacts/partitions/xiiotid__alpha0.25__seed1.json` | `84c20b4b…` | `6ffb10ae…` | `880a9f82…` | 1670706564 |
|
||||
| 2 | `artifacts/partitions/xiiotid__alpha0.25__seed2.json` | `d6e68ee1…` | `c8e1ad31…` | `8e0db0be…` | 795318573 |
|
||||
|
||||
Client-sample totals and class-support characterization (`N_ij`, training split). `CV` is the
|
||||
population coefficient of variation of the positive supports within a class; `max/min` is over
|
||||
contributing clients only.
|
||||
|
||||
### Seed 0 (`|S_j| < 10` in 6/9 classes)
|
||||
|
||||
| class | total | #clients | min+ | max | mean+ | CV | max/min |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| C&C | 2,004 | 10 | 21 | 1,065 | 200.4 | 1.478 | 50.7 |
|
||||
| Exfiltration | 15,494 | 8 | 4 | 7,950 | 1,936.8 | 1.405 | 1987.5 |
|
||||
| Exploitation | 793 | 6 | 2 | 595 | 132.2 | 1.580 | 297.5 |
|
||||
| Lateral_movement | 22,117 | 9 | 2 | 13,857 | 2,457.4 | 1.810 | 6928.5 |
|
||||
| Normal | 292,777 | 10 | 1 | 122,800 | 29,277.7 | 1.346 | 122800.0 |
|
||||
| RDOS | 98,883 | 10 | 166 | 31,089 | 9,888.3 | 1.080 | 187.3 |
|
||||
| Reconnaissance | 89,161 | 8 | 46 | 65,307 | 11,145.1 | 1.855 | 1419.7 |
|
||||
| Tampering | 3,586 | 8 | 9 | 2,272 | 448.2 | 1.569 | 252.4 |
|
||||
| Weaponization | 45,094 | 8 | 43 | 23,381 | 5,636.8 | 1.528 | 543.7 |
|
||||
|
||||
client-sample totals min 9,075 / max 141,358 (ratio 15.6).
|
||||
|
||||
### Seed 1 (`|S_j| < 10` in 7/9 classes)
|
||||
|
||||
| class | total | #clients | min+ | max | mean+ | CV | max/min |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| C&C | 2,004 | 9 | 1 | 986 | 222.7 | 1.559 | 986.0 |
|
||||
| Exfiltration | 15,494 | 10 | 1 | 6,204 | 1,549.4 | 1.238 | 6204.0 |
|
||||
| Exploitation | 793 | 8 | 1 | 321 | 99.1 | 1.295 | 321.0 |
|
||||
| Lateral_movement | 22,117 | 10 | 1 | 10,094 | 2,211.7 | 1.434 | 10094.0 |
|
||||
| Normal | 292,777 | 8 | 198 | 129,678 | 36,597.1 | 1.163 | 654.9 |
|
||||
| RDOS | 98,883 | 9 | 168 | 53,482 | 10,987.0 | 1.468 | 318.4 |
|
||||
| Reconnaissance | 89,161 | 8 | 138 | 65,101 | 11,145.1 | 1.852 | 471.8 |
|
||||
| Tampering | 3,586 | 9 | 2 | 1,804 | 398.4 | 1.348 | 902.0 |
|
||||
| Weaponization | 45,094 | 9 | 1 | 12,024 | 5,010.4 | 0.833 | 12024.0 |
|
||||
|
||||
client-sample totals min 9,278 / max 193,326 (ratio 20.8).
|
||||
|
||||
### Seed 2 (`|S_j| < 10` in 6/9 classes)
|
||||
|
||||
| class | total | #clients | min+ | max | mean+ | CV | max/min |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| C&C | 2,004 | 6 | 2 | 1,265 | 334.0 | 1.315 | 632.5 |
|
||||
| Exfiltration | 15,494 | 8 | 4 | 8,269 | 1,936.8 | 1.337 | 2067.3 |
|
||||
| Exploitation | 793 | 6 | 1 | 485 | 132.2 | 1.328 | 485.0 |
|
||||
| Lateral_movement | 22,117 | 7 | 1 | 12,496 | 3,159.6 | 1.475 | 12496.0 |
|
||||
| Normal | 292,777 | 9 | 452 | 118,937 | 32,530.8 | 1.104 | 263.1 |
|
||||
| RDOS | 98,883 | 10 | 2 | 48,839 | 9,888.3 | 1.421 | 24419.5 |
|
||||
| Reconnaissance | 89,161 | 10 | 1 | 46,541 | 8,916.1 | 1.518 | 46541.0 |
|
||||
| Tampering | 3,586 | 10 | 1 | 2,584 | 358.6 | 2.134 | 2584.0 |
|
||||
| Weaponization | 45,094 | 9 | 4 | 22,853 | 5,010.4 | 1.471 | 5713.3 |
|
||||
|
||||
client-sample totals min 15,664 / max 122,526 (ratio 7.8).
|
||||
|
||||
### Rare / unseen (training-derived, per client)
|
||||
|
||||
Every client has exactly two rare classes in every seed. Unseen classes are much more
|
||||
frequent than at alpha=0.50:
|
||||
|
||||
| seed | rare per client | union rare | unseen per client | union unseen |
|
||||
|---|---|---|---|---|
|
||||
| 0 | 2, 2, 2, 2, 2, 2, 2, 2, 2, 2 | 8 classes | 0, 3, 1, 2, 2, 1, 2, 0, 0, 2 | 6 classes |
|
||||
| 1 | 2 × 10 | 7 classes | 0, 0, 4, 1, 1, 0, 1, 0, 1, 0 | 6 classes |
|
||||
| 2 | 2 × 10 | 8 classes | 3, 0, 0, 2, 2, 2, 2, 0, 1, 2 | 5 classes |
|
||||
|
||||
### Comparison of realized heterogeneity (alpha=0.25 vs alpha=0.50)
|
||||
|
||||
| statistic | alpha=0.25 | alpha=0.50 |
|
||||
|---|---|---|
|
||||
| class-condition cells with `|S_j| < M` | **19 / 27** | 7 / 27 |
|
||||
| missing client-class cells | **38** | 9 |
|
||||
| client-sample-total max/min (per seed) | 15.6, 20.8, 7.8 | 5.1, 7.2, 9.9 |
|
||||
| union rare classes per seed | 8, 7, 8 | 6, 7, 7 |
|
||||
| union unseen classes per seed | 6, 6, 5 | 3, 3, 1 |
|
||||
| max unseen classes on a client | 4 | 2 |
|
||||
|
||||
The lower Dirichlet parameter does propagate to more missing classes, more unseen classes, and
|
||||
a larger client-size spread — but **not uniformly**: seed 2 at alpha=0.25 has a smaller
|
||||
client-size ratio (7.8) than alpha=0.50 seed 2 (9.9), and its `|S_j| < M` count is 6/9 versus
|
||||
1/9 at alpha=0.50. Interpretation uses the realized partitions, not alpha alone.
|
||||
|
||||
---
|
||||
|
||||
## C. Frozen protocol
|
||||
|
||||
Identical to `EXPERIMENT_PROTOCOL.md` §14 and to Session 6: 10 clients, 10 rounds, 3 local
|
||||
epochs, batch 256, Adam `lr=1e-3`, `mu=0.1`, `lambda=1.0`; unchanged architecture,
|
||||
preprocessing, train/validation/test split recipe, partition recipe, loss terms and inference.
|
||||
All 18 runs were performed this session with `scripts/run_baseline.py` at HEAD `6b97a0d`
|
||||
(no training/aggregation source changed).
|
||||
|
||||
### C.1 Manifest and build-level pairing
|
||||
|
||||
Within each seed all six runs share one manifest content hash and byte-identical deterministic
|
||||
artifacts:
|
||||
|
||||
| seed | manifest content hash | round-0 snapshot SHA-256 (all 6 runs) | support / characterization / rare_unseen |
|
||||
|---|---|---|---|
|
||||
| 0 | `839ba18e…` | `cd821beb…` | identical |
|
||||
| 1 | `84c20b4b…` | `59d2d2bc…` | identical |
|
||||
| 2 | `d6e68ee1…` | `47befeff…` | identical |
|
||||
|
||||
`manifest_reload_ok = true` and `smoke = false` in all 18 runs; metadata confirms
|
||||
`dataset=xiiotid`, `alpha=0.25`, seed, mode, beta, clients=10, rounds=10, local_epochs=3,
|
||||
batch=256, lr=1e-3, mu=0.1, lambda=1.0. Pairing is proven by content hashes, not filenames.
|
||||
|
||||
### C.2 Selection rule (unchanged)
|
||||
|
||||
- **PRIMARY:** pooled-across-seed validation per-client-mean rare-class macro-F1 (average the
|
||||
seed-level per-client mean over seeds `{0,1,2}`), selected with validation only.
|
||||
- **SECONDARY (diagnostic):** per-seed validation selection, for stability only.
|
||||
- **TEST:** read only after the pooled beta is fixed.
|
||||
|
||||
---
|
||||
|
||||
## D. Per-seed validation results
|
||||
|
||||
`val rareMacroF1 (pc mean)` is the primary metric; all 10 clients are eligible in every seed.
|
||||
Validation only.
|
||||
|
||||
### Seed 0
|
||||
|
||||
| beta | eligible | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | 10 | 0.616380 | 0.642939 | 0.841886 | 0.676032 | 0.874776 |
|
||||
| 0.25 | 10 | 0.609913 | 0.646423 | 0.834580 | 0.679089 | 0.867038 |
|
||||
| 0.50 | 10 | 0.595247 | 0.633677 | 0.807172 | 0.667755 | 0.853256 |
|
||||
| **0.75** | 10 | **0.654033** | **0.666477** | 0.797400 | **0.697353** | 0.854313 |
|
||||
| 1.00 | 10 | 0.578385 | 0.585755 | 0.710305 | 0.626018 | 0.847188 |
|
||||
|
||||
### Seed 1
|
||||
|
||||
| beta | eligible | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | 10 | 0.492425 | 0.580976 | 0.824292 | 0.665021 | 0.861707 |
|
||||
| 0.25 | 10 | 0.514237 | 0.605558 | 0.833188 | 0.684847 | 0.870391 |
|
||||
| **0.50** | 10 | **0.537040** | 0.647568 | 0.829322 | 0.718155 | 0.880917 |
|
||||
| 0.75 | 10 | 0.535672 | **0.655554** | 0.817017 | **0.723896** | 0.879210 |
|
||||
| 1.00 | 10 | 0.530942 | 0.652257 | 0.803940 | 0.721164 | 0.874346 |
|
||||
|
||||
### Seed 2
|
||||
|
||||
| beta | eligible | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **0.00** | 10 | **0.666307** | **0.673945** | 0.819363 | **0.703911** | **0.882895** |
|
||||
| 0.25 | 10 | 0.641922 | 0.655402 | 0.825502 | 0.687134 | 0.872024 |
|
||||
| 0.50 | 10 | 0.625267 | 0.636281 | 0.806218 | 0.670651 | 0.868462 |
|
||||
| 0.75 | 10 | 0.548685 | 0.558044 | 0.712391 | 0.601211 | 0.851058 |
|
||||
| 1.00 | 10 | 0.486970 | 0.499711 | 0.637821 | 0.549558 | 0.838517 |
|
||||
|
||||
Machine-readable: `artifacts/analysis/xiiotid__alpha0.25__multiseed__validation_per_seed.csv`.
|
||||
|
||||
---
|
||||
|
||||
## E. Per-seed diagnostic beta choices (SECONDARY, stability only)
|
||||
|
||||
| seed | selected beta | val rareMacroF1 (pc mean) | tie-break used | test_used_for_selection |
|
||||
|---|---|---|---|---|
|
||||
| 0 | 0.75 | 0.654033 | no | false |
|
||||
| 1 | 0.50 | 0.537040 | no | false |
|
||||
| 2 | 0.00 | 0.666307 | no | false |
|
||||
|
||||
The preferred beta is seed-dependent (0.75 / 0.50 / 0.00). These per-seed choices are
|
||||
**diagnostic only** and are not the condition-level selection rule.
|
||||
`val_selection.json` records `"selection_split": "validation"` and
|
||||
`"test_used_for_selection": false` for all 15 `support_weighted` runs.
|
||||
|
||||
---
|
||||
|
||||
## F. PRIMARY pooled-across-seed validation selection
|
||||
|
||||
| beta | pooled mean | std (sample, ddof=1) | std (pop) | min | max | pooled val rareRecall (pc mean) | selected |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **0.00** | **0.591704** | 0.089529 | 0.073100 | 0.492425 | 0.666307 | 0.759482 | **yes** |
|
||||
| 0.25 | 0.588691 | 0.066435 | 0.054244 | 0.514237 | 0.641922 | 0.780771 | |
|
||||
| 0.50 | 0.585851 | 0.044858 | 0.036626 | 0.537040 | 0.625267 | 0.772536 | |
|
||||
| 0.75 | 0.579463 | 0.064906 | 0.052996 | 0.535672 | 0.654033 | 0.760791 | |
|
||||
| 1.00 | 0.532099 | 0.045718 | 0.037329 | 0.486970 | 0.578385 | 0.704575 | |
|
||||
|
||||
- **PRIMARY selected beta (`beta_star_alpha025`) = 0.00.**
|
||||
- **Criterion:** maximum pooled validation per-client-mean rare-class macro-F1.
|
||||
- **Tie-break used:** no (no exact tie; the pooled means separate by > 1e-3).
|
||||
- **`test_used_for_selection = false`.** Test metrics were not inspected until the pooled beta
|
||||
was fixed.
|
||||
- **Interpretation:** `beta = 0.00` is the *best among the tested candidates under pooled
|
||||
validation across the three project-defined seeds for X-IIoTID alpha=0.25.* It is **not** a
|
||||
global optimum. The pooled validation ordering is **monotonically decreasing in beta**.
|
||||
|
||||
Machine-readable: `artifacts/analysis/xiiotid__alpha0.25__multiseed__validation_pooled.csv`
|
||||
and `...__selection_summary.json`.
|
||||
|
||||
---
|
||||
|
||||
## G. Final selected-beta test metrics (descriptive only)
|
||||
|
||||
`beta_star = 0.00` was fixed from pooled validation; only then were test metrics read.
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.641362 | 0.580555 | 0.674489 | 0.632136 ± 0.047642 |
|
||||
| rare-class recall | 0.830701 | 0.825699 | 0.807370 | 0.821257 ± 0.012283 |
|
||||
| macro-F1 | 0.674558 | 0.665039 | 0.704433 | 0.681343 ± 0.020555 |
|
||||
| accuracy | 0.874551 | 0.863490 | 0.884002 | 0.874015 ± 0.010267 |
|
||||
| weighted-F1 | 0.906156 | 0.899864 | 0.914627 | 0.906882 ± 0.007408 |
|
||||
|
||||
---
|
||||
|
||||
## H. `original_protean` comparison
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.639810 | 0.582522 | 0.676039 | 0.632790 ± 0.047152 |
|
||||
| rare-class recall | 0.829601 | 0.826700 | 0.809595 | 0.821965 ± 0.010811 |
|
||||
| macro-F1 | 0.673103 | 0.666981 | 0.705969 | 0.682018 ± 0.020967 |
|
||||
| accuracy | 0.873722 | 0.867341 | 0.885231 | 0.875431 ± 0.009067 |
|
||||
| weighted-F1 | 0.905466 | 0.902075 | 0.915416 | 0.907652 ± 0.006934 |
|
||||
|
||||
Descriptive difference (`beta_star` − `original_protean`) for test rare-class macro-F1 is
|
||||
+0.001552 (seed 0), −0.001967 (seed 1), −0.001550 (seed 2); mean −0.000655. This is expected:
|
||||
`beta = 0` is mathematically the equal mean over `S_j` (see §I), so the end-to-end difference is
|
||||
floating-point trajectory noise, not a method effect. No superiority or generalisation is
|
||||
claimed; this is one dataset, one alpha, three project-defined seeds.
|
||||
|
||||
Test columns are descriptive only and are recorded in
|
||||
`artifacts/analysis/xiiotid__alpha0.25__multiseed__test_descriptive.csv`.
|
||||
|
||||
---
|
||||
|
||||
## I. `beta = 0` equivalence vs `original_protean`
|
||||
|
||||
For all three seeds, `support_weighted beta=0` vs frozen `original_protean`:
|
||||
|
||||
| property | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| same manifest / split / hashes | yes | yes | yes |
|
||||
| round-0 snapshot byte-identical | yes (`cd821beb…`) | yes (`59d2d2bc…`) | yes (`47befeff…`) |
|
||||
| `train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv` identical | yes | yes | yes |
|
||||
| independent aggregation diff (`ΣC/n` vs `(1/n)ΣC`) | 4.4e-16 | 4.4e-16 | 2.2e-16 |
|
||||
| history rare-macro-F1 round-0 diff | 0.0 | 0.0 | 0.0 |
|
||||
| history rare-macro-F1 max diff (rounds ≥ 1) | 6.6e-3 | 3.6e-3 | 3.7e-3 |
|
||||
| test metric max diff | 1.6e-3 | 3.9e-3 | 2.2e-3 |
|
||||
| **classification** | **B** | **B** | **B** |
|
||||
|
||||
Class **B**: semantic equivalence + floating-point numerical divergence; **no semantic
|
||||
discrepancy** in any seed. Round 0 is byte-identical; the ~1 ULP ordering difference enters the
|
||||
round-1 prototype-alignment training and the trajectories diverge slowly. Equivalence remains
|
||||
conditional on the shared `S_j` reproduction assumption and is **not** proof that `beta = 0`
|
||||
reproduces original PROTEAN.
|
||||
|
||||
---
|
||||
|
||||
## J. Direct prototype-effect findings (fixed round-0 snapshots, no retraining)
|
||||
|
||||
Computed with the production aggregation from each seed's fixed `{C_ij, N_ij}` round-0
|
||||
snapshot. `||Cbar_j(beta) - Cbar_j(0)||` at `beta = 1.0`, cross-seed mean (sample std):
|
||||
|
||||
| class | mean disp | std | min | max | mean rel disp | typical `|S_j|` |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Exfiltration | 4.316 | 0.898 | 3.588 | 5.319 | 0.664 | 8 |
|
||||
| Tampering | 4.248 | 0.241 | 3.978 | 4.441 | 0.965 | 8 |
|
||||
| C&C | 4.020 | 0.290 | 3.691 | 4.242 | 0.847 | 10 |
|
||||
| Lateral_movement | 3.726 | 1.383 | 2.342 | 5.109 | 0.790 | 9 |
|
||||
| Exploitation | 3.464 | 1.113 | 2.180 | 4.152 | 0.702 | 6 |
|
||||
| Reconnaissance | 3.339 | 1.107 | 2.089 | 4.195 | 0.640 | 8 |
|
||||
| Weaponization | 3.124 | 0.585 | 2.486 | 3.636 | 0.563 | 8 |
|
||||
| RDOS | 2.566 | 1.219 | 1.848 | 3.974 | 0.328 | 10 |
|
||||
| Normal | 1.997 | 0.533 | 1.667 | 2.611 | 0.370 | 10 |
|
||||
|
||||
Findings (descriptive):
|
||||
|
||||
- Every class moves for `beta > 0` in every seed, because every class has unequal support across
|
||||
its contributors.
|
||||
- Classes with the fewest contributors / strongest support imbalance (`Tampering`, `C&C`,
|
||||
`Exfiltration`, `Exploitation`) have the largest relative displacement; `Tampering` at
|
||||
`rel = 0.965` moves almost one full prototype norm.
|
||||
- `RDOS` and `Normal` move least in relative terms despite large absolute class sizes, because
|
||||
their support is spread more evenly across all 10 clients (`RDOS` CV ≈ 1.08–1.47).
|
||||
- Displacement is a **direct-aggregation measure only** and is not interpreted as improved or
|
||||
degraded detection. The end-to-end effect (§F/§G) is reported separately.
|
||||
|
||||
Outputs: `artifacts/analysis/xiiotid__alpha0.25__seed{0,1,2}__support_weighted__beta0__prototype_snapshot__prototype_effect.csv`
|
||||
and `...__prototype_weights.csv`; cross-seed aggregate
|
||||
`artifacts/analysis/xiiotid__alpha0.25__multiseed__prototype_effect_pooled.csv`.
|
||||
|
||||
---
|
||||
|
||||
## K. Client-level effect (validation, relative to beta = 0)
|
||||
|
||||
Full per-client table:
|
||||
`artifacts/analysis/xiiotid__alpha0.25__multiseed__client_validation_detail.csv`; summary:
|
||||
`...__client_validation_summary.csv`.
|
||||
|
||||
| seed | beta | improve | degrade | unchanged | mean Δ | min Δ | max Δ |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 0 | 0.25 | 2 | 5 | 3 | −0.0065 | −0.0429 | +0.0188 |
|
||||
| 0 | 0.50 | 2 | 7 | 1 | −0.0211 | −0.0769 | +0.0574 |
|
||||
| 0 | 0.75 | 5 | 4 | 1 | +0.0377 | −0.1117 | +0.3759 |
|
||||
| 0 | 1.00 | 4 | 5 | 1 | −0.0380 | −0.2570 | +0.1410 |
|
||||
| 1 | 0.25 | 7 | 1 | 2 | +0.0218 | −0.0064 | +0.0718 |
|
||||
| 1 | 0.50 | 7 | 2 | 1 | +0.0446 | −0.0158 | +0.2356 |
|
||||
| 1 | 0.75 | 5 | 4 | 1 | +0.0432 | −0.0183 | +0.3167 |
|
||||
| 1 | 1.00 | 5 | 4 | 1 | +0.0385 | −0.0387 | +0.3774 |
|
||||
| 2 | 0.25 | 5 | 3 | 2 | −0.0244 | −0.3029 | +0.0819 |
|
||||
| 2 | 0.50 | 4 | 4 | 2 | −0.0410 | −0.3398 | +0.1185 |
|
||||
| 2 | 0.75 | 3 | 5 | 2 | −0.1176 | −0.6011 | +0.1241 |
|
||||
| 2 | 1.00 | 3 | 6 | 1 | −0.1793 | −0.6775 | +0.0754 |
|
||||
|
||||
Descriptive observations:
|
||||
|
||||
- Unlike alpha=0.50 (where `beta = 0.75` improved the majority of clients in every seed), at
|
||||
alpha=0.25 the higher-beta response is **seed-dependent and often negative**: seeds 0 and 2
|
||||
degrade on average for `beta ≥ 0.25`, while seed 1 improves for `0.25–1.00`.
|
||||
- The magnitude of losses grows with beta: seed 2 loses up to −0.677 on one client at
|
||||
`beta = 1.00`. The largest single gains also grow with beta (seed 1, +0.377), i.e. beta
|
||||
redistributes sharply rather than uniformly.
|
||||
- Client ids are per-seed realisations and are not treated as one population. This analysis is
|
||||
descriptive and non-causal; `N_ij` vs ΔF1 remains non-monotonic (some low-support clients
|
||||
gain, some lose).
|
||||
|
||||
---
|
||||
|
||||
## L. Comparison with alpha=0.50
|
||||
|
||||
Validation-side evidence (selection basis); test is not used to explain the selection.
|
||||
|
||||
| beta | alpha=0.50 pooled mean (std) | alpha=0.25 pooled mean (std) |
|
||||
|---|---|---|
|
||||
| 0.00 | 0.544361 (0.091447) | **0.591704 (0.089529)** |
|
||||
| 0.25 | 0.555434 (0.090068) | 0.588691 (0.066435) |
|
||||
| 0.50 | 0.567322 (0.088539) | 0.585851 (0.044858) |
|
||||
| 0.75 | **0.579367 (0.075121)** | 0.579463 (0.064906) |
|
||||
| 1.00 | 0.573479 (0.100450) | 0.532099 (0.045718) |
|
||||
| selected | **0.75** | **0.00** |
|
||||
| pooled mean range | 0.544–0.579 (0.035) | 0.532–0.592 (0.060) |
|
||||
|
||||
> **Disambiguation (audited, Session 7).** The condition-level pooled selections are
|
||||
> alpha=0.50 → **0.75** (Session 6) and alpha=0.25 → **0.00** (Session 7). The value **0.50**
|
||||
> is *not* the alpha=0.50 condition-level selection: it is only the Session-5 **seed-0
|
||||
> single-condition** choice and the Session-6 alpha=0.50 **per-seed diagnostic** for seed 0.
|
||||
> Any phrasing of the form "0.50 → 0.00" that refers to condition-level pooled selection is
|
||||
> incorrect. See `docs/research/SESSION7_AUDIT.md` §M.
|
||||
|
||||
Observations:
|
||||
|
||||
- **The pooled validation-selected beta changes with non-IID severity**: 0.75 at alpha=0.50 vs
|
||||
**0.00** at alpha=0.25. At alpha=0.25 the pooled validation curve is monotonically decreasing
|
||||
in beta; at alpha=0.50 it rises to a peak at 0.75.
|
||||
- **Across-seed std does not shrink at the more heterogeneous setting**: 0.045–0.090
|
||||
(alpha=0.25) vs 0.075–0.100 (alpha=0.50). The selected beta has a lower across-seed std at
|
||||
alpha=0.25 (0.0895) but this is not the selection criterion (no tie).
|
||||
- **Realized partition differences accompany the beta change** (§B): alpha=0.25 has 19/27
|
||||
`|S_j| < M` cell conditions vs 7/27, more unseen classes, and larger per-class support
|
||||
imbalance, so up-weighting high-support clients redistributes the global prototypes further
|
||||
(mean `||Cbar(1)−Cbar(0)||` 3.42 vs 2.10; §M/§J) and is more often harmful on validation.
|
||||
- **Client-level pattern flips**: at alpha=0.50 higher beta helped most clients in every seed;
|
||||
at alpha=0.25 it helps mainly in seed 1 and hurts on average in seeds 0 and 2.
|
||||
|
||||
These are descriptive associations. The realized partition characteristics (not alpha alone)
|
||||
are reported alongside, and **no causal claim is made that alpha by itself caused the
|
||||
difference**. The beta effect remains small relative to between-seed variance at both settings.
|
||||
|
||||
---
|
||||
|
||||
## M. Runtime and resources
|
||||
|
||||
CPU only, no GPU, no swap, exit status 0 for every run (`/usr/bin/time -v`). Per-run
|
||||
wall-clock 39:49–42:12.
|
||||
|
||||
| seed | tag | wall | CPU % | peak RSS |
|
||||
|---|---|---|---|---|
|
||||
| 0 | `original_protean` | 40:10.80 | 746 | 8.38 GiB |
|
||||
| 0 | beta 0.00 | 40:57.24 | 744 | 8.38 GiB |
|
||||
| 0 | beta 0.25 | 40:27.96 | 745 | 8.38 GiB |
|
||||
| 0 | beta 0.50 | 40:21.77 | 743 | 8.39 GiB |
|
||||
| 0 | beta 0.75 | 39:49.26 | 746 | 8.38 GiB |
|
||||
| 0 | beta 1.00 | 40:25.86 | 745 | 8.38 GiB |
|
||||
| 1 | `original_protean` | 40:29.86 | 745 | 10.80 GiB |
|
||||
| 1 | beta 0.00 | 39:52.79 | 745 | 10.80 GiB |
|
||||
| 1 | beta 0.25 | 40:45.79 | 745 | 10.80 GiB |
|
||||
| 1 | beta 0.50 | 40:18.93 | 745 | 10.80 GiB |
|
||||
| 1 | beta 0.75 | 39:58.73 | 744 | 10.80 GiB |
|
||||
| 1 | beta 1.00 | 40:47.41 | 745 | 10.80 GiB |
|
||||
| 2 | `original_protean` | 40:13.48 | 745 | 7.51 GiB |
|
||||
| 2 | beta 0.00 | 40:24.04 | 746 | 7.51 GiB |
|
||||
| 2 | beta 0.25 | 41:45.12 | 743 | 7.51 GiB |
|
||||
| 2 | beta 0.50 | 40:22.35 | 744 | 7.51 GiB |
|
||||
| 2 | beta 0.75 | 42:11.66 | 741 | 7.51 GiB |
|
||||
| 2 | beta 1.00 | 40:35.98 | 744 | 7.51 GiB |
|
||||
|
||||
- New compute this session: **18 runs, ≈ 43,799 s ≈ 12.17 h wall-clock**, sequential on an
|
||||
8-core host (seed chains ≈ 4.04 / 4.04 / 4.09 h).
|
||||
- Peak RSS is partition-dependent, not beta-dependent: 8.38 GiB (seed 0), 10.80 GiB (seed 1),
|
||||
7.51 GiB (seed 2); all well within the 29 GiB host.
|
||||
- No warnings, no client failures, 9/9 global prototypes in every round, no retries.
|
||||
- Machine-readable: `artifacts/analysis/xiiotid__alpha0.25__multiseed__runtime.csv`.
|
||||
|
||||
---
|
||||
|
||||
## N. Limitations
|
||||
|
||||
- **One dataset** (X-IIoTID). No cross-dataset conclusion.
|
||||
- **Only alpha 0.25 and 0.50 evaluated so far.** alpha=0.75 remains unmeasured.
|
||||
- **Three project-defined seeds** (`{0,1,2}`); the pooled mean’s standard error is of the same
|
||||
order as the between-beta spread, so the selected beta is statistically weak.
|
||||
- The `S_j` participant/denominator rule
|
||||
(`S_j = {i | N_ij > 0 and valid C_ij}`, equal weights in the baseline) remains a
|
||||
**REPRODUCTION ASSUMPTION**. At alpha=0.25 it is material in **19/27 class-condition cells**.
|
||||
The `beta = 0` comparison is conditional on both modes sharing this rule and does **not**
|
||||
prove `beta = 0` reproduces original PROTEAN.
|
||||
- Train/validation/test adaptation (70/10/20 stratified) differs from the paper’s literal
|
||||
80/20; the validation eval partition is a project decision.
|
||||
- **No 5G-NIDD beta result.**
|
||||
- **No global beta optimum claim.** `beta = 0.00` is only the best among the five tested
|
||||
candidates under pooled validation for X-IIoTID alpha=0.25 across seeds `{0,1,2}`.
|
||||
- The union of rare classes (and hence global rare support) differs by seed and by alpha;
|
||||
only the per-client-mean metric is directly comparable within a condition. Cross-alpha
|
||||
comparisons in §L are descriptive, not controlled.
|
||||
- Test metrics of the selected beta are single-condition and must not be read as
|
||||
generalisation; no test metric influenced selection.
|
||||
@@ -0,0 +1,378 @@
|
||||
# Session 7 Post-Experiment Audit (X-IIoTID alpha=0.25)
|
||||
|
||||
Status: **PASS WITH MINOR CORRECTIONS.** Independent recomputation from machine-readable
|
||||
artifacts confirms every Session 7 claim. No discovered issue changes `beta_star` or the
|
||||
scientific interpretation. Corrections are documentation-only (cross-alpha disambiguation and
|
||||
one wording clarification). **No training was run, no aggregation/model source changed, no
|
||||
test-based selection occurred.**
|
||||
|
||||
Audit performed: 2026-09-26. Audited HEAD: `6b97a0d38b839c737c49f41006817503d0672871`.
|
||||
|
||||
Companion documents: `docs/research/ALPHA025_BETA_VALIDATION.md`,
|
||||
`docs/research/MULTISEED_BETA_VALIDATION.md`, `docs/research/SESSION6_AUDIT.md`,
|
||||
`docs/research/EXPERIMENT_PROTOCOL.md`, `docs/research/PAIRED_BETA_VALIDATION.md`,
|
||||
`docs/handoff/SESSION_HANDOFF.md`.
|
||||
|
||||
---
|
||||
|
||||
## A. Scope
|
||||
|
||||
Independently verify the Session 7 claim for:
|
||||
|
||||
- dataset `xiiotid`, alpha `0.25`, seeds `{0,1,2}`, beta `{0, 0.25, 0.5, 0.75, 1.0}` plus
|
||||
`original_protean` (18 runs);
|
||||
- PRIMARY pooled-validation selection `beta_star = 0.00`, using validation per-client
|
||||
rare-class macro-F1 averaged within seed then across seeds.
|
||||
|
||||
The audit recomputed metrics from run artifacts and re-derived the selection; it did not repeat
|
||||
the report, and it ran no training. No alpha=0.75 or 5G-NIDD run was launched.
|
||||
|
||||
---
|
||||
|
||||
## B. Run inventory
|
||||
|
||||
18 full-data runs were enumerated from `results/*/metadata.json` (15 `support_weighted` + 3
|
||||
`original_protean`). Every expected run is present, none duplicated, none partial, none
|
||||
unexpected.
|
||||
|
||||
| seed | runs present | metadata fields | manifest reload | smoke | missing artifacts |
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | original + beta {0,.25,.5,.75,1} | all correct | true | false | none |
|
||||
| 1 | original + beta {0,.25,.5,.75,1} | all correct | true | false | none |
|
||||
| 2 | original + beta {0,.25,.5,.75,1} | all correct | true | false | none |
|
||||
|
||||
Verified per run: dataset=xiiotid, alpha=0.25, seed, aggregation mode, beta, clients=10,
|
||||
rounds=10, local_epochs=3, batch=256, lr=1e-3, mu=0.1, lambda=1.0, `manifest_reload_ok=true`,
|
||||
`smoke=false`; every run has all 14 expected artifacts. No run exists for any alpha ≠ 0.25
|
||||
(alpha=0.50 runs are the Session-6 condition) and no `5gnidd...support_weighted` run exists
|
||||
(only `5gnidd__alpha0.5__seed0__original_protean`).
|
||||
|
||||
---
|
||||
|
||||
## C. Pairing / manifest integrity
|
||||
|
||||
Within each seed, all six runs share one manifest content hash and the deterministic artifacts
|
||||
are byte-identical (SHA-256 computed by this audit):
|
||||
|
||||
| seed | manifest content hash | round-0 snapshot SHA-256 (all 6 runs) | support / characterization / rare_unseen |
|
||||
|---|---|---|---|
|
||||
| 0 | `839ba18e2d87245d198be8f5f89d4a24f13cdb6fc64c004d406c6d4f8246f3da` | `cd821beb8684ed762934a00b9f541bdfae5a4498f061ffe07132ad9d00ad300c` | identical |
|
||||
| 1 | `84c20b4bd6cd20010fa35d139a0416cb06e79c82fbd8e1d57441cafdcdfbf15d` | `59d2d2bc1a86a858d061d08219ee84669177ce8346e388fc8eada299e6fb5cf5` | identical |
|
||||
| 2 | `d6e68ee13cb11f98cfb4aea668f3e905d3ebc94884632352c0045fc5f86df28c` | `47befefff4cfa84250d1b6783fe00ce98dcea83b0649efe9745fa70bd4f3fa33` | identical |
|
||||
|
||||
The stored `train_support_Nij.csv` equals the manifest support matrix, and each manifest was
|
||||
independently reproduced from the raw dataset (`prepare_dataset` →
|
||||
`reproduce_train_assignments`): all three seeds return `reproduce_ok=True`, `support_match=True`,
|
||||
`split_hash_ok=True`, `partition_hash_ok=True`, `nij_sum = 569,909`, 10 clients, split sizes
|
||||
569,909 / 81,414 / 162,831 for both alpha=0.25 and alpha=0.50. `history.json` and
|
||||
`test_global_metrics.json` differ across runs of a seed, exactly as expected because beta
|
||||
changes training. Pairing is established by content hashes, not filenames.
|
||||
|
||||
---
|
||||
|
||||
## D. Partition characterization audit (realized)
|
||||
|
||||
Independently recomputed from `artifacts/partitions/xiiotid__alpha{0.25,0.5}__seed{0,1,2}.json`
|
||||
(`support` matrix, training split). `M = 10` clients; `|S_j| < M` means fewer than 10 clients
|
||||
contribute a class prototype; "missing cells" counts `(client, class)` pairs with `N_ij = 0`
|
||||
over attack+benign classes.
|
||||
|
||||
| statistic | alpha=0.25 seed0 / seed1 / seed2 | alpha=0.50 seed0 / seed1 / seed2 |
|
||||
|---|---|---|
|
||||
| class-condition cells with `|S_j| < M` | 6 / 7 / 6 (sum **19/27**) | 3 / 3 / 1 (sum **7/27**) |
|
||||
| missing client-class cells | 13 / 10 / 15 (sum **38**) | 4 / 4 / 1 (sum **9**) |
|
||||
| client-size max/min | 15.58 / 20.84 / 7.82 | 5.11 / 7.22 / 9.89 |
|
||||
| union unseen classes | 6 / 6 / 5 | 3 / 3 / 1 |
|
||||
| max unseen on one client | 3 / 4 / 3 | 2 / 2 / 1 |
|
||||
| union rare classes | 8 / 7 / 8 | 6 / 7 / 7 |
|
||||
| rare classes per client | 2 × 10 | 2 × 10 |
|
||||
|
||||
These match the report claims exactly: alpha=0.25 = **19/27 cells**, **38 missing cells**,
|
||||
client-size ratios **15.6 / 20.8 / 7.8**, unseen union/count **6/6/5** (max 4). The comparison
|
||||
against alpha=0.50 is confirmed (7/27, 9, 5.1/7.2/9.9, 3/3/1, max 2). Per-class `N_ij`, CV and
|
||||
max/min recomputed from the manifests equal `characterization.csv` for every seed and class.
|
||||
Interpretation uses these realized statistics, not alpha alone; note the non-uniform case
|
||||
(alpha=0.25 seed 2 ratio 7.82 < alpha=0.50 seed 2 ratio 9.89).
|
||||
|
||||
---
|
||||
|
||||
## E. Validation metric recomputation
|
||||
|
||||
Recomputed from raw artifacts (no report numbers copied):
|
||||
|
||||
- seed-level primary score = mean over eligible clients of `val_per_client.csv`
|
||||
`rare_class_macro_f1`; **equals `val_selection.json` per-client-mean to ≤ 1.1e-16**;
|
||||
- global validation accuracy / macro-F1 / weighted-F1 / rare macro-F1 / rare recall
|
||||
recomputed **independently from `val_confusion_matrix.json`** using the production formulas;
|
||||
**equal to `val_global_metrics.json` exactly (diff 0.0)**.
|
||||
|
||||
| seed | beta | eligible | recomputed seed mean | report claim |
|
||||
|---|---|---|---|---|
|
||||
| 0 | 0.00 / 0.25 / 0.50 / 0.75 / 1.00 | 10 each | 0.616380 / 0.609913 / 0.595247 / 0.654033 / 0.578385 | same |
|
||||
| 1 | 0.00 / 0.25 / 0.50 / 0.75 / 1.00 | 10 each | 0.492425 / 0.514237 / 0.537040 / 0.535672 / 0.530942 | same |
|
||||
| 2 | 0.00 / 0.25 / 0.50 / 0.75 / 1.00 | 10 each | 0.666307 / 0.641922 / 0.625267 / 0.548685 / 0.486970 | same |
|
||||
|
||||
All recomputed values agree with the report to six-decimal formatting precision; no mismatch
|
||||
beyond precision was found. Per-seed diagnostic argmax reproduces **0.75 / 0.50 / 0.00**.
|
||||
|
||||
Limitation (unavoidable, documented): the *per-client predictions* cannot be recomputed because
|
||||
the trained model is not persisted and retraining is forbidden by the audit. The per-client
|
||||
metric is verified via (a) its defining source code
|
||||
(`evaluation/metrics.py::rare_class_metrics` over each client's training-derived rare ids),
|
||||
(b) exact equality of its mean with `val_selection.json`, and (c) exact equality of the global
|
||||
union-rare metric recomputed from the confusion matrix. This does not affect `beta_star`, which
|
||||
depends only on the seed-level means.
|
||||
|
||||
---
|
||||
|
||||
## F. Rare / unseen integrity
|
||||
|
||||
Per-client rare classes recomputed from training `N_ij` (two smallest positive-support attack
|
||||
classes, tie by class id) and compared to `rare_unseen.csv` flags across **all 18 runs**:
|
||||
|
||||
- **0 mismatches** (flag mismatches counted over every `(seed, mode, beta, client, class)` cell).
|
||||
- Benign class (`Normal`) never flagged rare/unseen; rare and unseen never overlap.
|
||||
- Unseen = attack class with `N_ij = 0`; union of training-derived rare classes equals
|
||||
`val_global_metrics.evaluated_rare_classes` **and**
|
||||
`test_global_metrics.evaluated_rare_classes` for every run — i.e. validation/test labels did
|
||||
**not** influence rare-class identity.
|
||||
- Eligible clients = **10 in every seed and run**; every client has exactly 2 rare classes; no
|
||||
client has only 1 attack class.
|
||||
- Union rare classes: seed 0 = `[0,1,2,3,5,6,7,8]`, seed 1 = `[0,1,2,3,6,7,8]`,
|
||||
seed 2 = `[0,1,2,3,5,6,7,8]` (class ids; `Normal` is benign).
|
||||
|
||||
---
|
||||
|
||||
## G. Pooled-selection recomputation
|
||||
|
||||
Computed from the independently recomputed seed-level scores (not from the report). Pooled
|
||||
score = mean of the three seed-level per-client means; std is sample (`ddof=1`).
|
||||
|
||||
| beta | pooled mean | sample std | pop std | min | max |
|
||||
|---|---|---|---|---|---|
|
||||
| **0.00** | **0.591703815** | **0.089528837** | 0.073099989 | 0.492424736 | 0.666306547 |
|
||||
| 0.25 | 0.588690626 | 0.066435256 | 0.054244160 | 0.514237103 | 0.641922010 |
|
||||
| 0.50 | 0.585851480 | 0.044857850 | 0.036626281 | 0.537039860 | 0.625267154 |
|
||||
| 0.75 | 0.579463399 | 0.064906057 | 0.052995574 | 0.535672367 | 0.654032964 |
|
||||
| 1.00 | 0.532098701 | 0.045718339 | 0.037328867 | 0.486969813 | 0.578384529 |
|
||||
|
||||
- Recompute **matches the report** (mean and sample std) to ≤ 4e-7 (formatting precision).
|
||||
- `argmax` of the pooled validation score = **0.00** (no tie; `tie_break_used=false`).
|
||||
- The pooled validation curve is **numerically monotonically decreasing** in beta:
|
||||
0.591704 > 0.588691 > 0.585851 > 0.579463 > 0.532099.
|
||||
- The secondary pooled validation rare-class recall peaks at beta=0.25, but recall is only a
|
||||
tie-break and is not reached; the primary metric governs.
|
||||
- **Std convention:** pooled std is sample (`ddof=1`), explicitly labelled `std (sample)` in
|
||||
the report; `std_pop` is also emitted. Consistent with Session 6; no convention changed.
|
||||
|
||||
---
|
||||
|
||||
## H. Validation / test leakage audit
|
||||
|
||||
- `run_baseline.py`: `union_rare` is derived from the **training** support matrix only
|
||||
(lines 189–190). The `val_selection` payload (lines 297–313) is built **only** from
|
||||
`val_rare`, `val_metrics`, `val_per_client_rows`. Test variables appear in a separate block
|
||||
and feed only test artifacts. `test_used_for_selection: false` and
|
||||
`selection_split: "validation"` are recorded for all 15 `support_weighted` runs.
|
||||
- `analyze_multiseed_beta.py`: the PRIMARY selection block (lines 176–208) consumes only
|
||||
`per_seed_rows`, which are built from `val_selection.json`. Tie-break uses pooled validation
|
||||
recall then validation std then smaller beta. `test_rows` are read in a later block (line 348+)
|
||||
and are never referenced by the selection logic or tie-break.
|
||||
- No code path sorts/filters candidates by a test field.
|
||||
|
||||
Conclusion: selection is **programmatic and validation-only**; the boolean flag is corroborated
|
||||
by the independent recomputation of `beta_star` from validation artifacts alone.
|
||||
`test_used_for_selection = false` is upheld.
|
||||
|
||||
---
|
||||
|
||||
## I. Final test recomputation (selected beta and original)
|
||||
|
||||
`beta_star = 0.00` was fixed from pooled validation; only then were test metrics read.
|
||||
Test metrics recomputed independently from `test_confusion_matrix.json` (diff vs the JSON = 0.0),
|
||||
then averaged across seeds with sample std:
|
||||
|
||||
| metric | recomputed mean ± std | report claim |
|
||||
|---|---|---|
|
||||
| rare-class macro-F1 | 0.632136 ± 0.047642 | 0.632136 ± 0.047642 |
|
||||
| rare-class recall | 0.821257 ± 0.012283 | 0.821257 ± 0.012283 |
|
||||
| macro-F1 | 0.681343 ± 0.020555 | 0.681343 ± 0.020555 |
|
||||
| accuracy | 0.874015 ± 0.010267 | 0.874015 ± 0.010267 |
|
||||
| weighted-F1 | 0.906882 ± 0.007408 | 0.906882 ± 0.007408 |
|
||||
|
||||
`original_protean` recomputed identically: rare macro-F1 0.632790 ± 0.047152; rare recall
|
||||
0.821965 ± 0.010811; macro-F1 0.682018 ± 0.020967; accuracy 0.875431 ± 0.009067;
|
||||
weighted-F1 0.907652 ± 0.006934 — all match the report. No superiority claim is made.
|
||||
|
||||
---
|
||||
|
||||
## J. beta = 0 equivalence audit
|
||||
|
||||
For seeds 0, 1 and 2, `original_protean` vs `support_weighted beta=0`:
|
||||
|
||||
| property | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| same manifest / split / hashes | yes | yes | yes |
|
||||
| round-0 snapshot byte-identical | yes (`cd821beb…`) | yes (`59d2d2bc…`) | yes (`47befeff…`) |
|
||||
| support/characterization/rare_unseen identical | yes | yes | yes |
|
||||
| independent aggregation diff (`ΣC/n` vs `(1/n)ΣC`) | 4.4e-16 | 4.4e-16 | 2.2e-16 |
|
||||
| history rare-macro-F1 round-0 diff | 0.0 | 0.0 | 0.0 |
|
||||
| history rare-macro-F1 max diff (rounds ≥ 1) | 6.6e-3 | 3.6e-3 | 3.7e-3 |
|
||||
| test metric max diff | 1.6e-3 | 3.9e-3 | 2.2e-3 |
|
||||
| classification | **B** | **B** | **B** |
|
||||
|
||||
Class **B** (semantically equivalent + floating-point numerical divergence); **no semantic
|
||||
discrepancy** in any seed. Round 0 is byte-identical; the ~1 ULP ordering difference enters the
|
||||
round-1 prototype-alignment training and trajectories diverge slowly. Equivalence remains
|
||||
conditional on the shared `S_j` reproduction assumption and is **not** proof that `beta = 0`
|
||||
reproduces original PROTEAN. Machine-readable:
|
||||
`artifacts/analysis/xiiotid__alpha0.25__multiseed__beta0_equivalence.csv`.
|
||||
|
||||
---
|
||||
|
||||
## K. Direct-effect audit
|
||||
|
||||
Recomputed independently from the fixed round-0 snapshots using the production aggregation
|
||||
(`direct_prototype.prototype_effect_rows`). `||Cbar_j(1) − Cbar_j(0)||`, cross-seed mean:
|
||||
|
||||
| class | alpha=0.25 mean (std) | alpha=0.50 mean (std) |
|
||||
|---|---|---|
|
||||
| Exfiltration | 4.316 (0.898) | 2.020 (0.502) |
|
||||
| Tampering | 4.248 (0.241) | 2.354 (1.165) |
|
||||
| C&C | 4.020 (0.290) | 1.694 (0.505) |
|
||||
| Lateral_movement | 3.726 (1.383) | 2.724 (1.035) |
|
||||
| Exploitation | 3.464 (1.113) | 2.344 (0.412) |
|
||||
| Reconnaissance | 3.339 (1.107) | 1.002 (0.261) |
|
||||
| Weaponization | 3.124 (0.585) | 2.561 (1.228) |
|
||||
| RDOS | 2.566 (1.219) | 2.631 (1.165) |
|
||||
| Normal | 1.997 (0.533) | 1.557 (0.548) |
|
||||
|
||||
- Mean across classes: alpha=0.25 **3.422**, alpha=0.50 **2.099** (report: 3.42 / 2.10).
|
||||
- alpha=0.25 largest classes: Exfiltration 4.316, Tampering 4.248, C&C 4.020; smallest:
|
||||
Normal 1.997, RDOS 2.566 — all match the report.
|
||||
- Per-`(class, beta)` displacement and normalised weights recomputed from snapshots match the
|
||||
per-seed `...__prototype_effect.csv` and pooled `...__prototype_effect_pooled.csv` **exactly**.
|
||||
- Displacement is a **direct-aggregation measure only**; no detection causality is inferred.
|
||||
The end-to-end effect is reported separately and is not used to interpret displacement.
|
||||
|
||||
---
|
||||
|
||||
## L. Client-level audit
|
||||
|
||||
Recomputed from `val_per_client.csv` deltas vs `beta = 0` (independent of the summary CSV):
|
||||
|
||||
| seed | beta=0.75 improve/degrade/unchanged | mean Δ | min Δ | max Δ |
|
||||
|---|---|---|---|---|
|
||||
| 0 | **5 / 4 / 1** | +0.0377 | −0.1117 | +0.3759 |
|
||||
| 1 | **5 / 4 / 1** | +0.0432 | −0.0183 | +0.3167 |
|
||||
| 2 | **3 / 5 / 2** | −0.1176 | −0.6011 | +0.1241 |
|
||||
|
||||
- At beta=1.00 seed 2 degrades 6/10 clients with a worst client Δ = **−0.6775** (≈ −0.678),
|
||||
while seed 1 still gains on average — all confirmed.
|
||||
- `client_validation_detail.csv` deltas match recomputation with max abs diff **0.0**.
|
||||
- `min positive attack N_ij` vs ΔF1 is confirmed **non-monotonic** at beta=0.75 (e.g. seed 2:
|
||||
N=1 → −0.601, N=2 → 0.000, N=4 → +0.124; seed 0: N=2 → −0.029 and +0.002). Client ids are
|
||||
per-seed realisations and are not pooled across seeds.
|
||||
|
||||
---
|
||||
|
||||
## M. Cross-alpha audit (critical)
|
||||
|
||||
Use only audited condition-level pooled results.
|
||||
|
||||
| | alpha=0.50 | alpha=0.25 |
|
||||
|---|---|---|
|
||||
| **pooled-selected beta** | **0.75** | **0.00** |
|
||||
| pooled validation curve | rises to a peak at 0.75 | numerically monotone decreasing |
|
||||
| pooled mean range | 0.544361–0.579367 (0.035) | 0.532099–0.591704 (0.060) |
|
||||
| pooled sample std range | 0.075121–0.100450 | 0.044858–0.089529 |
|
||||
| `|S_j| < M` cells | 7/27 | 19/27 |
|
||||
| missing client-class cells | 9 | 38 |
|
||||
| union unseen per seed | 3/3/1 (max 2) | 6/6/5 (max 4) |
|
||||
| mean `||Cbar(1)−Cbar(0)||` | 2.099 | 3.422 |
|
||||
| beta=0.75 client pattern | majority improve every seed | seed-dependent, often negative |
|
||||
|
||||
Alpha=0.50 pooled selection was independently reproduced from Session-6 artifacts
|
||||
(`results/xiiotid__alpha0.5__seed{0,1,2}__support_weighted__beta{0,.25,.5,.75,1}`):
|
||||
pooled means 0.544361 / 0.555434 / 0.567322 / **0.579367** / 0.573479; `argmax = 0.75`.
|
||||
|
||||
**Reporting-confusion check.** The audit searched the entire repository (docs, research notes,
|
||||
handoff) for any statement of the form "selected beta changes (0.50 -> 0.00)" or equivalent
|
||||
conflating the two conditions. **No such statement exists in the repository.** Every written
|
||||
occurrence is correct: alpha=0.50 pooled `beta_star = 0.75`; alpha=0.25 pooled
|
||||
`beta_star = 0.00`. The value `0.50` appears only as (a) the Session-5 **seed-0 single-condition**
|
||||
selection for alpha=0.50 and (b) the Session-6 **per-seed diagnostic** for alpha=0.50 seed 0.
|
||||
It is **not** the condition-level pooled selection for alpha=0.50. This disambiguation has been
|
||||
added explicitly to the report (§L) and handoff.
|
||||
|
||||
These are descriptive associations; the realized partition characteristics (not alpha alone) are
|
||||
reported alongside, and **no causal claim is made that alpha by itself caused the difference**.
|
||||
The beta effect remains small relative to between-seed variance at both settings.
|
||||
|
||||
---
|
||||
|
||||
## N. Artifact provenance
|
||||
|
||||
Every reported quantity traces to a machine-readable artifact or a deterministic recomputation.
|
||||
|
||||
| Reported quantity | Source artifact(s) | Derivation | Verified |
|
||||
|---|---|---|---|
|
||||
| run existence / config | `results/<run>/metadata.json` | direct read | yes |
|
||||
| manifest content/partition/split hashes | `artifacts/partitions/xiiotid__alpha0.25__seed{0,1,2}.json` | module hash + `reproduce_train_assignments` | yes |
|
||||
| pairing proof | `artifacts/analysis/...__prototype_snapshot.json` | sha256 across 6 runs/seed | yes |
|
||||
| seed-level validation score | `results/.../val_per_client.csv` + `val_selection.json` | mean of per-client rare macro-F1 | yes |
|
||||
| validation global metrics | `results/.../val_confusion_matrix.json` | independent numpy recompute | yes |
|
||||
| pooled mean/std, `beta_star` | recomputed seed scores | mean/`stdev` over seeds, argmax | yes |
|
||||
| test metrics (beta 0, original) | `results/.../test_confusion_matrix.json` | independent recompute + mean/std | yes |
|
||||
| test descriptive (all candidates) | `artifacts/analysis/...__test_descriptive.csv` | direct read | yes |
|
||||
| beta=0 equivalence | original vs beta0 artifacts/snapshots | sha256 + independent aggregation + history diff | yes |
|
||||
| direct-effect displacement | snapshots + `...__prototype_effect_pooled.csv` | independent `direct_prototype` recompute | yes |
|
||||
| client improve/degrade | `results/.../val_per_client.csv` + `...__client_validation_{summary,detail}.csv` | deltas vs beta0 | yes |
|
||||
| rare-class identity | `artifacts/analysis/...__rare_unseen.csv` + manifest | recompute definition | yes |
|
||||
| partition characterization | manifests + `results/.../characterization.csv` | independent numpy recompute | yes |
|
||||
| runtime / RSS | `artifacts/analysis/xiiotid__alpha0.25__multiseed__runtime.csv` (source `/tmp/opencode/s7_logs/*.time`, present) | parse logs; sum = 43,799.03 s = 12.166 h | yes |
|
||||
|
||||
No scientific number was found to exist only in Markdown without a machine-readable or
|
||||
deterministic source.
|
||||
|
||||
---
|
||||
|
||||
## O. Deviations / corrections found
|
||||
|
||||
1. **Cross-alpha label disambiguation (documentation-only).** The reported external phrasing
|
||||
"Selected beta changes (0.50 -> 0.00)" is **not present anywhere in the repository**; the
|
||||
repo already states alpha=0.50 pooled = 0.75 and alpha=0.25 pooled = 0.00. An explicit
|
||||
disambiguation note was added to `docs/research/ALPHA025_BETA_VALIDATION.md` §L and
|
||||
`docs/handoff/SESSION_HANDOFF.md` §11 so the Session-5 seed-0 `0.50` and Session-6 pooled
|
||||
`0.75` cannot be conflated. No scientific value changed.
|
||||
2. **Wording nuance (clarified).** Handoff §14 item 6 ("the beta direction may reverse under
|
||||
stronger heterogeneity … monotone in heterogeneity") is a speculation, not a settled trend;
|
||||
reworded to a neutral "one more condition; not yet a trend across alpha" framing.
|
||||
3. **No arithmetic, metric-label, std-convention, or provenance defect was found** in the
|
||||
Session 7 report or analysis artifacts. No derived CSV required correction.
|
||||
|
||||
No training outputs, aggregation, model, manifests, or scientific protocol were altered.
|
||||
|
||||
---
|
||||
|
||||
## P. Final audit verdict
|
||||
|
||||
**PASS WITH MINOR CORRECTIONS.**
|
||||
|
||||
- Run inventory: **PASS** (18/18, no duplicates/partials; no alpha=0.75 or 5G-NIDD run).
|
||||
- Pairing / manifest reproduction: **PASS** (hash-identical; raw-data reproduction exact).
|
||||
- Partition characterization: **PASS** (19/27, 38, 15.6/20.8/7.8, 6/6/5; alpha=0.50 7/27, 9).
|
||||
- Validation recomputation (seed-level and global): **PASS** (exact).
|
||||
- Rare/unseen integrity: **PASS** (0 mismatches; training-derived; no val/test influence).
|
||||
- Pooled selection: **PASS**; independent `beta_star = 0.00` using validation only.
|
||||
- Validation/test leakage: **PASS**; selection reads validation only and is reproducible.
|
||||
- Test summaries (selected beta and `original_protean`): **PASS** (exact).
|
||||
- beta=0 equivalence: **PASS**, class **B** (semantically equivalent + float divergence).
|
||||
- Direct-effect and client-level analyses: **PASS** (independent recompute exact).
|
||||
- Cross-alpha comparison: **PASS**; alpha=0.50 pooled = 0.75, alpha=0.25 pooled = 0.00.
|
||||
- Source/data integrity, pytest (75 passed, 0 failed, 0 skipped, 0 warnings): **PASS**.
|
||||
|
||||
The corrections are documentation/wording only and do not affect `beta_star` or the scientific
|
||||
interpretation. Session 7 is **safe to checkpoint** with the documented limitations (single
|
||||
dataset, two alphas, three project-defined seeds, weak effect relative to seed variance, `S_j`
|
||||
reproduction assumption, no global-optimum claim).
|
||||
Reference in new issue
Block a user