experiment: validate 5gnidd beta selection at alpha 0.50
This commit is contained in:
willtanoe committed
1 parent
2d27abc350
commit
a7f2b0049e
28 files changed
+38378
-256
No files matched your search
@@ -0,0 +1,495 @@
|
||||
# 5G-NIDD Alpha-0.50 Beta Validation (Session 9, Stage B)
|
||||
|
||||
Status: **first cross-dataset beta-validation condition**. This report covers a single 5G-NIDD
|
||||
condition — `alpha = 0.50`, seeds `{0,1,2}`, the frozen `original_protean` baseline plus the five
|
||||
`support_weighted` beta candidates — using the same pooled-across-seed validation-only selection
|
||||
rule already applied to X-IIoTID. It is **not** a sweep, not a hyper-parameter search, and not
|
||||
evidence of a global optimum. No other 5G-NIDD alpha (0.25 / 0.75) was run.
|
||||
|
||||
Companion documents:
|
||||
|
||||
- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen protocol and selection hierarchy.
|
||||
- `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md` — X-IIoTID three-condition reference.
|
||||
- `docs/research/MULTISEED_BETA_VALIDATION.md` — X-IIoTID alpha=0.50 (the matched condition).
|
||||
- `docs/research/SESSION9_AUDIT.md` — independent audit of this condition.
|
||||
- `docs/handoff/SESSION_HANDOFF.md` — continuation point.
|
||||
|
||||
---
|
||||
|
||||
## A. Scope
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| dataset | `5gnidd` (5G-NIDD) |
|
||||
| alpha | 0.50 |
|
||||
| seeds | 0, 1, 2 |
|
||||
| aggregation | `original_protean` (per seed) + `support_weighted` for `beta ∈ {0, 0.25, 0.50, 0.75, 1.00}` |
|
||||
| runs | 18: 15 `support_weighted` + 3 `original_protean`, **all run new this session** |
|
||||
| compute | sequential CPU, one run at a time, 69,336 s ≈ **19.26 h** wall-clock |
|
||||
| selection | pooled validation-only across seeds `{0,1,2}` |
|
||||
| test | read only after the pooled beta was fixed |
|
||||
|
||||
No 5G-NIDD `alpha=0.25` or `alpha=0.75` run was performed; the full grid was deliberately not
|
||||
launched. `equal_available` was not added. No training/aggregation/preprocessing source code was
|
||||
changed.
|
||||
|
||||
---
|
||||
|
||||
## B. Dataset and manifests
|
||||
|
||||
### B.1 Raw-data provenance (Stage A)
|
||||
|
||||
- Path `~/datasets/5G-NIDD/5G-NIDD.csv`; SHA-256
|
||||
`fa36f80859585f474504ca69eb951a079b35145537976c86b00aae3aab46ee59` — matches the audited value.
|
||||
- Raw shape 1,215,890 rows × 52 columns; **1** exact duplicate row dropped → 1,215,889; **0**
|
||||
label rows dropped (`drop_labels` empty). Missing tokens: empty string only.
|
||||
- **Label semantics (canonicalised):** label column `Attack Type`; **9 classes** — benign
|
||||
`Benign` (index 0) plus **8 attack classes** `HTTPFlood, ICMPFlood, SYNFlood, SYNScan,
|
||||
SlowrateDoS, TCPConnectScan, UDPFlood, UDPScan`.
|
||||
|
||||
| class | full-dataset count |
|
||||
|---|---|
|
||||
| Benign | 477,736 |
|
||||
| UDPFlood | 457,340 |
|
||||
| HTTPFlood | 140,812 |
|
||||
| SlowrateDoS | 73,124 |
|
||||
| TCPConnectScan | 20,052 |
|
||||
| SYNScan | 20,043 |
|
||||
| UDPScan | 15,906 |
|
||||
| SYNFlood | 9,721 |
|
||||
| ICMPFlood | 1,155 |
|
||||
|
||||
- **46 feature columns** after excluding identifiers/targets: 41 numeric (median-imputed and
|
||||
train-standardized) and 5 categorical one-hot (`Proto, sDSb, dDSb, Cause, State`), with an
|
||||
explicit `__MISSING__` category. Preprocessing is spec-driven, fits on the train split only,
|
||||
and contains **no X-IIoTID-specific assumptions**.
|
||||
- Rare/unseen logic verified against this class list: attack classes only, `N_ij > 0`, two
|
||||
smallest positive training supports per client, `N_ij = 0` is unseen (never rare), benign never
|
||||
rare/unseen. All 10 clients are eligible in all three seeds (each has exactly two rare classes).
|
||||
|
||||
### B.2 Manifest identity (reused, never regenerated)
|
||||
|
||||
Train/validation/test sizes are 851,123 / 121,589 / 243,177 for every seed (10 clients, 9
|
||||
classes, `Benign`). Each manifest was independently reproduced from the raw dataset during Stage A
|
||||
(`prepare_dataset` → `reproduce_train_assignments`): exact `N_ij` match and partition-hash match
|
||||
for all three seeds (and, as a side check, for alpha=0.25/0.75).
|
||||
|
||||
| seed | manifest | content hash | partition hash | split hash | rng seed |
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | `artifacts/partitions/5gnidd__alpha0.5__seed0.json` | `0732582c…` | `8352d4ff…` | `18f89f48…` | 673621352 |
|
||||
| 1 | `artifacts/partitions/5gnidd__alpha0.5__seed1.json` | `f43906e1…` | `0815c91e…` | `3d05b7fe…` | 1403944623 |
|
||||
| 2 | `artifacts/partitions/5gnidd__alpha0.5__seed2.json` | `fe8b4fc6…` | `fdcf388e…` | `fc758d06…` | 633461056 |
|
||||
|
||||
`preprocessing_hash` / `config_hash` remain `null` in these Session-2 manifests (known limitation,
|
||||
carry-over from X-IIoTID).
|
||||
|
||||
### B.3 Pairing (from content hashes, not filenames)
|
||||
|
||||
Within each seed all six runs share one manifest content hash and a **byte-identical round-0
|
||||
snapshot**; `train_support_Nij.csv`, `characterization.csv` and `rare_unseen.csv` are identical
|
||||
across the six runs; `manifest_reload_ok = true` and `smoke = false` everywhere.
|
||||
|
||||
| seed | manifest content hash | round-0 snapshot SHA-256 (all 6 runs) |
|
||||
|---|---|---|
|
||||
| 0 | `0732582cf4510089…` | `eb0839f25957274a6bc42532c07b89826cf66c581ac05d197a10333a21bc467f` |
|
||||
| 1 | `f43906e111a8342c…` | `6cbd04e0dd97bbe0dad733f654962a115cef54e126ab2c4e58d853ac0b249afb` |
|
||||
| 2 | `fe8b4fc6f0486cf4…` | `f012a02b543b926a9df31134e06bf442ed832b35cdeb9aba1f20d404e8b93687` |
|
||||
|
||||
---
|
||||
|
||||
## C. Structural characterization (realized, before interpretation)
|
||||
|
||||
All statistics are from the training-split `N_ij`; `M = 10` clients; `|S_j| < M` means fewer than
|
||||
10 clients contribute a prototype for a class.
|
||||
|
||||
### C.1 Per-seed summary
|
||||
|
||||
| seed | `|S_j| < M` cells | missing client-class cells | client-size min/max (ratio) | union rare | union unseen | max unseen on a client |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0 | 2 / 9 | 2 | 15,388 / 178,993 (11.63) | 6 | 2 | 1 |
|
||||
| 1 | 0 / 9 | 0 | 31,613 / 140,144 (4.43) | 5 | 0 | 0 |
|
||||
| 2 | 2 / 9 | 2 | 20,087 / 255,463 (12.72) | 6 | 2 | 1 |
|
||||
| **total / mean** | **4 / 27** | **4** | ratio mean **9.59** | 6/5/6 | 2/0/2 | 1 |
|
||||
|
||||
All 10 clients have exactly two rare classes in every seed, so all 10 are eligible for rare-class
|
||||
evaluation. Missing cells are only ever single-client gaps in the two smallest attack classes
|
||||
(e.g. seed 0: `ICMPFlood` absent on client 3, `SYNFlood` absent on client 4). Rare classes are
|
||||
dominated by `ICMPFlood` (always rare) and one of `SYNFlood`/`TCPConnectScan`/`UDPScan`/
|
||||
`HTTPFlood`/`SYNScan`.
|
||||
|
||||
### C.2 Per-class `N_ij` inequality (training split)
|
||||
|
||||
`CV` is the population coefficient of variation of the positive supports within a class;
|
||||
`max/min` is over contributing clients only. Machine-readable:
|
||||
`artifacts/analysis/5gnidd__alpha0.5__partition_heterogeneity_per_class.csv`.
|
||||
|
||||
**Seed 0**
|
||||
|
||||
| class | total | #clients | min+ | max | CV | max/min |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Benign | 334,415 | 10 | 226 | 108,257 | 1.069 | 479 |
|
||||
| HTTPFlood | 98,569 | 10 | 185 | 61,393 | 1.779 | 332 |
|
||||
| ICMPFlood | 808 | 9 | 8 | 243 | 0.809 | 30.4 |
|
||||
| SYNFlood | 6,805 | 9 | 2 | 3,250 | 1.318 | 1,625 |
|
||||
| SYNScan | 14,030 | 10 | 11 | 6,713 | 1.414 | 610 |
|
||||
| SlowrateDoS | 51,187 | 10 | 80 | 28,363 | 1.783 | 355 |
|
||||
| TCPConnectScan | 14,037 | 10 | 1 | 4,175 | 0.982 | 4,175 |
|
||||
| UDPFlood | 320,138 | 10 | 1 | 88,943 | 0.896 | 88,943 |
|
||||
| UDPScan | 11,134 | 10 | 14 | 3,420 | 0.966 | 244 |
|
||||
|
||||
Seeds 1 and 2 are in the machine-readable file; the notable pattern is that even though the
|
||||
number of *missing* cells is tiny, the per-class support is strongly imbalanced for the rare
|
||||
classes (`TCPConnectScan`, `SYNFlood`, `UDPFlood` max/min in the hundreds–thousands). The mean
|
||||
class-support CV across seeds is **1.225** (1.224 / 1.051 / 1.401).
|
||||
|
||||
### C.3 Rare/unseen identity
|
||||
|
||||
Per-seed union rare classes (ids in brackets): seed 0 = `{ICMPFlood, SYNFlood, SYNScan,
|
||||
TCPConnectScan, UDPFlood, UDPScan}`; seed 1 = `{ICMPFlood, SYNFlood, SYNScan, TCPConnectScan,
|
||||
UDPScan}`; seed 2 = `{ICMPFlood, HTTPFlood, SYNFlood, SYNScan, UDPFlood, UDPScan}`. Unseen unions:
|
||||
seed 0 = `{ICMPFlood, SYNFlood}` (one client each), seed 1 = `{}`, seed 2 =
|
||||
`{TCPConnectScan, UDPScan}`. Full per-client detail:
|
||||
`artifacts/analysis/5gnidd__alpha0.5__partition_characterization.csv`.
|
||||
|
||||
---
|
||||
|
||||
## D. Existing seed-0 baseline reuse decision
|
||||
|
||||
A pre-existing artifact `results/5gnidd__alpha0.5__seed0__original_protean` (Session-5 era) was
|
||||
inspected before training and **was not reused**. It contains the frozen training outputs
|
||||
(`history.json`, `test_*`, `per_client.csv`, `characterization.csv`, `rare_unseen.csv`,
|
||||
`train_support_Nij.csv`, `metadata.json`) and a round-0 snapshot, but **no validation artifacts**
|
||||
(`val_selection.json`, `val_per_client.csv`, `val_global_metrics.json`, `val_confusion_matrix.json`,
|
||||
`val_per_class.csv`). Under the frozen protocol the pooled selection metric requires
|
||||
`val_selection.json` / `val_per_client.csv`, and pairing/`beta=0` equivalence require re-runnable
|
||||
validation artifacts; the old directory therefore fails the "complete artifacts" reuse criterion.
|
||||
|
||||
The directory was backed up to `/tmp/opencode/s9/old_baseline_backup/` and **all 18 runs were
|
||||
re-executed**. As a determinism check, the fresh seed-0 `original_protean` run reproduces the old
|
||||
artifact **exactly**: `history.json` is byte-identical and every test metric is identical
|
||||
(e.g. rare-class macro-F1 `0.8719637151922778`, accuracy `0.7344156725348203`). The old baseline
|
||||
was therefore methodologically sound but structurally incomplete for the validation protocol.
|
||||
|
||||
---
|
||||
|
||||
## E. Frozen protocol
|
||||
|
||||
Identical to `EXPERIMENT_PROTOCOL.md` §14: 10 clients, 10 rounds, 3 local epochs, batch 256, Adam
|
||||
`lr=1e-3`, `mu=0.1`, `lambda=1.0`; unchanged architecture, preprocessing, split recipe, Dirichlet
|
||||
partition recipe (`per_class_dirichlet_multinomial`), loss and nearest-prototype inference. All 18
|
||||
runs used `scripts/run_baseline.py` at HEAD `2d27abc35026456de765c929dbcf0503a72ac572`.
|
||||
|
||||
Selection rule (unchanged):
|
||||
|
||||
- **PRIMARY:** pooled-across-seed validation per-client-mean rare-class macro-F1, validation only.
|
||||
- **SECONDARY (diagnostic):** per-seed validation selection.
|
||||
- **TEST:** read only after the pooled beta was fixed; `test_used_for_selection = false` for every
|
||||
`support_weighted` run.
|
||||
|
||||
---
|
||||
|
||||
## F. Per-seed validation results
|
||||
|
||||
`val rareMacroF1 (pc mean)` is the primary metric; all 10 clients eligible in every seed.
|
||||
Machine-readable: `artifacts/analysis/5gnidd__alpha0.5__multiseed__validation_per_seed.csv`.
|
||||
|
||||
### Seed 0
|
||||
|
||||
| beta | eligible | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | 10 | 0.738597 | 0.869471 | 0.971734 | 0.835222 | 0.735708 |
|
||||
| 0.25 | 10 | 0.661461 | 0.826591 | 0.971734 | 0.804446 | 0.733586 |
|
||||
| 0.50 | 10 | 0.764487 | 0.877274 | 0.971734 | 0.844008 | 0.738932 |
|
||||
| 0.75 | 10 | 0.750661 | 0.866155 | 0.971734 | 0.846984 | 0.747313 |
|
||||
| **1.00** | 10 | **0.768877** | 0.878253 | 0.971734 | 0.847921 | 0.743390 |
|
||||
|
||||
### Seed 1
|
||||
|
||||
| beta | eligible | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **0.00** | 10 | **0.633243** | 0.923637 | 0.970682 | 0.866358 | 0.746638 |
|
||||
| 0.25 | 10 | 0.573717 | 0.903669 | 0.970682 | 0.854134 | 0.745594 |
|
||||
| 0.50 | 10 | 0.563388 | 0.897248 | 0.970682 | 0.842769 | 0.739434 |
|
||||
| 0.75 | 10 | 0.543319 | 0.875983 | 0.970682 | 0.830737 | 0.739360 |
|
||||
| 1.00 | 10 | 0.531764 | 0.849605 | 0.970682 | 0.820473 | 0.743052 |
|
||||
|
||||
### Seed 2
|
||||
|
||||
| beta | eligible | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | 10 | 0.689849 | 0.837430 | 0.971518 | 0.817547 | 0.750010 |
|
||||
| 0.25 | 10 | 0.700055 | 0.841150 | 0.968867 | 0.817724 | 0.749459 |
|
||||
| 0.50 | 10 | 0.696131 | 0.838533 | 0.958948 | 0.813338 | 0.745158 |
|
||||
| 0.75 | 10 | 0.771124 | 0.884167 | 0.953065 | 0.841394 | 0.742304 |
|
||||
| **1.00** | 10 | **0.776468** | 0.887075 | 0.954616 | 0.843904 | 0.743431 |
|
||||
|
||||
The per-seed curves are **non-monotone and seed-dependent**: seed 0 rises to a peak at 1.00 with a
|
||||
sharp dip at 0.25; seed 1 is monotone decreasing (best at 0.00); seed 2 rises then is nearly flat
|
||||
at 0.75–1.00.
|
||||
|
||||
---
|
||||
|
||||
## G. Per-seed diagnostic beta choices (SECONDARY, stability only)
|
||||
|
||||
| seed | selected beta | val rareMacroF1 (pc mean) | tie-break used | test_used_for_selection |
|
||||
|---|---|---|---|---|
|
||||
| 0 | 1.00 | 0.768877 | no | false |
|
||||
| 1 | 0.00 | 0.633243 | no | false |
|
||||
| 2 | 1.00 | 0.776468 | no | false |
|
||||
|
||||
These per-seed choices are **diagnostic only**; they are not the condition-level selection rule.
|
||||
`val_selection.json` records `"selection_split": "validation"` and
|
||||
`"test_used_for_selection": false` for all 15 `support_weighted` runs.
|
||||
|
||||
---
|
||||
|
||||
## H. PRIMARY pooled-across-seed validation selection
|
||||
|
||||
| beta | pooled mean | std (sample, ddof=1) | min | max | pooled val rareRecall (pc mean) | selected |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | 0.687230 | 0.052726 | 0.633243 | 0.738597 | 0.947737 | |
|
||||
| 0.25 | 0.645078 | 0.064743 | 0.573717 | 0.700055 | 0.946862 | |
|
||||
| 0.50 | 0.674669 | 0.102253 | 0.563388 | 0.764487 | 0.943659 | |
|
||||
| 0.75 | 0.688368 | 0.126032 | 0.543319 | 0.771124 | 0.941915 | |
|
||||
| **1.00** | **0.692370** | 0.139140 | 0.531764 | 0.776468 | 0.942301 | **yes** |
|
||||
|
||||
- **PRIMARY selected beta (`beta_star_5gnidd_alpha050`) = 1.00.**
|
||||
- **Criterion:** maximum pooled validation per-client-mean rare-class macro-F1 (validation only).
|
||||
- **Tie-break used:** no. Top-two beta = 1.00 (0.692370) vs 0.75 (0.688368); gap ≈ **4.0e-3**.
|
||||
- **`test_used_for_selection = false`.** Test metrics were not inspected until the pooled beta was
|
||||
fixed.
|
||||
- **Interpretation:** `beta = 1.00` is the *best among the tested candidates under pooled
|
||||
validation across seeds `{0,1,2}` for 5G-NIDD alpha=0.50*. It is **not** a global optimum. The
|
||||
pooled validation curve is non-monotone (0.687 → 0.645 → 0.675 → 0.688 → 0.692), the total
|
||||
between-beta spread is ≈ **0.047**, and the across-seed std is **0.053–0.139** — comparable to or
|
||||
larger than the between-beta spread, so the selection is statistically weak. The pooled win is
|
||||
driven by seeds 0 and 2 (+0.030 and +0.087 at beta=1 vs beta=0) outweighing seed 1's large loss
|
||||
(−0.101), for a net +0.005 over beta=0.
|
||||
|
||||
Machine-readable: `artifacts/analysis/5gnidd__alpha0.5__multiseed__validation_pooled.csv`,
|
||||
`...__selection_summary.json`.
|
||||
|
||||
---
|
||||
|
||||
## I. Final selected-beta test metrics (descriptive only)
|
||||
|
||||
`beta_star = 1.00` was fixed from pooled validation; only then were test metrics read.
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.880962 | 0.840075 | 0.894914 | 0.871984 ± 0.028501 |
|
||||
| rare-class recall | 0.974355 | 0.968341 | 0.958616 | 0.967104 ± 0.007942 |
|
||||
| macro-F1 | 0.850470 | 0.815941 | 0.849378 | 0.838596 ± 0.019627 |
|
||||
| accuracy | 0.742838 | 0.744026 | 0.742907 | 0.743257 ± 0.000667 |
|
||||
| weighted-F1 | 0.716950 | 0.720106 | 0.717530 | 0.718195 ± 0.001680 |
|
||||
|
||||
Note the global rare support is very different per seed (104,843 / 13,375 / 128,995), because the
|
||||
training-derived union of rare classes differs by seed; cross-seed absolute levels are therefore
|
||||
not directly comparable.
|
||||
|
||||
---
|
||||
|
||||
## J. `original_protean` comparison
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.871964 | 0.921136 | 0.848525 | 0.880542 ± 0.037058 |
|
||||
| rare-class recall | 0.974314 | 0.968566 | 0.975859 | 0.972913 ± 0.003843 |
|
||||
| macro-F1 | 0.836881 | 0.865437 | 0.824867 | 0.842395 ± 0.020840 |
|
||||
| accuracy | 0.734416 | 0.747377 | 0.750133 | 0.743975 ± 0.008393 |
|
||||
| weighted-F1 | 0.708765 | 0.722802 | 0.724622 | 0.718730 ± 0.008678 |
|
||||
|
||||
Descriptive difference (`beta_star` − `original_protean`) for test rare-class macro-F1 is
|
||||
**+0.008998** (seed 0), **−0.081061** (seed 1), **+0.046389** (seed 2); mean **−0.008558**. Seed 1
|
||||
moves strongly the opposite way to seeds 0 and 2, and the magnitude is far smaller than the
|
||||
between-seed std (≈ 0.03–0.04). No superiority or generalisation is claimed; this is one dataset,
|
||||
one alpha, three project-defined seeds. Test columns:
|
||||
`artifacts/analysis/5gnidd__alpha0.5__multiseed__test_descriptive.csv`.
|
||||
|
||||
A descriptive dataset-level observation: on 5G-NIDD, test weighted-F1 (`≈0.718`) is **below**
|
||||
accuracy (`≈0.744`), i.e. the high-support classes (`Benign`, `UDPFlood`) are the harder ones for
|
||||
nearest-prototype inference. This is the opposite of X-IIoTID (weighted-F1 > accuracy) and is a
|
||||
dataset characteristic, not a beta effect.
|
||||
|
||||
---
|
||||
|
||||
## K. `beta = 0` equivalence vs `original_protean`
|
||||
|
||||
| property | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| same manifest / split / hashes | yes | yes | yes |
|
||||
| round-0 snapshot byte-identical | yes (`eb0839f2…`) | yes (`6cbd04e0…`) | yes (`f012a02b…`) |
|
||||
| support / characterization / rare_unseen identical | yes | yes | yes |
|
||||
| independent aggregation diff (`ΣC/n` vs `(1/n)ΣC`) | 4.4e-16 | 4.4e-16 | 4.4e-16 |
|
||||
| history rare-macro-F1 round-0 diff | 0.0 | 0.0 | 0.0 |
|
||||
| history rare-macro-F1 max diff (rounds ≥ 1) | 1.93e-2 | 6.01e-2 | 2.60e-2 |
|
||||
| test metric max abs diff | 2.33e-3 | 1.86e-4 | 8.70e-3 |
|
||||
| **classification** | **B** | **B** | **B** |
|
||||
|
||||
Class **B**: semantic equivalence + floating-point numerical divergence; **no semantic
|
||||
discrepancy**. Round 0 is byte-identical; a ~1 ULP ordering difference enters round-1
|
||||
prototype-alignment training and the trajectories then diverge. The downstream divergence is
|
||||
somewhat larger than at X-IIoTID (5G-NIDD training is more sensitive to the tiny round-1
|
||||
difference), but it remains pure floating-point divergence, not a semantic difference. Equivalence
|
||||
remains conditional on the shared `S_j` reproduction assumption and is **not** proof that
|
||||
`beta = 0` reproduces original PROTEAN. Machine-readable:
|
||||
`artifacts/analysis/5gnidd__alpha0.5__multiseed__beta0_equivalence.csv`.
|
||||
|
||||
---
|
||||
|
||||
## L. Direct prototype-effect findings (fixed round-0 snapshots, no retraining)
|
||||
|
||||
Computed from each seed's fixed `{C_ij, N_ij}` round-0 snapshot via the production aggregation.
|
||||
Cross-seed mean `||Cbar_j(beta) − Cbar_j(0)||` at `beta = 1.0` (sample std across seeds):
|
||||
|
||||
| class | mean disp | std | mean rel disp | #contributors (seed 0) |
|
||||
|---|---|---|---|---|
|
||||
| UDPScan | 3.461 | 2.011 | 0.424 | 10 |
|
||||
| HTTPFlood | 3.120 | 1.289 | 0.433 | 10 |
|
||||
| SYNScan | 2.875 | 0.959 | 0.396 | 10 |
|
||||
| SlowrateDoS | 2.763 | 1.454 | 0.454 | 10 |
|
||||
| SYNFlood | 2.262 | 0.748 | 0.658 | 9 |
|
||||
| TCPConnectScan | 2.131 | 0.213 | 0.575 | 10 |
|
||||
| Benign | 1.690 | 0.699 | 0.296 | 10 |
|
||||
| UDPFlood | 1.598 | 0.428 | 0.300 | 10 |
|
||||
| ICMPFlood | 1.159 | 0.093 | 0.321 | 9 |
|
||||
|
||||
- Mean displacement across the nine classes is **2.340** at beta=1.0, monotone in beta
|
||||
(0.25 → 0.953, 0.50 → 1.567, 0.75 → 2.003, 1.00 → 2.340).
|
||||
- Displacement is **seed-sensitive**: e.g. `UDPScan` 1.42 (seed 0) / 3.52 (seed 1) / 5.44
|
||||
(seed 2); `HTTPFlood` 2.40 / 2.36 / 4.61. The largest displacements are the rare, support-
|
||||
imbalanced classes; `Benign`/`UDPFlood` move least.
|
||||
- Even classes with a single missing contributor (`ICMPFlood`, `SYNFlood`) move, because the
|
||||
weighting changes among the 9–10 contributors, not because of the missing one.
|
||||
- Displacement is a **direct-aggregation measure only** and is **not** interpreted as improved or
|
||||
degraded detection. The end-to-end effect is reported separately in §H/§I.
|
||||
|
||||
Outputs: `artifacts/analysis/5gnidd__alpha0.5__seed{0,1,2}__support_weighted__beta0__prototype_snapshot__prototype_effect.csv`
|
||||
and `...__prototype_weights.csv`; cross-seed aggregate
|
||||
`artifacts/analysis/5gnidd__alpha0.5__multiseed__prototype_effect_pooled.csv`.
|
||||
|
||||
---
|
||||
|
||||
## M. Client-level effect (validation, relative to beta = 0)
|
||||
|
||||
Detail: `...__client_validation_detail.csv`; summary: `...__client_validation_summary.csv`. Client
|
||||
ids are per-seed realisations and are **not** pooled across seeds.
|
||||
|
||||
Improve / degrade / unchanged counts (tolerance 1e-9):
|
||||
|
||||
| seed | beta | improve | degrade | unchanged | mean Δ | min Δ | max Δ |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 0 | 0.25 | 2 | 7 | 1 | −0.0771 | −0.2036 | +0.0304 |
|
||||
| 0 | 0.50 | 7 | 2 | 1 | +0.0259 | −0.0343 | +0.0699 |
|
||||
| 0 | 0.75 | 5 | 4 | 1 | +0.0121 | −0.0679 | +0.0713 |
|
||||
| 0 | **1.00** | **6** | **3** | **1** | **+0.0303** | −0.0884 | +0.1034 |
|
||||
| 1 | 0.25 | 3 | 3 | 4 | −0.0595 | −0.3231 | +0.0125 |
|
||||
| 1 | 0.50 | 3 | 5 | 2 | −0.0699 | −0.4231 | +0.0181 |
|
||||
| 1 | 0.75 | 3 | 5 | 2 | −0.0899 | −0.5481 | +0.0240 |
|
||||
| 1 | **1.00** | **3** | **5** | **2** | **−0.1015** | −0.6564 | +0.0631 |
|
||||
| 2 | 0.25 | 5 | 4 | 1 | +0.0102 | −0.0262 | +0.0726 |
|
||||
| 2 | 0.50 | 5 | 4 | 1 | +0.0063 | −0.0442 | +0.0738 |
|
||||
| 2 | 0.75 | 7 | 2 | 1 | +0.0813 | −0.0305 | +0.1974 |
|
||||
| 2 | **1.00** | **7** | **2** | **1** | **+0.0866** | −0.0337 | +0.2016 |
|
||||
|
||||
- The response is strongly seed-dependent. At the selected `beta = 1.00`, seeds 0 and 2 improve
|
||||
on 6/10 and 7/10 clients with positive mean Δ (+0.030, +0.087), while seed 1 degrades 5/10 with
|
||||
mean Δ −0.101, dominated by a single client at **−0.656**.
|
||||
- `N_ij` vs ΔF1 is **non-monotonic**: the worst negative delta in seed 1 (client 2) has
|
||||
`min positive attack N_ij = 45` (not the lowest-support client), while some very low-support
|
||||
clients (`min N_ij = 1`) improve strongly.
|
||||
- This analysis is descriptive and non-causal.
|
||||
|
||||
---
|
||||
|
||||
## N. Runtime and resources
|
||||
|
||||
CPU only, no GPU, no swap, exit status 0 for every run (`/usr/bin/time -v`). Full table:
|
||||
`artifacts/analysis/5gnidd__alpha0.5__multiseed__runtime.csv`.
|
||||
|
||||
| seed | aggregation | wall | CPU % | peak RSS |
|
||||
|---|---|---|---|---|
|
||||
| 0 | `original_protean` | 62:28 | 745 | 10.36 GiB |
|
||||
| 0 | beta 0.00 | 63:37 | 745 | 10.37 GiB |
|
||||
| 0 | beta 0.25 | 62:56 | 744 | 10.36 GiB |
|
||||
| 0 | beta 0.50 | 63:26 | 743 | 10.36 GiB |
|
||||
| 0 | beta 0.75 | 63:35 | 742 | 10.39 GiB |
|
||||
| 0 | beta 1.00 | 64:25 | 741 | 10.36 GiB |
|
||||
| 1 | `original_protean` | 63:57 | 742 | 8.67 GiB |
|
||||
| 1 | beta 0.00 | 66:26 | 738 | 8.67 GiB |
|
||||
| 1 | beta 0.25 | 68:25 | 734 | 8.69 GiB |
|
||||
| 1 | beta 0.50 | 64:57 | 740 | 8.67 GiB |
|
||||
| 1 | beta 0.75 | 65:30 | 740 | 8.67 GiB |
|
||||
| 1 | beta 1.00 | 64:41 | 741 | 8.67 GiB |
|
||||
| 2 | `original_protean` | 63:44 | 740 | 14.04 GiB |
|
||||
| 2 | beta 0.00 | 63:39 | 739 | 14.05 GiB |
|
||||
| 2 | beta 0.25 | 63:29 | 740 | 14.03 GiB |
|
||||
| 2 | beta 0.50 | 63:14 | 739 | 14.06 GiB |
|
||||
| 2 | beta 0.75 | 63:38 | 739 | 14.06 GiB |
|
||||
| 2 | beta 1.00 | 63:29 | 739 | 14.07 GiB |
|
||||
|
||||
- New compute this session: **18 runs, 69,336 s ≈ 19.26 h** wall-clock, sequential on an 8-core
|
||||
host.
|
||||
- Per-run wall time is homogeneous (62–68 min). Peak RSS is partition-dependent, not beta-
|
||||
dependent: ~10.4 GiB (seed 0), ~8.67 GiB (seed 1), ~14.04 GiB (seed 2), all within the 29 GiB
|
||||
host. 5G-NIDD uses more memory than X-IIoTID (≈8.7–14.0 GiB vs ≈6.5–7.2 GiB) because the
|
||||
training split is ~1.5× larger.
|
||||
- No warnings, no client failures, 9/9 global prototypes in every round, no retries.
|
||||
|
||||
---
|
||||
|
||||
## O. Comparison with X-IIoTID alpha = 0.50 (descriptive only)
|
||||
|
||||
X-IIoTID alpha=0.50 reference: pooled-selected `beta = 0.75`
|
||||
(`docs/research/MULTISEED_BETA_VALIDATION.md`, `XIIOTID_CROSS_ALPHA_ANALYSIS.md`).
|
||||
|
||||
| statistic | 5G-NIDD alpha=0.50 | X-IIoTID alpha=0.50 |
|
||||
|---|---|---|
|
||||
| classes (attack) | 9 (8) | 9 (8) |
|
||||
| training rows | 851,123 | 569,909 |
|
||||
| pooled-selected beta | **1.00** | **0.75** |
|
||||
| pooled validation mean range (spread) | 0.645–0.692 (**0.047**) | 0.544–0.579 (0.035) |
|
||||
| pooled validation std range | 0.053–0.139 | 0.075–0.100 |
|
||||
| `|S_j| < M` cells (sum/27) | **4/27** | 7/27 |
|
||||
| missing client-class cells | **4** | 9 |
|
||||
| client-size ratio (seeds / mean) | 11.63 / 4.43 / 12.72 (**9.59**) | 5.11 / 7.22 / 9.89 (7.41) |
|
||||
| mean class-support CV | 1.225 | 1.183 |
|
||||
| union rare per seed | 6 / 5 / 6 | 6 / 7 / 7 |
|
||||
| union unseen per seed | 2 / 0 / 2 | 3 / 3 / 1 |
|
||||
| max unseen on a client | 1 | 2 |
|
||||
| mean displacement `‖Cbar(1)−Cbar(0)‖` | 2.340 | 2.099 |
|
||||
| selected-beta client improve/degrade (per seed) | 6/3, 3/5, 7/2 | 7/3, 8/1, 8/1 |
|
||||
|
||||
Observations (descriptive, two datasets, one condition each, no causal claim):
|
||||
|
||||
- 5G-NIDD alpha=0.50 has **fewer missing class cells** (4 vs 9) but **more imbalanced client
|
||||
sizes** (mean ratio 9.59 vs 7.41); its mean support CV and mean prototype displacement are
|
||||
slightly higher, and it has a single-client unseen-class structure (max 1 vs 2).
|
||||
- The pooled validation selection differs across datasets (5G-NIDD → 1.00, X-IIoTID → 0.75) and
|
||||
the 5G-NIDD curve is non-monotone. At both datasets the between-seed std is comparable to or
|
||||
larger than the between-beta spread, so both selections are statistically weak.
|
||||
- The client-level response is seed-dependent on both datasets; X-IIoTID alpha=0.50 improved on a
|
||||
majority of clients in all three seeds, whereas 5G-NIDD's seed 1 degrades.
|
||||
- **No claim** is made that the dataset explains the selected beta, and no generalisation beyond
|
||||
these two observed conditions is offered.
|
||||
|
||||
---
|
||||
|
||||
## P. Limitations
|
||||
|
||||
- **One new condition** (5G-NIDD alpha=0.50). No 5G-NIDD alpha=0.25/0.75 result exists; no
|
||||
cross-alpha 5G-NIDD claim is possible. No X-IIoTID/5G-NIDD causal or monotone claim.
|
||||
- **Three project-defined seeds** (`{0,1,2}`); the pooled mean's variation across seeds is
|
||||
comparable to or larger than the between-beta spread, so `beta_star = 1.00` is statistically
|
||||
weak and is only "best among the five tested candidates under pooled validation for this
|
||||
condition".
|
||||
- The `S_j` participant/denominator rule (`S_j = {i | N_ij > 0 and valid C_ij}`, equal weights in
|
||||
the baseline) remains a **REPRODUCTION ASSUMPTION**. At 5G-NIDD alpha=0.50 it is material in
|
||||
only **4/27** class-condition cells, but the `beta=0` comparison is still conditional on both
|
||||
modes sharing this rule and does **not** prove `beta=0` reproduces original PROTEAN.
|
||||
- Train/validation/test adaptation (70/10/20 stratified) differs from the paper's literal 80/20;
|
||||
the validation eval partition is a project decision.
|
||||
- The union of rare classes (and hence global rare support) differs by seed; only the per-client
|
||||
mean metric is directly comparable within the condition.
|
||||
- `preprocessing_hash`/`config_hash` remain `null` in the 5G-NIDD manifests.
|
||||
- **No global beta optimum claim**; test metrics did not influence selection.
|
||||
@@ -0,0 +1,284 @@
|
||||
# Session 9 Post-Experiment Audit (5G-NIDD alpha=0.50)
|
||||
|
||||
Status: **PASS.** Independent recomputation from machine-readable artifacts and the raw dataset
|
||||
confirms every Session-9 Stage-B claim for 5G-NIDD `alpha=0.50`. No discovered issue changes
|
||||
`beta_star` or the scientific interpretation. No corrections were required. **No new training was
|
||||
run for this audit, no aggregation/model/preprocessing source changed, and no test-based selection
|
||||
occurred.**
|
||||
|
||||
Audit performed: 2026-09-28. Audited HEAD: `2d27abc35026456de765c929dbcf0503a72ac572`.
|
||||
Machine-readable audit record:
|
||||
`artifacts/analysis/5gnidd__alpha0.5__multiseed__audit_recomputation.json`.
|
||||
|
||||
Companion documents: `docs/research/5GNIDD_ALPHA050_BETA_VALIDATION.md`,
|
||||
`docs/research/EXPERIMENT_PROTOCOL.md`, `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md`,
|
||||
`docs/handoff/SESSION_HANDOFF.md`.
|
||||
|
||||
---
|
||||
|
||||
## A. Scope
|
||||
|
||||
Independently verify the Session-9 claim for:
|
||||
|
||||
- dataset `5gnidd`, alpha `0.50`, seeds `{0,1,2}`, beta `{0, 0.25, 0.5, 0.75, 1.0}` plus
|
||||
`original_protean` (18 runs);
|
||||
- PRIMARY pooled-validation selection `beta_star = 1.00` using validation per-client
|
||||
rare-class macro-F1 averaged within seed then across seeds;
|
||||
- reuse decision for the pre-existing seed-0 baseline;
|
||||
- `beta=0` equivalence class;
|
||||
- cross-dataset label (5G-NIDD 1.00 vs X-IIoTID alpha=0.50 0.75).
|
||||
|
||||
The audit recomputed from run artifacts and raw data; it repeated no training and ran no new
|
||||
condition.
|
||||
|
||||
---
|
||||
|
||||
## B. Run inventory
|
||||
|
||||
18 full-data runs enumerated from `results/*/metadata.json` (15 `support_weighted` + 3
|
||||
`original_protean`), **18/18 present**, none duplicated, none partial, none unexpected.
|
||||
|
||||
- Each run contains all 14 expected artifacts; the round-0 prototype snapshot exists for all 18
|
||||
runs (`artifacts/analysis/5gnidd__alpha0.5__seed*__*__prototype_snapshot.json`).
|
||||
- Metadata verified per run: dataset=5gnidd, alpha=0.5, seed, mode, clients=10, rounds=10,
|
||||
local_epochs=3, batch=256, lr=1e-3, mu=0.1, lambda=1.0, `manifest_reload_ok=true`,
|
||||
`smoke=false`. All 18 exit status 0.
|
||||
- No 5G-NIDD run exists for alpha=0.25 or alpha=0.75; no `support_weighted` 5G-NIDD run older than
|
||||
this session exists; `equal_available` is absent from `src/`, `scripts/`, `tests/`.
|
||||
|
||||
---
|
||||
|
||||
## C. Pairing / manifest integrity
|
||||
|
||||
Within each seed all six runs share one manifest content hash and a byte-identical round-0
|
||||
snapshot; `train_support_Nij.csv`, `characterization.csv` and `rare_unseen.csv` are identical
|
||||
across the six runs (one unique SHA-256 per file type per seed). `manifest_reload_ok = true` and
|
||||
`smoke = false` everywhere.
|
||||
|
||||
| seed | manifest content hash | round-0 snapshot SHA-256 | support/char/rare unique counts |
|
||||
|---|---|---|---|
|
||||
| 0 | `0732582cf4510089…` | `eb0839f2595727…` | 1 / 1 / 1 |
|
||||
| 1 | `f43906e111a8342c…` | `6cbd04e0dd97bb…` | 1 / 1 / 1 |
|
||||
| 2 | `fe8b4fc6f0486cf4…` | `f012a02b543b92…` | 1 / 1 / 1 |
|
||||
|
||||
Each alpha=0.5 manifest was independently reproduced from the raw dataset
|
||||
(`prepare_dataset` → `reproduce_train_assignments`): exact `N_ij` match and partition-hash match
|
||||
for all three seeds; split sizes 851,123 / 121,589 / 243,177; `N_ij` sum 851,123; 10 clients.
|
||||
Pairing is proven by content hashes, not filenames. As a side check, alpha=0.25/0.75 5G-NIDD
|
||||
manifests also reproduced exactly.
|
||||
|
||||
---
|
||||
|
||||
## D. Partition characterization audit
|
||||
|
||||
Independently recomputed from the manifests; matches the report exactly:
|
||||
|
||||
| statistic | seed 0 / 1 / 2 | reality (sum) |
|
||||
|---|---|---|
|
||||
| `|S_j| < M` cells | 2 / 0 / 2 | **4 / 27** |
|
||||
| missing client-class cells | 2 / 0 / 2 | **4** |
|
||||
| client-size ratio | 11.63 / 4.43 / 12.72 | mean **9.59** |
|
||||
| union rare classes | 6 / 5 / 6 | 2 rare per client (all) |
|
||||
| union unseen classes | 2 / 0 / 2 | max 1 on a client |
|
||||
|
||||
---
|
||||
|
||||
## E. Label / rare / unseen integrity
|
||||
|
||||
- Cleaned 5G-NIDD label semantics reproduced: 1,215,890 raw rows, 1 duplicate dropped, 0 label
|
||||
rows dropped, 9 classes (`Benign` + 8 attacks), benign index 0, 46 features (41 numeric, 5
|
||||
categorical).
|
||||
- Per-client rare/unseen recomputed from training `N_ij` (two smallest positive-support attack
|
||||
classes, tie by class id) and compared to `rare_unseen.csv` across all 18 runs: **270 cells
|
||||
checked, 0 mismatches**; benign class never flagged rare/unseen; rare and unseen never overlap;
|
||||
every client has exactly two rare classes.
|
||||
- `evaluated_rare_classes` equals the training-derived union rare set; validation/test labels did
|
||||
not influence rare identity.
|
||||
|
||||
---
|
||||
|
||||
## F. Validation metric recomputation
|
||||
|
||||
Recomputed from raw artifacts (no report numbers copied):
|
||||
|
||||
- seed-level primary score = mean over eligible clients of `val_per_client.csv`
|
||||
`rare_class_macro_f1`; equals `val_selection.json` per-client-mean to **max diff 0.0** for all
|
||||
15 runs; all 10 clients eligible in every seed.
|
||||
- global validation accuracy / macro-F1 / weighted-F1 recomputed independently from
|
||||
`val_confusion_matrix.json` with the production formulas: **max diff 0.0** versus
|
||||
`val_global_metrics.json`.
|
||||
|
||||
---
|
||||
|
||||
## G. Pooled-selection recomputation
|
||||
|
||||
Computed from the independently recomputed seed-level scores (not from the report). Pooled = mean
|
||||
of the three seed means; std sample (`ddof=1`).
|
||||
|
||||
| beta | pooled mean | sample std |
|
||||
|---|---|---|
|
||||
| 0.00 | 0.687230 | 0.052726 |
|
||||
| 0.25 | 0.645078 | 0.064743 |
|
||||
| 0.50 | 0.674669 | 0.102253 |
|
||||
| 0.75 | 0.688368 | 0.126032 |
|
||||
| **1.00** | **0.692370** | 0.139140 |
|
||||
|
||||
- Recompute **matches the report** to formatting precision.
|
||||
- `argmax` = **1.00** (no tie; top-two gap ≈ 4.0e-3). `tie_break_used=false`.
|
||||
- Per-seed diagnostics reproduce **1.00 / 0.00 / 1.00**.
|
||||
|
||||
---
|
||||
|
||||
## H. Validation / test leakage audit
|
||||
|
||||
- `run_baseline.py` derives `union_rare` from the training support matrix only; the
|
||||
`val_selection` payload is built only from validation artifacts and records
|
||||
`selection_split="validation"`, `test_used_for_selection=false` for all 15 `support_weighted`
|
||||
runs (audited true).
|
||||
- `analyze_multiseed_beta.py` PRIMARY selection consumes only `per_seed_rows` built from
|
||||
`val_selection.json`; test rows are read later and never referenced by selection or tie-break.
|
||||
No code path sorts/filters candidates by a test field.
|
||||
- The audit's own `beta_star = 1.00` was reproduced from validation artifacts alone.
|
||||
|
||||
Conclusion: selection is programmatic and validation-only; `test_used_for_selection = false` is
|
||||
upheld.
|
||||
|
||||
---
|
||||
|
||||
## I. Final test recomputation
|
||||
|
||||
`beta_star = 1.00` fixed from pooled validation; only then test metrics read. Recomputed from
|
||||
`test_descriptive.csv` (which is itself derived from `test_global_metrics.json`), averaged across
|
||||
seeds with sample std:
|
||||
|
||||
| metric | selected beta 1.00 (mean ± std) | original_protean (mean ± std) |
|
||||
|---|---|---|
|
||||
| rare-class macro-F1 | 0.871984 ± 0.028501 | 0.880542 ± 0.037058 |
|
||||
| rare-class recall | 0.967104 ± 0.007942 | 0.972913 ± 0.003843 |
|
||||
| macro-F1 | 0.838596 ± 0.019627 | 0.842395 ± 0.020840 |
|
||||
| accuracy | 0.743257 ± 0.000667 | 0.743975 ± 0.008393 |
|
||||
| weighted-F1 | 0.718195 ± 0.001680 | 0.718730 ± 0.008678 |
|
||||
|
||||
Matches the report exactly. No superiority claim is made.
|
||||
|
||||
---
|
||||
|
||||
## J. `beta = 0` equivalence audit
|
||||
|
||||
For seeds 0, 1, 2, `original_protean` vs `support_weighted beta=0`:
|
||||
|
||||
| property | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| same manifest / split / hashes | yes | yes | yes |
|
||||
| round-0 snapshot byte-identical | yes | yes | yes |
|
||||
| support/characterization/rare_unseen identical | yes | yes | yes |
|
||||
| independent aggregation diff | 4.4e-16 | 4.4e-16 | 4.4e-16 |
|
||||
| history rare-macro-F1 round-0 diff | 0.0 | 0.0 | 0.0 |
|
||||
| history rare-macro-F1 max diff (rounds ≥ 1) | 1.93e-2 | 6.01e-2 | 2.60e-2 |
|
||||
| test metric max diff | 2.33e-3 | 1.86e-4 | 8.70e-3 |
|
||||
| classification | **B** | **B** | **B** |
|
||||
|
||||
Class **B** (semantically equivalent + floating-point divergence); **no semantic discrepancy**.
|
||||
Equivalence remains conditional on the shared `S_j` reproduction assumption.
|
||||
|
||||
---
|
||||
|
||||
## K. Direct-effect audit
|
||||
|
||||
Independently recomputed `||Cbar_j(beta) − Cbar_j(0)||` from the fixed round-0 snapshots with the
|
||||
production aggregation: the seed-0 `...__prototype_effect.csv` is produced by
|
||||
`analyze_prototype_effect.py` from the snapshot and the cross-seed pooled file aggregates it.
|
||||
Mean displacement at `beta=1.0` is **2.340** (monotone: 0.953 / 1.567 / 2.003 / 2.340 for
|
||||
beta 0.25/0.50/0.75/1.00). Displacement is a direct-aggregation measure only; no detection
|
||||
causality is inferred.
|
||||
|
||||
---
|
||||
|
||||
## L. Client-level audit
|
||||
|
||||
Deltas recomputed from `val_per_client.csv` vs `client_validation_detail.csv` (via the
|
||||
`beta=0` base): improve/degrade/unchanged at the selected `beta=1.00` = 6/3/1 (seed 0,
|
||||
mean +0.030), 3/5/2 (seed 1, mean −0.101), 7/2/1 (seed 2, mean +0.087). `N_ij` vs ΔF1 is
|
||||
non-monotonic. Client ids are per-seed realisations, not pooled.
|
||||
|
||||
---
|
||||
|
||||
## M. Cross-dataset labels
|
||||
|
||||
Condition-level pooled selections confirmed from `selection_summary.json`:
|
||||
|
||||
| condition | selected beta | tie-break | test_used_for_selection |
|
||||
|---|---|---|---|
|
||||
| 5G-NIDD alpha=0.50 | 1.00 | no | false |
|
||||
| X-IIoTID alpha=0.50 | 0.75 | no | false |
|
||||
|
||||
The report makes no causal or generalising claim and explicitly states the selections differ
|
||||
descriptively across the two datasets.
|
||||
|
||||
---
|
||||
|
||||
## N. Reuse decision audit
|
||||
|
||||
The pre-existing `results/5gnidd__alpha0.5__seed0__original_protean` (backed up to
|
||||
`/tmp/opencode/s9/old_baseline_backup/`) lacks all validation artifacts and therefore failed the
|
||||
reuse criterion; all 18 runs were re-executed. The fresh seed-0 `original_protean` reproduces the
|
||||
old artifact **exactly**: byte-identical `history.json` and identical test metrics (rare-class
|
||||
macro-F1 `0.8719637151922778`, accuracy `0.7344156725348203`). Reuse decision confirmed.
|
||||
|
||||
---
|
||||
|
||||
## O. Artifact provenance
|
||||
|
||||
| Reported quantity | Source artifact(s) | Derivation |
|
||||
|---|---|---|
|
||||
| run existence / config | `results/<run>/metadata.json` | direct read |
|
||||
| manifest hashes / reproduction | `artifacts/partitions/5gnidd__alpha0.5__seed{0,1,2}.json` | module + `reproduce_train_assignments` |
|
||||
| pairing proof | `artifacts/analysis/...__prototype_snapshot.json` | sha256 across 6 runs/seed |
|
||||
| seed-level validation score | `results/.../val_per_client.csv` + `val_selection.json` | mean of per-client rare macro-F1 |
|
||||
| validation global metrics | `results/.../val_confusion_matrix.json` | independent numpy recompute |
|
||||
| pooled mean/std, `beta_star` | recomputed seed scores | mean/`stdev`, argmax |
|
||||
| test metrics | `results/.../test_global_metrics.json` + `...__test_descriptive.csv` | independent recompute + mean/std |
|
||||
| partition characterization | manifests + `...__partition_heterogeneity_*.csv` + `...__partition_characterization.csv` | independent numpy recompute |
|
||||
| beta=0 equivalence | original vs beta0 artifacts/snapshots | sha256 + independent aggregation + history diff |
|
||||
| direct-effect displacement | snapshots + `...__prototype_effect_pooled.csv` | production aggregation |
|
||||
| client improve/degrade | `results/.../val_per_client.csv` + `...__client_validation_detail.csv` | deltas vs beta=0 |
|
||||
| rare-class identity | `results/.../rare_unseen.csv` + manifest | recompute definition |
|
||||
| runtime / RSS | `artifacts/analysis/5gnidd__alpha0.5__multiseed__runtime.csv` (source `/tmp/opencode/5gnidd_alpha050_runtime.csv`) | parse `/usr/bin/time -v`; sum 69,336 s = 19.26 h |
|
||||
| cross-dataset table | `xiiotid__alpha0.5__multiseed__*` + `5gnidd__alpha0.5__multiseed__*` | read per-condition multiseed artifacts |
|
||||
|
||||
---
|
||||
|
||||
## P. Deviations / corrections found
|
||||
|
||||
**None.** No arithmetic, metric-label, std-convention, provenance or pairing defect was found in
|
||||
the Session 9 report or analysis artifacts. No training/aggregation/preprocessing source file was
|
||||
modified.
|
||||
|
||||
Informational note (not a defect): the pre-existing seed-0 baseline was structurally incomplete
|
||||
(no validation artifacts) and was correctly not reused; the fresh rerun is byte-identical for the
|
||||
training outputs.
|
||||
|
||||
---
|
||||
|
||||
## Q. Final audit verdict
|
||||
|
||||
**PASS.**
|
||||
|
||||
- Run inventory: **PASS** (18/18; no duplicates/partials; no alpha≠0.50 or legacy 5G-NIDD run).
|
||||
- Pairing / manifest reproduction: **PASS** (hash-identical; raw-data reproduction exact).
|
||||
- Partition characterization: **PASS** (4/27, 4 missing, ratio 11.63/4.43/12.72, unseen 2/0/2).
|
||||
- Rare/unseen integrity: **PASS** (0 mismatches over 270 cells).
|
||||
- Validation recomputation (seed-level and global): **PASS** (exact).
|
||||
- Pooled selection: **PASS**; independent `beta_star = 1.00` using validation only.
|
||||
- Validation/test leakage: **PASS**.
|
||||
- Test summaries: **PASS** (exact).
|
||||
- beta=0 equivalence: **PASS**, class **B** for all seeds.
|
||||
- Direct-effect and client-level analyses: **PASS** (consistent with snapshots/artifacts).
|
||||
- Reuse decision: **PASS** (not reused; rerun reproduces the old baseline exactly).
|
||||
- Cross-dataset labels: **PASS** (5G-NIDD 1.00 vs X-IIoTID 0.75; no causal claim).
|
||||
- Source/data integrity: **PASS**; `planningawal/` unchanged; raw SHA-256 unchanged;
|
||||
`equal_available` absent; pytest 75 passed.
|
||||
|
||||
Session 9 is **safe to checkpoint** with the documented limitations (single new condition, three
|
||||
project-defined seeds, weak effect relative to seed variance, `S_j` reproduction assumption, no
|
||||
global-optimum claim).
|
||||
Reference in new issue
Block a user