experiment: validate 5gnidd beta selection at alpha 0.75
This commit is contained in:
willtanoe committed
1 parent
a7f2b0049e
commit
bbd3508506
36 files changed
+38705
-189
No files matched your search
@@ -0,0 +1,450 @@
|
||||
# 5G-NIDD Alpha-0.75 Beta Validation (Session 10)
|
||||
|
||||
Status: **second 5G-NIDD beta-validation condition**. This report covers a single 5G-NIDD
|
||||
condition — `alpha = 0.75`, seeds `{0,1,2}`, the frozen `original_protean` baseline plus the five
|
||||
`support_weighted` beta candidates — using the same pooled-across-seed validation-only selection
|
||||
rule already applied to X-IIoTID (`alpha=0.25/0.50/0.75`) and 5G-NIDD `alpha=0.50`. It is **not** a
|
||||
sweep, not a hyper-parameter search, and not evidence of a global optimum. **No other 5G-NIDD alpha
|
||||
(0.25 / 0.50) was run in this session**; `alpha=0.50` was completed and audited in Session 9 and is
|
||||
reused only as an audited reference.
|
||||
|
||||
Companion documents:
|
||||
|
||||
- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen protocol and selection hierarchy.
|
||||
- `docs/research/5GNIDD_ALPHA050_BETA_VALIDATION.md` — audited 5G-NIDD alpha=0.50 reference.
|
||||
- `docs/research/SESSION9_AUDIT.md` — audit of the alpha=0.50 condition.
|
||||
- `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md` — X-IIoTID three-condition reference.
|
||||
- `docs/research/SESSION10_AUDIT.md` — independent audit of this condition.
|
||||
- `docs/handoff/SESSION_HANDOFF.md` — continuation point.
|
||||
|
||||
---
|
||||
|
||||
## A. Scope
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| dataset | `5gnidd` (5G-NIDD) |
|
||||
| alpha | 0.75 |
|
||||
| seeds | 0, 1, 2 |
|
||||
| aggregation | `original_protean` (per seed) + `support_weighted` for `beta ∈ {0, 0.25, 0.50, 0.75, 1.00}` |
|
||||
| runs | 18: 15 `support_weighted` + 3 `original_protean`, **all run new this session** |
|
||||
| compute | sequential CPU, one run at a time, 69,064 s ≈ **19.18 h** wall-clock |
|
||||
| selection | pooled validation-only across seeds `{0,1,2}` |
|
||||
| test | read only after the pooled beta was fixed |
|
||||
|
||||
No 5G-NIDD `alpha=0.25` run was performed; `alpha=0.50` was not re-run. The full grid was
|
||||
deliberately not launched. `equal_available` was not added. No training/aggregation/preprocessing
|
||||
source code was changed (HEAD `a7f2b0049e99fb66021b4660b82a48d5dbbef072`, unchanged).
|
||||
|
||||
---
|
||||
|
||||
## B. Dataset and manifests
|
||||
|
||||
### B.1 Raw-data provenance (Stage A)
|
||||
|
||||
- Path `~/datasets/5G-NIDD/5G-NIDD.csv`; SHA-256
|
||||
`fa36f80859585f474504ca69eb951a079b35145537976c86b00aae3aab46ee59` — matches the Session-9
|
||||
audited value; **unmutated**.
|
||||
- Raw 1,215,890 rows; **1** exact duplicate dropped → 1,215,889; **0** label rows dropped.
|
||||
- **Label semantics (canonicalised):** label column `Attack Type`; **9 classes** — benign
|
||||
`Benign` (index 0) plus **8 attack classes** (`HTTPFlood, ICMPFlood, SYNFlood, SYNScan,
|
||||
SlowrateDoS, TCPConnectScan, UDPFlood, UDPScan`). Full-data counts unchanged from Session 9
|
||||
(Benign 477,736; UDPFlood 457,340; HTTPFlood 140,812; SlowrateDoS 73,124; TCPConnectScan
|
||||
20,052; SYNScan 20,043; UDPScan 15,906; SYNFlood 9,721; ICMPFlood 1,155).
|
||||
- **46 feature columns** (41 numeric median-imputed + train-standardized; 5 categorical one-hot
|
||||
with an explicit `__MISSING__` category). Preprocessing is spec-driven, fits on the train split
|
||||
only, and is identical to the previously audited pipeline (no source change).
|
||||
- Rare/unseen logic verified against this class list: attack classes only, `N_ij > 0`, two
|
||||
smallest positive training supports per client, `N_ij = 0` is unseen (never rare), benign never
|
||||
rare/unseen. All 10 clients are eligible in all three seeds (each has exactly two rare classes).
|
||||
|
||||
### B.2 Manifest identity (frozen, never regenerated)
|
||||
|
||||
Each `(alpha=0.75, seed)` manifest was independently reproduced from the raw dataset during Stage A
|
||||
(`prepare_dataset` → `reproduce_train_assignments`): exact `N_ij` match, partition-hash match and
|
||||
split present. Train/validation/test sizes are 851,123 / 121,589 / 243,177 for every seed.
|
||||
|
||||
| seed | manifest | manifest content hash | partition hash | split hash | rng seed |
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | `artifacts/partitions/5gnidd__alpha0.75__seed0.json` | `96cd9b90b71895da…` | `74a91845627dbe57…` | `18f89f4807de5493…` | 892370747 |
|
||||
| 1 | `artifacts/partitions/5gnidd__alpha0.75__seed1.json` | `007c23564a44919b…` | `fc04ba00f4942bf9…` | `3d05b7fe48d5a06c…` | 1817661572 |
|
||||
| 2 | `artifacts/partitions/5gnidd__alpha0.75__seed2.json` | `3134bea2d47a7336…` | `4ee33d0a28326ae1…` | `fc758d060dc51dec…` | 306220443 |
|
||||
|
||||
`preprocessing_hash` / `config_hash` remain `null` in these Session-2 manifests (known
|
||||
limitation, carry-over).
|
||||
|
||||
### B.3 Pairing (from content hashes, not filenames)
|
||||
|
||||
Within each seed all six runs share one manifest content hash and a **byte-identical round-0
|
||||
snapshot**; `train_support_Nij.csv`, `characterization.csv` and `rare_unseen.csv` each have exactly
|
||||
one SHA-256 across the six runs; `manifest_reload_ok = true` and `smoke = false` everywhere.
|
||||
|
||||
| seed | manifest content hash | round-0 snapshot SHA-256 (all 6 runs) |
|
||||
|---|---|---|
|
||||
| 0 | `96cd9b90b71895da4fcfb8c575c76d3a07ca14e824d8190ca6299eedeba6d6a5` | `266a6e9976d5c138f1382218242d08fd4ff95a215bb114bfaec0a058240ec65b` |
|
||||
| 1 | `007c23564a44919bba2520d09645e6a99c2a60dd347e8ff8d69a59aaa8ca6db3` | `acecab9a60852e4591eac991ae76894af36ac57d87cb869fe0d053db898e73c0` |
|
||||
| 2 | `3134bea2d47a7336c725e8d1a904b1e15c115336c0f018d685e1780cd952590d` | `ef90c51aa9998ec83080b8347473d0621a669de3b8aed575f45356a7b82f8c46` |
|
||||
|
||||
18/18 runs present with `dataset=5gnidd, alpha=0.75`, 10 clients, 10 rounds, 3 local epochs,
|
||||
batch 256, lr 1e-3, mu 0.1, lambda 1.0; all exit status 0.
|
||||
|
||||
---
|
||||
|
||||
## C. Frozen protocol
|
||||
|
||||
Identical to `EXPERIMENT_PROTOCOL.md` §14: 10 clients, 10 rounds, 3 local epochs, batch 256, Adam
|
||||
`lr=1e-3`, `mu=0.1`, `lambda=1.0`; unchanged architecture, preprocessing, split recipe, Dirichlet
|
||||
partition recipe (`per_class_dirichlet_multinomial`), loss and nearest-prototype inference. All 18
|
||||
runs used `scripts/run_baseline.py` at HEAD `a7f2b0049e99fb66021b4660b82a48d5dbbef072`.
|
||||
|
||||
Selection rule (unchanged):
|
||||
|
||||
- **PRIMARY:** pooled-across-seed validation per-client-mean rare-class macro-F1, validation only.
|
||||
- **SECONDARY (diagnostic):** per-seed validation selection.
|
||||
- **TEST:** read only after the pooled beta was fixed; `test_used_for_selection = false` for every
|
||||
`support_weighted` run.
|
||||
|
||||
---
|
||||
|
||||
## D. Partition characterization (realized, before interpretation)
|
||||
|
||||
All statistics are from the training-split `N_ij`; `M = 10` clients; `|S_j| < M` means fewer than
|
||||
10 clients contribute a prototype for a class.
|
||||
|
||||
### D.1 Per-seed summary (alpha=0.75)
|
||||
|
||||
| seed | `|S_j| < M` cells | missing client-class cells | client-size min/max (ratio) | union rare | union unseen | max unseen on a client |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0 | 2 / 9 | 3 | 38,839 / 143,547 (3.70) | 6 | 2 | 1 |
|
||||
| 1 | 2 / 9 | 2 | 17,861 / 202,829 (11.36) | 6 | 2 | 1 |
|
||||
| 2 | 0 / 9 | 0 | 49,537 / 191,908 (3.87) | 6 | 0 | 0 |
|
||||
| **total / mean** | **4 / 27** | **5** | ratio mean **6.31** | 6/6/6 | 2/2/0 | 1 |
|
||||
|
||||
- Unlike 5G-NIDD `alpha=0.50`, the **union of rare classes is identical for all three seeds**:
|
||||
`{ICMPFlood, SYNFlood, SYNScan, SlowrateDoS, TCPConnectScan, UDPScan}` (ICMPFlood is always
|
||||
rare, together with one of several low-support attack classes per client). All 10 clients have
|
||||
exactly two rare classes and are eligible in every seed.
|
||||
- Missing cells are single-client gaps: seed 0 — `ICMPFlood` absent on clients c0/c3 and `SYNFlood`
|
||||
absent on c4; seed 1 — `ICMPFlood` absent on c3 and `SYNScan` absent on c6.
|
||||
- `|S_j| < M` cells: seed 0 — `ICMPFlood` on 8 clients, `SYNFlood` on 9; seed 1 — `ICMPFlood` on 9,
|
||||
`SYNScan` on 9; seed 2 — all classes have 10 contributors.
|
||||
- Union unseen classes: seed 0 = `{ICMPFlood, SYNFlood}`, seed 1 = `{ICMPFlood, SYNScan}`,
|
||||
seed 2 = `{}`.
|
||||
- Machine-readable: `artifacts/analysis/5gnidd__alpha0.75__partition_heterogeneity_per_seed.csv`,
|
||||
`..._per_class.csv`, `..._summary.csv`, and per-client
|
||||
`artifacts/analysis/5gnidd__alpha0.75__partition_characterization.csv`.
|
||||
|
||||
### D.2 Per-class support inequality
|
||||
|
||||
Mean class-support CV across seeds = **0.916** (seed 0 0.969, seed 1 0.854, seed 2 0.925); the
|
||||
maximum per-class `max/min` ratio across seeds is 7,642.5 (driven by high-support `UDPFlood`).
|
||||
Complete per-class tables: `..._per_class.csv`.
|
||||
|
||||
### D.3 Comparison with the audited 5G-NIDD alpha=0.50
|
||||
|
||||
`alpha=0.75` is **not uniformly less heterogeneous** in every realized statistic:
|
||||
|
||||
| statistic (training split) | 5G-NIDD alpha=0.50 | 5G-NIDD alpha=0.75 |
|
||||
|---|---|---|
|
||||
| `|S_j| < M` cells (sum / 27) | 4 / 27 | **4 / 27** |
|
||||
| missing client-class cells | 4 | **5** |
|
||||
| client-size ratio (seeds) | 11.63 / 4.43 / 12.72 | 3.70 / 11.36 / 3.87 |
|
||||
| client-size ratio mean | 9.59 | **6.31** |
|
||||
| mean class-support CV | 1.225 | **0.916** |
|
||||
| union rare per seed | 6 / 5 / 6 | **6 / 6 / 6** |
|
||||
| union unseen per seed | 2 / 0 / 2 | 2 / 2 / 0 |
|
||||
| max unseen on a client | 1 | 1 |
|
||||
|
||||
Higher `alpha` reduces mean class-support CV and mean client-size spread, but the count of
|
||||
`|S_j| < M` cells is unchanged and missing cells are actually **higher** (5 vs 4); seed 1 retains a
|
||||
large client-size ratio (11.36). Realized statistics, not alpha alone, are used for interpretation.
|
||||
|
||||
---
|
||||
|
||||
## E. Per-seed validation results
|
||||
|
||||
`val rareMacroF1 (pc mean)` is the primary metric; all 10 clients eligible in every seed.
|
||||
Machine-readable: `artifacts/analysis/5gnidd__alpha0.75__multiseed__validation_per_seed.csv`.
|
||||
|
||||
### Seed 0
|
||||
|
||||
| beta | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|
|
||||
| 0.00 | 0.834681 | 0.896267 | 0.955238 | 0.851140 | 0.747485 |
|
||||
| 0.25 | 0.823388 | 0.899224 | 0.955542 | 0.853750 | 0.749155 |
|
||||
| **0.50** | **0.836792** | 0.901588 | 0.954228 | 0.855700 | 0.750191 |
|
||||
| 0.75 | 0.824667 | 0.901765 | 0.957174 | 0.856162 | 0.751178 |
|
||||
| 1.00 | 0.815774 | 0.896524 | 0.960685 | 0.852728 | 0.751334 |
|
||||
|
||||
### Seed 1
|
||||
|
||||
| beta | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|
|
||||
| 0.00 | 0.763787 | 0.860129 | 0.960640 | 0.829213 | 0.753489 |
|
||||
| 0.25 | 0.821190 | 0.896285 | 0.960906 | 0.853393 | 0.753687 |
|
||||
| 0.50 | 0.848836 | 0.916475 | 0.961261 | 0.866771 | 0.753555 |
|
||||
| 0.75 | 0.833255 | 0.913219 | 0.961421 | 0.864621 | 0.753662 |
|
||||
| **1.00** | **0.852469** | 0.911192 | 0.961375 | 0.863190 | 0.753300 |
|
||||
|
||||
### Seed 2
|
||||
|
||||
| beta | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|
|
||||
| 0.00 | 0.735812 | 0.870022 | 0.958369 | 0.835202 | 0.752198 |
|
||||
| 0.25 | 0.726871 | 0.862443 | 0.955793 | 0.830314 | 0.752379 |
|
||||
| 0.50 | 0.741617 | 0.874029 | 0.956090 | 0.838279 | 0.753078 |
|
||||
| 0.75 | 0.716668 | 0.850796 | 0.955018 | 0.822194 | 0.751170 |
|
||||
| **1.00** | **0.744677** | 0.874155 | 0.954768 | 0.838259 | 0.752881 |
|
||||
|
||||
The per-seed curves are **non-monotone and seed-dependent**: seed 0 peaks at 0.50; seed 1 rises
|
||||
then peaks at 1.00; seed 2 peaks at 1.00 with a dip at 0.75. Relative to `beta=0`, seeds 1 and 2
|
||||
gain substantially at intermediate/high beta while seed 0 loses at 0.25/0.75/1.00.
|
||||
|
||||
---
|
||||
|
||||
## F. Per-seed diagnostic beta choices (SECONDARY, stability only)
|
||||
|
||||
| seed | selected beta | val rareMacroF1 (pc mean) | tie-break used | test_used_for_selection |
|
||||
|---|---|---|---|---|
|
||||
| 0 | 0.50 | 0.836792 | no | false |
|
||||
| 1 | 1.00 | 0.852469 | no | false |
|
||||
| 2 | 1.00 | 0.744677 | no | false |
|
||||
|
||||
These per-seed choices are **diagnostic only**; they are not the condition-level selection rule.
|
||||
`val_selection.json` records `"selection_split": "validation"` and
|
||||
`"test_used_for_selection": false` for all 15 `support_weighted` runs.
|
||||
|
||||
---
|
||||
|
||||
## G. PRIMARY pooled-across-seed validation selection
|
||||
|
||||
| beta | pooled mean | std (sample, ddof=1) | min | max | pooled val rareRecall (pc mean) | selected |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | 0.778094 | 0.050963 | 0.735812 | 0.834681 | 0.959858 | |
|
||||
| 0.25 | 0.790483 | 0.055100 | 0.726871 | 0.823388 | 0.959614 | |
|
||||
| **0.50** | **0.809081** | 0.058735 | 0.741617 | 0.848836 | 0.959268 | **yes** |
|
||||
| 0.75 | 0.791530 | 0.064974 | 0.716668 | 0.833255 | 0.959624 | |
|
||||
| 1.00 | 0.804306 | 0.054803 | 0.744677 | 0.852469 | 0.960041 | |
|
||||
|
||||
- **PRIMARY selected beta (`beta_star_5gnidd_alpha075`) = 0.50.**
|
||||
- **Criterion:** maximum pooled validation per-client-mean rare-class macro-F1 (validation only).
|
||||
- **Tie-break used:** no. Top-two beta = 0.50 (0.809081) vs 1.00 (0.804306); gap ≈ **4.8e-3**.
|
||||
- **`test_used_for_selection = false`.** Test metrics were not inspected until the pooled beta was
|
||||
fixed.
|
||||
- **Interpretation:** `beta = 0.50` is the *best among the tested candidates under pooled
|
||||
validation across seeds `{0,1,2}` for 5G-NIDD alpha=0.75*. It is **not** a global optimum. The
|
||||
pooled validation curve is non-monotone (0.778 → 0.790 → 0.809 → 0.792 → 0.804); the total
|
||||
between-beta spread is ≈ **0.031** and the across-seed sample std is 0.051–0.065 — comparable to
|
||||
or larger than the between-beta spread, so the selection is statistically weak. The pooled win
|
||||
is driven by seeds 1 and 2 (both strongly positive versus `beta=0`) partly offset by seed 0.
|
||||
|
||||
Machine-readable: `artifacts/analysis/5gnidd__alpha0.75__multiseed__validation_pooled.csv`,
|
||||
`...__selection_summary.json`.
|
||||
|
||||
---
|
||||
|
||||
## H. Final selected-beta test metrics (descriptive only)
|
||||
|
||||
`beta_star = 0.50` was fixed from pooled validation; only then were test metrics read.
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.904447 | 0.905721 | 0.880045 | 0.896738 ± 0.014470 |
|
||||
| rare-class recall | 0.958666 | 0.959723 | 0.960715 | 0.959701 ± 0.001025 |
|
||||
| macro-F1 | 0.857309 | 0.859939 | 0.842201 | 0.853150 ± 0.009572 |
|
||||
| accuracy | 0.749508 | 0.754381 | 0.752933 | 0.752274 ± 0.002503 |
|
||||
| weighted-F1 | 0.723885 | 0.730109 | 0.728408 | 0.727467 ± 0.003217 |
|
||||
|
||||
The global rare evaluation support is **28,000 for every seed** (the training-derived union of rare
|
||||
classes is identical across seeds at this alpha), so cross-seed test levels **are** directly
|
||||
comparable at alpha=0.75 — unlike alpha=0.50. Test columns:
|
||||
`artifacts/analysis/5gnidd__alpha0.75__multiseed__test_descriptive.csv`.
|
||||
|
||||
---
|
||||
|
||||
## I. `original_protean` comparison
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.905589 | 0.860758 | 0.873356 | 0.879901 ± 0.023121 |
|
||||
| rare-class recall | 0.961285 | 0.959036 | 0.963321 | 0.961214 ± 0.002144 |
|
||||
| macro-F1 | 0.857109 | 0.829867 | 0.837238 | 0.841405 ± 0.014091 |
|
||||
| accuracy | 0.747048 | 0.754072 | 0.751745 | 0.750955 ± 0.003578 |
|
||||
| weighted-F1 | 0.721059 | 0.729950 | 0.727133 | 0.726047 ± 0.004544 |
|
||||
|
||||
Descriptive difference (`beta_star` − `original_protean`) for test rare-class macro-F1 is
|
||||
**−0.001141** (seed 0), **+0.044963** (seed 1), **+0.006689** (seed 2); mean **+0.016837**. Seed 0
|
||||
is essentially flat/weakly negative; seeds 1 and 2 are positive, but all differences are smaller
|
||||
than the between-seed std (≈ 0.023). No superiority or generalisation is claimed.
|
||||
|
||||
Descriptive dataset-level observation (consistent with alpha=0.50): on 5G-NIDD test weighted-F1
|
||||
(`≈0.727`) sits **below** accuracy (`≈0.752`) — the opposite of X-IIoTID — because the
|
||||
high-support classes are the harder ones for nearest-prototype inference.
|
||||
|
||||
---
|
||||
|
||||
## J. `beta = 0` equivalence vs `original_protean`
|
||||
|
||||
| property | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| same manifest / split / hashes | yes | yes | yes |
|
||||
| round-0 snapshot byte-identical | yes (`266a6e99…`) | yes (`acecab9a…`) | yes (`ef90c51a…`) |
|
||||
| support / characterization / rare_unseen identical | yes | yes | yes |
|
||||
| independent aggregation diff (`ΣC/n` vs `(1/n)ΣC`) | 4.4e-16 | 4.4e-16 | 4.4e-16 |
|
||||
| history rare-macro-F1 round-0 diff | 0.0 | 0.0 | 0.0 |
|
||||
| history rare-macro-F1 max diff (rounds ≥ 1) | 3.22e-2 | 5.04e-2 | 2.72e-2 |
|
||||
| test metric max abs diff | 5.22e-3 | 5.32e-3 | 2.74e-4 |
|
||||
| **classification** | **B** | **B** | **B** |
|
||||
|
||||
Class **B**: semantic equivalence + floating-point numerical divergence; **no semantic
|
||||
discrepancy**. Round 0 is byte-identical; a ~1 ULP ordering difference enters round-1
|
||||
prototype-alignment training and the trajectories then diverge. Equivalence remains conditional on
|
||||
the shared `S_j` reproduction assumption and is **not** proof that `beta=0` reproduces original
|
||||
PROTEAN. Machine-readable:
|
||||
`artifacts/analysis/5gnidd__alpha0.75__multiseed__beta0_equivalence.csv`.
|
||||
|
||||
---
|
||||
|
||||
## K. Direct prototype-effect findings (fixed round-0 snapshots, no retraining)
|
||||
|
||||
Computed from each seed's fixed `{C_ij, N_ij}` round-0 snapshot via the production aggregation.
|
||||
Cross-seed mean `||Cbar_j(beta) − Cbar_j(0)||` at `beta = 1.0` (sample std across seeds):
|
||||
|
||||
| class | mean disp | std | mean rel disp | #contributors (seed 0) |
|
||||
|---|---|---|---|---|
|
||||
| UDPScan | 2.445 | 0.484 | 0.285 | 10 |
|
||||
| SYNFlood | 1.897 | 0.171 | 0.542 | 9 |
|
||||
| SlowrateDoS | 1.565 | 0.654 | 0.252 | 10 |
|
||||
| SYNScan | 1.563 | 0.640 | 0.199 | 10 |
|
||||
| HTTPFlood | 1.366 | 0.544 | 0.175 | 10 |
|
||||
| TCPConnectScan | 1.226 | 0.476 | 0.326 | 10 |
|
||||
| UDPFlood | 0.903 | 0.103 | 0.163 | 10 |
|
||||
| Benign | 0.824 | 0.272 | 0.138 | 10 |
|
||||
| ICMPFlood | 0.673 | 0.331 | 0.179 | 8 |
|
||||
|
||||
- Mean displacement across the nine classes is **1.385** at beta=1.0, monotone in beta
|
||||
(0.25 → 0.497, 0.50 → 0.875, 0.75 → 1.163, 1.00 → 1.385).
|
||||
- Displacement is seed-sensitive (e.g. `SYNScan` std 0.640, `SlowrateDoS` std 0.654) and largest
|
||||
for the support-imbalanced rare classes (`UDPScan`, `SYNFlood`); `Benign`/`ICMPFlood` move least.
|
||||
- At alpha=0.75 the mean displacement is **smaller** than at alpha=0.50 (1.385 vs 2.340), tracking
|
||||
the lower realized support imbalance, though the per-class rank order differs.
|
||||
- Displacement is a **direct-aggregation measure only** and is **not** interpreted as improved or
|
||||
degraded detection. The end-to-end effect is reported separately in §G/§H.
|
||||
|
||||
Outputs: `artifacts/analysis/5gnidd__alpha0.75__seed{0,1,2}__support_weighted__beta0__prototype_snapshot__prototype_effect.csv`
|
||||
and `...__prototype_weights.csv`; cross-seed aggregate
|
||||
`artifacts/analysis/5gnidd__alpha0.75__multiseed__prototype_effect_pooled.csv`.
|
||||
|
||||
---
|
||||
|
||||
## L. Client-level effect (validation, relative to beta = 0)
|
||||
|
||||
Detail: `...__client_validation_detail.csv`; summary: `...__client_validation_summary.csv`. Client
|
||||
ids are per-seed realisations and are **not** pooled across seeds. Tolerance 1e-9.
|
||||
|
||||
At the selected `beta = 0.50`:
|
||||
|
||||
| seed | improve | degrade | unchanged | mean Δ | min Δ | max Δ |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0 | 4 | 6 | 0 | +0.0021 | −0.0434 | +0.0744 (+0.0744 at c0, min N_ij=2) |
|
||||
| 1 | 6 | 3 | 1 | +0.0850 | −0.0172 | +0.2667 (c8) |
|
||||
| 2 | 4 | 2 | 4 | +0.0058 | −0.0054 | +0.0237 |
|
||||
|
||||
- The response is seed-dependent. Seed 1 improves strongly on 6/10 clients (mean Δ +0.085, max
|
||||
+0.267); seed 0 is mixed (mean +0.002, but 6 degrade); seed 2 is mostly flat (4 unchanged).
|
||||
- `N_ij` vs ΔF1 is **non-monotonic**: seed 1's largest gain (c8, +0.267) has the largest min
|
||||
positive attack support (134), while seed 0's largest gain (c0, +0.074) has the smallest
|
||||
(min N_ij = 2); the worst seed-0 degradation (c7, −0.043) is not the lowest-support client.
|
||||
- This analysis is descriptive and non-causal.
|
||||
|
||||
Full per-beta client summaries: `...__client_validation_summary.csv`.
|
||||
|
||||
---
|
||||
|
||||
## M. Runtime and resources
|
||||
|
||||
CPU only, no GPU, no swap, exit status 0 for every run (`/usr/bin/time -v`). Full table:
|
||||
`artifacts/analysis/5gnidd__alpha0.75__multiseed__runtime.csv`.
|
||||
|
||||
| group | wall range | CPU % | peak RSS |
|
||||
|---|---|---|---|
|
||||
| seed 0 (6 runs) | 61:55 – 62:59 | 742–745 | 8.68 GiB |
|
||||
| seed 1 (6 runs) | 64:10 – 66:37 | 739–742 | 11.78 GiB |
|
||||
| seed 2 (6 runs) | 63:38 – 65:12 | 738–740 | 10.99 GiB |
|
||||
|
||||
- New compute this session: **18 runs, 69,064 s ≈ 19.18 h** wall-clock, sequential on an 8-core
|
||||
host.
|
||||
- Per-run wall time is homogeneous (61:55–66:37). Peak RSS is partition-dependent, not
|
||||
beta-dependent: ~8.69 GiB (seed 0), ~11.78 GiB (seed 1), ~11.00 GiB (seed 2), all within the
|
||||
host memory. 5G-NIDD uses more memory than X-IIoTID because the training split is ~1.5× larger.
|
||||
- No warnings, no client failures, 9/9 global prototypes in every round, no retries.
|
||||
|
||||
---
|
||||
|
||||
## N. Comparison with 5G-NIDD alpha = 0.50 (descriptive only)
|
||||
|
||||
Reference (Session 9, audited): `5GNIDD_ALPHA050_BETA_VALIDATION.md`.
|
||||
|
||||
| statistic | 5G-NIDD alpha=0.50 | 5G-NIDD alpha=0.75 |
|
||||
|---|---|---|
|
||||
| pooled-selected beta | **1.00** | **0.50** |
|
||||
| pooled validation mean range (spread) | 0.645–0.692 (0.047) | 0.778–0.809 (0.031) |
|
||||
| pooled validation std range | 0.053–0.139 | 0.051–0.065 |
|
||||
| `|S_j| < M` cells (sum / 27) | 4 / 27 | 4 / 27 |
|
||||
| missing client-class cells | 4 | 5 |
|
||||
| client-size ratio mean | 9.59 | 6.31 |
|
||||
| mean class-support CV | 1.225 | 0.916 |
|
||||
| union rare per seed | 6 / 5 / 6 | 6 / 6 / 6 |
|
||||
| union unseen per seed | 2 / 0 / 2 | 2 / 2 / 0 |
|
||||
| test rare support per seed | 104,843 / 13,375 / 128,995 | 28,000 / 28,000 / 28,000 |
|
||||
| mean displacement `‖Cbar(1)−Cbar(0)‖` | 2.340 | 1.385 |
|
||||
| selected-beta client improve/degrade (per seed) | 6/3, 3/5, 7/2 | 4/6, 6/3, 4/2 |
|
||||
| `beta=0` equivalence class (per seed) | B / B / B | B / B / B |
|
||||
|
||||
Descriptive observations (two alpha levels, one dataset, three project-defined seeds — **no
|
||||
monotone or causal claim**):
|
||||
|
||||
- Both pooled validation curves are **non-monotone**; the selected beta differs (0.50 at alpha=0.75
|
||||
vs 1.00 at alpha=0.50). Across-seed std is comparable to the between-beta spread at both
|
||||
conditions, so both selections are statistically weak.
|
||||
- Realized support imbalance is lower on average at alpha=0.75 (CV 0.916 vs 1.225; client ratio
|
||||
6.31 vs 9.59) and mean prototype displacement is smaller (1.385 vs 2.340), yet the number of
|
||||
`|S_j| < M` cells is identical (4/27) and missing cells are slightly higher.
|
||||
- At alpha=0.75 the rare-class union is identical across seeds (constant test rare support),
|
||||
whereas at alpha=0.50 it varies; this is a property of the realized partitions, not of beta.
|
||||
- Validation **levels** are not directly comparable across alpha (different rare-class sets and
|
||||
per-client eval partitions at alpha=0.50); only within-condition contrasts are interpreted.
|
||||
|
||||
Machine-readable: `artifacts/analysis/5gnidd__cross_alpha__cross_alpha_beta_table.csv`,
|
||||
`..._displacement.csv`, `..._client_summary.csv`, `..._summary.json` and
|
||||
`5gnidd__cross_alpha__partition_heterogeneity_*`.
|
||||
|
||||
---
|
||||
|
||||
## O. Optional cross-dataset context: X-IIoTID alpha = 0.75
|
||||
|
||||
Audited X-IIoTID reference (`XIIOTID_CROSS_ALPHA_ANALYSIS.md`): pooled-selected **beta = 0.50**
|
||||
(pooled mean 0.476749 ± 0.046624), realized heterogeneity cells `3/27`, 4 missing cells, client
|
||||
ratio mean 4.67, support CV 0.881, mean displacement 1.305. At alpha=0.75 both datasets select
|
||||
beta = 0.50 descriptively. This is **not** evidence of a dataset-independent rule: only two
|
||||
datasets, one alpha each in common, three seeds per condition, weak effects, and no inferential
|
||||
test. No causal or generalisation claim is made.
|
||||
|
||||
---
|
||||
|
||||
## P. Limitations
|
||||
|
||||
- **One new condition** (5G-NIDD alpha=0.75). No 5G-NIDD alpha=0.25 result exists; no full
|
||||
cross-alpha 5G-NIDD claim is possible yet. No X-IIoTID/5G-NIDD causal or monotone claim.
|
||||
- **Three project-defined seeds** (`{0,1,2}`); the pooled mean's variation across seeds is
|
||||
comparable to or larger than the between-beta spread, so `beta_star = 0.50` is statistically
|
||||
weak and is only "best among the five tested candidates under pooled validation for this
|
||||
condition".
|
||||
- The `S_j` participant/denominator rule (`S_j = {i | N_ij > 0 and valid C_ij}`, equal weights in
|
||||
the baseline) remains a **REPRODUCTION ASSUMPTION**. At 5G-NIDD alpha=0.75 it is material in
|
||||
**4/27** class-condition cells, but the `beta=0` comparison is still conditional on both modes
|
||||
sharing this rule and does **not** prove `beta=0` reproduces original PROTEAN.
|
||||
- Train/validation/test adaptation (70/10/20 stratified) differs from the paper's literal 80/20;
|
||||
the validation eval partition is a project decision.
|
||||
- `preprocessing_hash`/`config_hash` remain `null` in the 5G-NIDD manifests.
|
||||
- **No global beta optimum claim**; test metrics did not influence selection.
|
||||
@@ -0,0 +1,277 @@
|
||||
# Session 10 Post-Experiment Audit (5G-NIDD alpha=0.75)
|
||||
|
||||
Status: **PASS.** Independent recomputation from machine-readable artifacts and the raw dataset
|
||||
confirms every Session-10 Stage-B claim for 5G-NIDD `alpha=0.75`. No discovered issue changes
|
||||
`beta_star` or the scientific interpretation. **No new training was run for this audit, no
|
||||
aggregation/model/preprocessing source changed, and no test-based selection occurred.**
|
||||
|
||||
Audit performed: 2026-09-29. Audited HEAD: `a7f2b0049e99fb66021b4660b82a48d5dbbef072`.
|
||||
Machine-readable audit record:
|
||||
`artifacts/analysis/5gnidd__alpha0.75__multiseed__audit_recomputation.json`.
|
||||
|
||||
Companion documents: `docs/research/5GNIDD_ALPHA075_BETA_VALIDATION.md`,
|
||||
`docs/research/5GNIDD_ALPHA050_BETA_VALIDATION.md`, `docs/research/EXPERIMENT_PROTOCOL.md`,
|
||||
`docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md`, `docs/handoff/SESSION_HANDOFF.md`.
|
||||
|
||||
---
|
||||
|
||||
## A. Scope
|
||||
|
||||
Independently verify the Session-10 claim for:
|
||||
|
||||
- dataset `5gnidd`, alpha `0.75`, seeds `{0,1,2}`, beta `{0, 0.25, 0.5, 0.75, 1.0}` plus
|
||||
`original_protean` (18 runs);
|
||||
- PRIMARY pooled-validation selection `beta_star = 0.50` using validation per-client rare-class
|
||||
macro-F1 averaged within seed then across seeds;
|
||||
- `beta=0` equivalence class;
|
||||
- cross-alpha (5G-NIDD alpha=0.50) and cross-dataset (X-IIoTID alpha=0.75) labels.
|
||||
|
||||
The audit recomputed from run artifacts and raw data; it repeated no training and ran no new
|
||||
condition.
|
||||
|
||||
---
|
||||
|
||||
## B. Run inventory
|
||||
|
||||
18 full-data runs enumerated from `results/5gnidd__alpha0.75__*/metadata.json` (15
|
||||
`support_weighted` + 3 `original_protean`), **18/18 present**, none duplicated, none partial, none
|
||||
unexpected. Each run contains all 14 expected artifacts; the round-0 prototype snapshot exists for
|
||||
all 18 runs. Metadata verified per run: dataset=5gnidd, alpha=0.75, seed, mode, clients=10,
|
||||
rounds=10, local_epochs=3, batch=256, lr=1e-3, mu=0.1, lambda=1.0, `manifest_reload_ok=true`,
|
||||
`smoke=false`. All 18 exit status 0.
|
||||
|
||||
No 5G-NIDD run exists for alpha=0.25; no 5G-NIDD `support_weighted` run for any other alpha exists;
|
||||
`equal_available` is absent from `src/`, `scripts/`, `tests/`.
|
||||
|
||||
---
|
||||
|
||||
## C. Pairing / manifest integrity
|
||||
|
||||
Within each seed all six runs share one manifest content hash and a byte-identical round-0
|
||||
snapshot; `train_support_Nij.csv`, `characterization.csv` and `rare_unseen.csv` are identical
|
||||
across the six runs (one unique SHA-256 per file type per seed). `manifest_reload_ok = true` and
|
||||
`smoke = false` everywhere.
|
||||
|
||||
| seed | manifest content hash | round-0 snapshot SHA-256 | support/char/rare unique counts |
|
||||
|---|---|---|---|
|
||||
| 0 | `96cd9b90b71895da…` | `266a6e9976d5c138…` | 1 / 1 / 1 |
|
||||
| 1 | `007c23564a44919b…` | `acecab9a60852e45…` | 1 / 1 / 1 |
|
||||
| 2 | `3134bea2d47a7336…` | `ef90c51aa9998ec8…` | 1 / 1 / 1 |
|
||||
|
||||
Each alpha=0.75 manifest was independently reproduced from the raw dataset
|
||||
(`prepare_dataset` → `reproduce_train_assignments`): exact `N_ij` match, partition-hash match and
|
||||
split-hash match for all three seeds; split sizes 851,123 / 121,589 / 243,177; `N_ij` sum
|
||||
851,123; 10 clients. Pairing is proven by content hashes, not filenames.
|
||||
|
||||
---
|
||||
|
||||
## D. Partition characterization audit
|
||||
|
||||
Independently recomputed from the manifests; matches the report exactly:
|
||||
|
||||
| statistic | seed 0 / 1 / 2 | reality (sum / mean) |
|
||||
|---|---|---|
|
||||
| `|S_j| < M` cells | 2 / 2 / 0 | **4 / 27** |
|
||||
| missing client-class cells | 3 / 2 / 0 | **5** |
|
||||
| client-size ratio | 3.70 / 11.36 / 3.87 | mean **6.31** |
|
||||
| union rare classes | 6 / 6 / 6 | identical set across seeds |
|
||||
| union unseen classes | 2 / 2 / 0 | max 1 on a client |
|
||||
|
||||
Mean class-support CV = **0.916**; maximum per-class `max/min` ratio across seeds = 7,642.5.
|
||||
|
||||
---
|
||||
|
||||
## E. Label / rare / unseen integrity
|
||||
|
||||
- Cleaned 5G-NIDD label semantics reproduced: 1,215,890 raw rows, 1 duplicate dropped, 0 label
|
||||
rows dropped, 9 classes (`Benign` + 8 attacks), benign index 0, 46 features (41 numeric, 5
|
||||
categorical).
|
||||
- Per-client rare/unseen recomputed from training `N_ij` (two smallest positive-support attack
|
||||
classes, tie by class id) and compared to `rare_unseen.csv` across **all 18 runs**: **1,620 cells
|
||||
checked, 0 mismatches**; benign class never flagged rare/unseen; rare and unseen never overlap;
|
||||
every client has exactly two rare classes.
|
||||
- `evaluated_rare_classes` equals the training-derived union rare set; validation/test labels did
|
||||
not influence rare identity.
|
||||
|
||||
---
|
||||
|
||||
## F. Validation metric recomputation
|
||||
|
||||
Recomputed from raw artifacts (no report numbers copied):
|
||||
|
||||
- seed-level primary score = mean over eligible clients of `val_per_client.csv`
|
||||
`rare_class_macro_f1`; equals `val_selection.json` per-client-mean to **max diff 0.0** for all
|
||||
15 runs; all 10 clients eligible in every seed.
|
||||
- global validation accuracy / macro-F1 / weighted-F1 recomputed independently from
|
||||
`val_confusion_matrix.json` with the production formulas: **max diff 0.0** versus
|
||||
`val_global_metrics.json`.
|
||||
|
||||
---
|
||||
|
||||
## G. Pooled-selection recomputation
|
||||
|
||||
Computed from the independently recomputed seed-level scores (not from the report). Pooled = mean
|
||||
of the three seed means; std sample (`ddof=1`).
|
||||
|
||||
| beta | pooled mean | sample std | recomputed selected |
|
||||
|---|---|---|---|
|
||||
| 0.00 | 0.778094 | 0.050963 | |
|
||||
| 0.25 | 0.790483 | 0.055100 | |
|
||||
| **0.50** | **0.809081** | 0.058735 | **yes** |
|
||||
| 0.75 | 0.791530 | 0.064974 | |
|
||||
| 1.00 | 0.804306 | 0.054803 | |
|
||||
|
||||
- Recompute **matches the report** to formatting precision.
|
||||
- `argmax` = **0.50** (no tie; top-two gap ≈ 4.8e-3). `tie_break_used=false`.
|
||||
- Per-seed diagnostics reproduce **0.50 / 1.00 / 1.00**.
|
||||
|
||||
---
|
||||
|
||||
## H. Validation / test leakage audit
|
||||
|
||||
- `run_baseline.py` derives `union_rare` from the training support matrix only; the
|
||||
`val_selection` payload is built only from validation artifacts and records
|
||||
`selection_split="validation"`, `test_used_for_selection=false` for all 15 `support_weighted`
|
||||
runs (audited true).
|
||||
- `analyze_multiseed_beta.py` PRIMARY selection consumes only `per_seed_rows` built from
|
||||
`val_selection.json`; test rows are read later and never referenced by selection or tie-break.
|
||||
No code path sorts/filters candidates by a test field.
|
||||
- The audit's own `beta_star = 0.50` was reproduced from validation artifacts alone.
|
||||
|
||||
Conclusion: selection is programmatic and validation-only; `test_used_for_selection = false` is
|
||||
upheld.
|
||||
|
||||
---
|
||||
|
||||
## I. Final test recomputation
|
||||
|
||||
`beta_star = 0.50` fixed from pooled validation; only then test metrics read. Recomputed from
|
||||
`test_global_metrics.json` per seed (and cross-checked byte-for-byte against `test_descriptive.csv`,
|
||||
max diff 0.0), averaged across seeds with sample std:
|
||||
|
||||
| metric | selected beta 0.50 (mean ± std) | original_protean (mean ± std) |
|
||||
|---|---|---|
|
||||
| rare-class macro-F1 | 0.896738 ± 0.014470 | 0.879901 ± 0.023121 |
|
||||
| rare-class recall | 0.959701 ± 0.001025 | 0.961214 ± 0.002144 |
|
||||
| macro-F1 | 0.853150 ± 0.009572 | 0.841405 ± 0.014091 |
|
||||
| accuracy | 0.752274 ± 0.002503 | 0.750955 ± 0.003578 |
|
||||
| weighted-F1 | 0.727467 ± 0.003217 | 0.726047 ± 0.004544 |
|
||||
|
||||
Matches the report exactly. No superiority claim is made.
|
||||
|
||||
---
|
||||
|
||||
## J. `beta = 0` equivalence audit
|
||||
|
||||
For seeds 0, 1, 2, `original_protean` vs `support_weighted beta=0`:
|
||||
|
||||
| property | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| same manifest / split / hashes | yes | yes | yes |
|
||||
| round-0 snapshot byte-identical | yes | yes | yes |
|
||||
| support/characterization/rare_unseen identical | yes | yes | yes |
|
||||
| independent aggregation diff | 4.4e-16 | 4.4e-16 | 4.4e-16 |
|
||||
| history rare-macro-F1 round-0 diff | 0.0 | 0.0 | 0.0 |
|
||||
| history rare-macro-F1 max diff (rounds ≥ 1) | 3.22e-2 | 5.04e-2 | 2.72e-2 |
|
||||
| test metric max diff | 5.22e-3 | 5.32e-3 | 2.74e-4 |
|
||||
| classification | **B** | **B** | **B** |
|
||||
|
||||
Class **B** (semantically equivalent + floating-point divergence); **no semantic discrepancy**.
|
||||
Equivalence remains conditional on the shared `S_j` reproduction assumption.
|
||||
|
||||
---
|
||||
|
||||
## K. Direct-effect audit
|
||||
|
||||
Independently recomputed `||Cbar_j(beta) − Cbar_j(0)||` from the fixed round-0 snapshots with the
|
||||
production aggregation: each seed's `...__prototype_effect.csv` is produced by
|
||||
`analyze_prototype_effect.py` from the snapshot and the cross-seed pooled file aggregates it. Mean
|
||||
displacement at `beta=1.0` is **1.385** (monotone: 0.497 / 0.875 / 1.163 / 1.385 for
|
||||
beta 0.25/0.50/0.75/1.00). Displacement is a direct-aggregation measure only; no detection
|
||||
causality is inferred.
|
||||
|
||||
---
|
||||
|
||||
## L. Client-level audit
|
||||
|
||||
Deltas recomputed from `val_per_client.csv` vs `client_validation_detail.csv` (via the `beta=0`
|
||||
base): improve/degrade/unchanged at the selected `beta=0.50` = 4/6/0 (seed 0, mean +0.0021),
|
||||
6/3/1 (seed 1, mean +0.0850), 4/2/4 (seed 2, mean +0.0058). `N_ij` vs ΔF1 is non-monotonic. Client
|
||||
ids are per-seed realisations, not pooled.
|
||||
|
||||
---
|
||||
|
||||
## M. Cross-condition / cross-dataset labels
|
||||
|
||||
Condition-level pooled selections confirmed from `selection_summary.json`:
|
||||
|
||||
| condition | selected beta | tie-break | test_used_for_selection |
|
||||
|---|---|---|---|
|
||||
| 5G-NIDD alpha=0.75 (this session) | 0.50 | no | false |
|
||||
| 5G-NIDD alpha=0.50 (Session 9) | 1.00 | no | false |
|
||||
| X-IIoTID alpha=0.75 (Session 8) | 0.50 | no | false |
|
||||
|
||||
The report makes no causal, monotone or generalising claim and explicitly states the selections
|
||||
differ descriptively across conditions.
|
||||
|
||||
---
|
||||
|
||||
## N. Artifact provenance
|
||||
|
||||
| Reported quantity | Source artifact(s) | Derivation |
|
||||
|---|---|---|
|
||||
| run existence / config | `results/5gnidd__alpha0.75__*/metadata.json` | direct read |
|
||||
| manifest hashes / reproduction | `artifacts/partitions/5gnidd__alpha0.75__seed{0,1,2}.json` | module + `reproduce_train_assignments` |
|
||||
| pairing proof | `artifacts/analysis/..._prototype_snapshot.json` | sha256 across 6 runs/seed |
|
||||
| seed-level validation score | `results/.../val_per_client.csv` + `val_selection.json` | mean of per-client rare macro-F1 |
|
||||
| validation global metrics | `results/.../val_confusion_matrix.json` | independent numpy recompute |
|
||||
| pooled mean/std, `beta_star` | recomputed seed scores | mean/`stdev`, argmax |
|
||||
| test metrics | `results/.../test_global_metrics.json` + `..._test_descriptive.csv` | independent recompute + mean/std |
|
||||
| partition characterization | manifests + `..._partition_heterogeneity_*.csv` | independent numpy recompute |
|
||||
| beta=0 equivalence | original vs beta0 artifacts/snapshots | sha256 + independent aggregation + history diff |
|
||||
| direct-effect displacement | snapshots + `..._prototype_effect_pooled.csv` | production aggregation |
|
||||
| client improve/degrade | `results/.../val_per_client.csv` + `..._client_validation_detail.csv` | deltas vs beta=0 |
|
||||
| rare-class identity | `results/.../rare_unseen.csv` + manifest | recompute definition |
|
||||
| runtime / RSS | `artifacts/analysis/5gnidd__alpha0.75__multiseed__runtime.csv` (source `/tmp/opencode/5gnidd_alpha075_runtime.csv`) | parse `/usr/bin/time -v`; sum 69,064 s = 19.18 h |
|
||||
| cross-alpha table | `5gnidd__cross_alpha__*` (built from alpha0.5 + alpha0.75 multiseed artifacts) | read per-condition artifacts |
|
||||
|
||||
---
|
||||
|
||||
## O. Deviations / corrections found
|
||||
|
||||
**None** affecting science. One audit-script (not repository) comparison bug was self-corrected
|
||||
during the audit: the first version of the audit compared per-seed `test_descriptive.csv` rows to
|
||||
pooled means; the check was corrected to compare per-seed values against per-seed
|
||||
`test_global_metrics.json` (max diff then 0.0). No repository artifact, report number, or selection
|
||||
was affected. No training/aggregation/preprocessing source file was modified.
|
||||
|
||||
Informational note (not a defect): two runner instances were accidentally started and immediately
|
||||
killed before any artifact was written; the final run directory contains exactly the 18 intended
|
||||
runs (18/18, no duplicates).
|
||||
|
||||
---
|
||||
|
||||
## P. Final audit verdict
|
||||
|
||||
**PASS.**
|
||||
|
||||
- Run inventory: **PASS** (18/18; no duplicates/partials; no alpha=0.25 run; no legacy 5G-NIDD
|
||||
alpha=0.75 run).
|
||||
- Pairing / manifest reproduction: **PASS** (hash-identical; raw-data reproduction exact incl.
|
||||
partition hash).
|
||||
- Partition characterization: **PASS** (4/27, 5 missing, ratios 3.70/11.36/3.87, unseen 2/2/0).
|
||||
- Rare/unseen integrity: **PASS** (0 mismatches over 1,620 cells).
|
||||
- Validation recomputation (seed-level and global): **PASS** (exact).
|
||||
- Pooled selection: **PASS**; independent `beta_star = 0.50` using validation only.
|
||||
- Validation/test leakage: **PASS**.
|
||||
- Test summaries: **PASS** (exact).
|
||||
- beta=0 equivalence: **PASS**, class **B** for all seeds.
|
||||
- Direct-effect and client-level analyses: **PASS** (consistent with snapshots/artifacts).
|
||||
- Cross-condition labels: **PASS** (5G-NIDD a0.75 = 0.50 vs a0.50 = 1.00 vs X-IIoTID a0.75 = 0.50;
|
||||
no causal claim).
|
||||
- Source/data integrity: **PASS**; `planningawal/` unchanged; raw SHA-256 unchanged;
|
||||
`equal_available` absent; pytest 75 passed.
|
||||
|
||||
Session 10 is **safe to checkpoint** with the documented limitations (single new condition, three
|
||||
project-defined seeds, weak effect relative to seed variance, `S_j` reproduction assumption, no
|
||||
global-optimum claim).
|
||||
Reference in new issue
Block a user