experiment: validate 5gnidd beta selection at alpha 0.50
This commit is contained in:
willtanoe committed
1 parent
2d27abc350
commit
a7f2b0049e
28 files changed
+38378
-256
No files matched your search
+231
-256
@@ -3,26 +3,26 @@
|
||||
This is the canonical continuation point for the next agent session. It is written to be
|
||||
readable without prior chat history, terminal scrollback, or git history.
|
||||
|
||||
Last updated: 2026-09-26 (Session 8). Session 8 type: **protocol-scaling /
|
||||
non-IID-severity experiment**. Session 7 completed X-IIoTID `alpha=0.25`; Session 6 completed
|
||||
`alpha=0.50`. Session 8 completed X-IIoTID **`alpha=0.75`** for seeds `{0,1,2}` across the five
|
||||
frozen beta candidates plus `original_protean`, using the **same pooled-across-seed
|
||||
validation-only** selection rule, and produced the first **three-alpha (0.25/0.50/0.75)
|
||||
comparison** for X-IIoTID. **5G-NIDD was not run.** No commit or push was made.
|
||||
Last updated: 2026-09-28 (Session 9). Session 9 type: **first cross-dataset (5G-NIDD)
|
||||
beta-validation condition**. Sessions 6–8 completed the X-IIoTID three-alpha study
|
||||
(`alpha=0.25/0.50/0.75`, seeds `{0,1,2}`). Session 9 ran **only** 5G-NIDD `alpha=0.50` for seeds
|
||||
`{0,1,2}` across the five frozen beta candidates plus `original_protean`, using the same
|
||||
pooled-across-seed validation-only selection rule. **5G-NIDD alpha=0.25 and alpha=0.75 were NOT
|
||||
run.** No commit or push was made.
|
||||
|
||||
> **Audit status (Session 8).** Independent audit verdict: **PASS** (`docs/research/SESSION8_AUDIT.md`).
|
||||
> `beta_star(alpha=0.75) = 0.50` confirmed from validation only; no corrections required; run
|
||||
> inventory 18/18; pairing hash-identical and reproduced from raw data; leakage audit PASS;
|
||||
> `beta=0` class **B**; source/data integrity PASS; pytest 75 passed. **Session 8 is safe to
|
||||
> checkpoint.**
|
||||
> **Audit status (Session 9).** Independent audit verdict: **PASS**
|
||||
> (`docs/research/SESSION9_AUDIT.md`). `beta_star(5gnidd, alpha=0.50) = 1.00` confirmed from
|
||||
> validation only; reuse decision resolved; run inventory 18/18; pairing hash-identical and
|
||||
> reproduced from raw data; rare/unseen 0 mismatches over 270 cells; leakage audit PASS; `beta=0`
|
||||
> class **B**; source/data integrity PASS; pytest 75 passed. **Session 9 is safe to checkpoint.**
|
||||
|
||||
> **Headline (Session 8).** PRIMARY pooled-validation selected beta for X-IIoTID `alpha=0.75`
|
||||
> = **0.50** (validation only; per-seed diagnostics 0.50 / 0.50 / 0.00). The pooled validation
|
||||
> curve peaks at 0.50 and declines both sides, but the total between-beta spread (≈ 0.0081) is
|
||||
> much smaller than the between-seed std (≈ 0.045). `beta=0` is class **B** for all three seeds.
|
||||
> Realized alpha=0.75 partitions are the mildest of the three (3/27 cells with `|S_j| < M`,
|
||||
> 4 missing cells, client-ratio mean 4.67). **Three-alpha pooled selections: 0.25 → 0.00,
|
||||
> 0.50 → 0.75, 0.75 → 0.50 — not monotone in alpha.**
|
||||
> **Headline (Session 9).** PRIMARY pooled-validation selected beta for 5G-NIDD `alpha=0.50` =
|
||||
> **1.00** (validation only; per-seed diagnostics 1.00 / 0.00 / 1.00). The pooled validation curve
|
||||
> is **non-monotone** (0.687 → 0.645 → 0.675 → 0.688 → 0.692) and the total between-beta spread
|
||||
> (≈ 0.047) is comparable to the between-seed std (0.053–0.139), so the selection is statistically
|
||||
> weak. `beta=0` is class **B** for all three seeds. The pre-existing seed-0 baseline was **not
|
||||
> reused** (missing validation artifacts); the fresh rerun reproduces its training outputs
|
||||
> byte-identically. X-IIoTID alpha=0.50 selected 0.75; the two datasets differ descriptively.
|
||||
|
||||
---
|
||||
|
||||
@@ -30,8 +30,8 @@ comparison** for X-IIoTID. **5G-NIDD was not run.** No commit or push was made.
|
||||
|
||||
Build the code and experiment infrastructure for a thesis study that modifies **only** the
|
||||
server-side prototype aggregation of PROTEAN (Chennoufi et al., 2025), a federated
|
||||
intrusion-detection framework. The mechanism weights each contributing client's class
|
||||
prototype by its class sample support:
|
||||
intrusion-detection framework. The mechanism weights each contributing client's class prototype
|
||||
by its class sample support:
|
||||
|
||||
```
|
||||
Cbar_j(beta) = ( sum_{i in S_j} N_ij^beta * C_ij ) / ( sum_{i in S_j} N_ij^beta )
|
||||
@@ -49,287 +49,267 @@ bitwise identical to the baseline (see §7).
|
||||
- `planningawal/` is immutable. Re-verified unchanged this session: `git status --short
|
||||
planningawal/` and `git diff -- planningawal/` are both empty.
|
||||
- Raw datasets re-verified unchanged (SHA-256):
|
||||
`~/datasets/X-IIoTID/X-IIoTID-dataset.csv`
|
||||
`7b9290057ee42e784da3c0d84b781815502c9205c74175c96374e71a5ffd98a0`;
|
||||
`~/datasets/5G-NIDD/5G-NIDD.csv`
|
||||
`fa36f80859585f474504ca69eb951a079b35145537976c86b00aae3aab46ee59`.
|
||||
- Frozen baseline (`original_protean`) source unchanged; no `equal_available` mode added;
|
||||
beta candidate set unchanged (`{0, .25, .5, .75, 1}`). No tracked source/test/config file
|
||||
differs from HEAD.
|
||||
`fa36f80859585f474504ca69eb951a079b35145537976c86b00aae3aab46ee59`;
|
||||
`~/datasets/X-IIoTID/X-IIoTID-dataset.csv`
|
||||
`7b9290057ee42e784da3c0d84b781815502c9205c74175c96374e71a5ffd98a0`.
|
||||
- Frozen baseline (`original_protean`) source unchanged; no `equal_available` mode added; beta
|
||||
candidate set unchanged (`{0, .25, .5, .75, 1}`). No tracked source/test/config file differs
|
||||
from HEAD. The only modified tracked file is this handoff.
|
||||
|
||||
## 3. Session 8 objective and outcome
|
||||
## 3. Session 9 objective and outcome
|
||||
|
||||
Objective: evaluate the final X-IIoTID non-IID severity (`alpha=0.75`, seeds `{0,1,2}`) across
|
||||
the five beta candidates using the exact pooled-validation protocol, then compare alpha=0.25,
|
||||
0.50 and 0.75 using realized partition characteristics. All 18 runs (15 `support_weighted` +
|
||||
3 `original_protean`) completed with rc=0.
|
||||
Objective: begin the 5G-NIDD phase conservatively — Stage A verification, then (if clean) **one**
|
||||
condition: 5G-NIDD `alpha=0.50`, seeds `{0,1,2}`, `original_protean` + five betas.
|
||||
|
||||
Outcome: **PRIMARY pooled-validation selected beta = 0.50** (validation only);
|
||||
`test_used_for_selection = false` for all 15 runs. Pooled means: 0.00 → 0.471706,
|
||||
0.25 → 0.474368, 0.50 → 0.476749, 0.75 → 0.472541, 1.00 → 0.468605 (peak at 0.50). Between-seed
|
||||
std 0.0447–0.0553; between-beta spread ≈ 0.0081, so the effect is small relative to seed
|
||||
variance.
|
||||
Stage A result: raw hash valid; all nine 5G-NIDD manifests (alpha 0.25/0.50/0.75 × seeds) reproduce
|
||||
exactly from raw data; label semantics and rare/unseen logic valid; pre-existing seed-0 baseline
|
||||
not reusable. Gate = **GO**.
|
||||
|
||||
Reports: `docs/research/ALPHA075_BETA_VALIDATION.md`,
|
||||
`docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md`, audit `docs/research/SESSION8_AUDIT.md`.
|
||||
Outcome: 18/18 runs rc=0. **PRIMARY pooled-validation selected beta = 1.00** (validation only);
|
||||
`test_used_for_selection = false` for all 15 runs. Pooled means: 0.00 → 0.687230,
|
||||
0.25 → 0.645078, 0.50 → 0.674669, 0.75 → 0.688368, 1.00 → 0.692370. Reports:
|
||||
`docs/research/5GNIDD_ALPHA050_BETA_VALIDATION.md`, audit `docs/research/SESSION9_AUDIT.md`.
|
||||
|
||||
## 4. Condition and pairing
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| dataset / alpha / seeds | `xiiotid` / 0.75 / `{0,1,2}` |
|
||||
| dataset / alpha / seeds | `5gnidd` / 0.50 / `{0,1,2}` |
|
||||
| runs | 15 `support_weighted` + 3 `original_protean`, all new this session |
|
||||
| training | 10 clients, 10 rounds, 3 local epochs, batch 256, lr 1e-3, mu 0.1, lambda 1.0, Adam |
|
||||
| code revision | unchanged from HEAD `ff68d380e23859d21c07c8a3a7b56b1b7f06c91b`; only docs/analysis/scripts added |
|
||||
| code revision | unchanged from HEAD `2d27abc35026456de765c929dbcf0503a72ac572`; only docs/analysis added |
|
||||
|
||||
Manifests (reused, never regenerated; one content hash per seed across all six runs; each
|
||||
reproduced from the raw dataset with split/`N_ij`/partition hash verified):
|
||||
|
||||
| seed | manifest | content hash | partition hash | split hash | rng seed |
|
||||
| seed | manifest content hash | partition hash | split hash | rng seed | round-0 snapshot SHA-256 |
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | `artifacts/partitions/xiiotid__alpha0.75__seed0.json` | `759f7e50…` | `473b1c4a…` | `d0acc0c1…` | 957653162 |
|
||||
| 1 | `artifacts/partitions/xiiotid__alpha0.75__seed1.json` | `4a50ba21…` | `a0ec9e9c…` | `880a9f82…` | 762050403 |
|
||||
| 2 | `artifacts/partitions/xiiotid__alpha0.75__seed2.json` | `922b7bb1…` | `00345255…` | `8e0db0be…` | 843543396 |
|
||||
| 0 | `0732582cf4510089…` | `8352d4ff…` | `18f89f48…` | 673621352 | `eb0839f2595727…` |
|
||||
| 1 | `f43906e111a8342c…` | `0815c91e…` | `3d05b7fe…` | 1403944623 | `6cbd04e0dd97bb…` |
|
||||
| 2 | `fe8b4fc6f0486cf4…` | `fdcf388e…` | `fc758d06…` | 633461056 | `f012a02b543b92…` |
|
||||
|
||||
Pairing proof: round-0 snapshot byte-identical across the five betas **and** `original_protean`
|
||||
within each seed — seed 0 `8c477df6…`, seed 1 `41f0187c…`, seed 2 `be2ad5fb…`.
|
||||
`train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv` identical across each seed's
|
||||
runs. `manifest_reload_ok = true`, `smoke = false` everywhere.
|
||||
Splits: train/validation/test = 851,123 / 121,589 / 243,177 for every seed (10 clients, 9 classes,
|
||||
`Benign`). `train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv` identical across each
|
||||
seed's runs. `manifest_reload_ok = true`, `smoke = false` everywhere. Pairing proven by content
|
||||
hashes.
|
||||
|
||||
## 5. Validation results (USED FOR SELECTION) and pooled selection
|
||||
|
||||
Primary metric: validation per-client-mean rare-class macro-F1. All 10 clients eligible in
|
||||
every seed. Full tables:
|
||||
`artifacts/analysis/xiiotid__alpha0.75__multiseed__validation_per_seed.csv`.
|
||||
Primary metric: validation per-client-mean rare-class macro-F1. All 10 clients eligible in every
|
||||
seed. Full tables: `artifacts/analysis/5gnidd__alpha0.5__multiseed__validation_per_seed.csv`.
|
||||
|
||||
Per-seed validation per-client-mean rare-class macro-F1:
|
||||
|
||||
| beta | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| 0.00 | 0.496803 | 0.408276 | 0.510038 |
|
||||
| 0.25 | 0.496892 | 0.421386 | 0.504827 |
|
||||
| 0.50 | 0.504388 | 0.422919 | 0.502942 |
|
||||
| 0.75 | 0.486868 | 0.421053 | 0.509702 |
|
||||
| 1.00 | 0.479146 | 0.419534 | 0.507136 |
|
||||
| 0.00 | 0.738597 | 0.633243 | 0.689849 |
|
||||
| 0.25 | 0.661461 | 0.573717 | 0.700055 |
|
||||
| 0.50 | 0.764487 | 0.563388 | 0.696131 |
|
||||
| 0.75 | 0.750661 | 0.543319 | 0.771124 |
|
||||
| 1.00 | 0.768877 | 0.531764 | 0.776468 |
|
||||
|
||||
Pooled across seeds (`...__validation_pooled.csv`):
|
||||
|
||||
| beta | pooled mean | std (sample) | min | max | selected |
|
||||
|---|---|---|---|---|---|
|
||||
| 0.00 | 0.471706 | 0.055329 | 0.408276 | 0.510038 | |
|
||||
| 0.25 | 0.474368 | 0.046055 | 0.421386 | 0.504827 | |
|
||||
| **0.50** | **0.476749** | 0.046624 | 0.422919 | 0.504388 | **yes** |
|
||||
| 0.75 | 0.472541 | 0.046028 | 0.421053 | 0.509702 | |
|
||||
| 1.00 | 0.468605 | 0.044742 | 0.419534 | 0.507136 | |
|
||||
| 0.00 | 0.687230 | 0.052726 | 0.633243 | 0.738597 | |
|
||||
| 0.25 | 0.645078 | 0.064743 | 0.573717 | 0.700055 | |
|
||||
| 0.50 | 0.674669 | 0.102253 | 0.563388 | 0.764487 | |
|
||||
| 0.75 | 0.688368 | 0.126032 | 0.543319 | 0.771124 | |
|
||||
| **1.00** | **0.692370** | 0.139140 | 0.531764 | 0.776468 | **yes** |
|
||||
|
||||
- **PRIMARY selected beta (`beta_star_alpha075`) = 0.50**, max pooled validation
|
||||
per-client-mean rare-class macro-F1; **no tie-break used**; selection is validation-only.
|
||||
- **SECONDARY per-seed diagnostics only:** seed 0 → 0.50, seed 1 → 0.50, seed 2 → 0.00.
|
||||
- **PRIMARY selected beta (`beta_star_5gnidd_alpha050`) = 1.00**, max pooled validation
|
||||
per-client-mean rare-class macro-F1; **no tie-break used**; validation-only.
|
||||
- **SECONDARY per-seed diagnostics only:** seed 0 → 1.00, seed 1 → 0.00, seed 2 → 1.00.
|
||||
- Between-beta spread ≈ 0.047; between-seed std 0.053–0.139 → statistically weak, descriptive.
|
||||
|
||||
## 6. Final test evaluation of the pooled-selected beta
|
||||
|
||||
`beta_star = 0.50` was fixed from pooled validation before test was read.
|
||||
`beta_star = 1.00` was fixed from pooled validation before test was read.
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.511889 | 0.624144 | 0.679704 | 0.605246 ± 0.085489 |
|
||||
| rare-class recall | 0.803887 | 0.876726 | 0.865681 | 0.848765 ± 0.039256 |
|
||||
| macro-F1 | 0.741977 | 0.761752 | 0.769179 | 0.757636 ± 0.014060 |
|
||||
| accuracy | 0.914365 | 0.932611 | 0.937352 | 0.928110 ± 0.012137 |
|
||||
| weighted-F1 | 0.935566 | 0.948344 | 0.951740 | 0.945216 ± 0.008528 |
|
||||
| rare-class macro-F1 | 0.880962 | 0.840075 | 0.894914 | 0.871984 ± 0.028501 |
|
||||
| rare-class recall | 0.974355 | 0.968341 | 0.958616 | 0.967104 ± 0.007942 |
|
||||
| macro-F1 | 0.850470 | 0.815941 | 0.849378 | 0.838596 ± 0.019627 |
|
||||
| accuracy | 0.742838 | 0.744026 | 0.742907 | 0.743257 ± 0.000667 |
|
||||
| weighted-F1 | 0.716950 | 0.720106 | 0.717530 | 0.718195 ± 0.001680 |
|
||||
|
||||
`original_protean` for the same seeds/condition:
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.521540 | 0.614061 | 0.676068 | 0.603890 ± 0.077764 |
|
||||
| rare-class recall | 0.816440 | 0.883238 | 0.869835 | 0.856504 ± 0.035338 |
|
||||
| macro-F1 | 0.742471 | 0.747395 | 0.765680 | 0.751849 ± 0.012228 |
|
||||
| accuracy | 0.913254 | 0.913898 | 0.934576 | 0.920576 ± 0.012129 |
|
||||
| weighted-F1 | 0.932563 | 0.935937 | 0.948908 | 0.939136 ± 0.008629 |
|
||||
| rare-class macro-F1 | 0.871964 | 0.921136 | 0.848525 | 0.880542 ± 0.037058 |
|
||||
| rare-class recall | 0.974314 | 0.968566 | 0.975859 | 0.972913 ± 0.003843 |
|
||||
| macro-F1 | 0.836881 | 0.865437 | 0.824867 | 0.842395 ± 0.020840 |
|
||||
| accuracy | 0.734416 | 0.747377 | 0.750133 | 0.743975 ± 0.008393 |
|
||||
| weighted-F1 | 0.708765 | 0.722802 | 0.724622 | 0.718730 ± 0.008678 |
|
||||
|
||||
Descriptive difference for test rare-class macro-F1 (`beta_star` − `original_protean`) is
|
||||
−0.009651 / +0.010083 / +0.003636 (mean +0.001356). Test is descriptive only; no superiority
|
||||
or generalisation claimed.
|
||||
+0.008998 / −0.081061 / +0.046389 (mean −0.008558). Test is descriptive only; no superiority or
|
||||
generalisation claimed. Note 5G-NIDD test weighted-F1 (`≈0.718`) sits **below** accuracy
|
||||
(`≈0.744`) — the opposite of X-IIoTID — because the high-support classes are the harder ones for
|
||||
nearest-prototype inference.
|
||||
|
||||
## 7. `beta = 0` equivalence result (all three seeds)
|
||||
|
||||
For seeds 0, 1, 2, `support_weighted beta=0` vs frozen `original_protean`:
|
||||
|
||||
- Same manifest/split/hashes: **yes**.
|
||||
- Round-0 snapshot byte-identical: **yes** (`8c477df6…`, `41f0187c…`, `be2ad5fb…`).
|
||||
- `train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv`: **identical**.
|
||||
- Independent aggregation (`ΣC/n` vs `(1/n)ΣC`): 4.4e-16 / 4.4e-16 / 2.2e-16.
|
||||
- History rare-macro-F1 max diff (rounds ≥ 1): 1.10e-2 / 8.12e-3 / 3.64e-3; test metric max
|
||||
diff 4.33e-3 / 1.14e-3 / 0.85e-3.
|
||||
- Classification: **B — mathematically equivalent + floating-point numerical divergence; no
|
||||
semantic discrepancy** in any seed. Machine-readable:
|
||||
`artifacts/analysis/xiiotid__alpha0.75__multiseed__beta0_equivalence.csv`.
|
||||
- Same manifest/split/hashes, round-0 snapshot byte-identical, support/char/rare_unseen
|
||||
identical: **yes** for all three.
|
||||
- Independent aggregation (`ΣC/n` vs `(1/n)ΣC`): 4.4e-16 / 4.4e-16 / 4.4e-16.
|
||||
- History rare-macro-F1 max diff (rounds ≥ 1): 1.93e-2 / 6.01e-2 / 2.60e-2; test metric max diff
|
||||
2.33e-3 / 1.86e-4 / 8.70e-3.
|
||||
- Classification: **B** in all seeds — semantic equivalence + floating-point divergence; no
|
||||
semantic discrepancy. Machine-readable:
|
||||
`artifacts/analysis/5gnidd__alpha0.5__multiseed__beta0_equivalence.csv`.
|
||||
|
||||
Conditional on the shared `S_j` reproduction assumption; not proof `beta=0` reproduces original
|
||||
PROTEAN.
|
||||
|
||||
## 8. Partition characterization (realized)
|
||||
## 8. Partition characterization (realized) and dataset structure
|
||||
|
||||
`|S_j| < M` class-condition cells, missing client-class cells, and client-size ratios
|
||||
(training split):
|
||||
5G-NIDD `alpha=0.50` (training split), `M = 10`:
|
||||
|
||||
| condition | `|S_j| < M` cells | missing client-class cells | client-size ratio (seeds) | union unseen | mean class CV |
|
||||
|---|---|---|---|---|---|
|
||||
| alpha=0.25 | **19 / 27** | **38** | 15.6, 20.8, 7.8 | 6/6/5 | 1.442 |
|
||||
| alpha=0.50 | 7 / 27 | 9 | 5.1, 7.2, 9.9 | 3/3/1 | 1.183 |
|
||||
| alpha=0.75 | **3 / 27** | **4** | 4.15, 5.07, 4.78 | 2/1/0 | 0.881 |
|
||||
| seed | `|S_j| < M` cells | missing client-class cells | client-size ratio | union rare | union unseen | max unseen/client |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0 | 2 / 9 | 2 | 11.63 | 6 | 2 | 1 |
|
||||
| 1 | 0 / 9 | 0 | 4.43 | 5 | 0 | 0 |
|
||||
| 2 | 2 / 9 | 2 | 12.72 | 6 | 2 | 1 |
|
||||
| **sum/mean** | **4 / 27** | **4** | mean **9.59** | 6/5/6 | 2/0/2 | 1 |
|
||||
|
||||
Union rare classes per seed: alpha=0.75 = 4/5/6; all clients have exactly two rare classes.
|
||||
Full per-class `N_ij`, CV and max/min in `docs/research/ALPHA075_BETA_VALIDATION.md` §C and
|
||||
`artifacts/analysis/xiiotid__cross_alpha__partition_heterogeneity_*.csv`. Heterogeneity
|
||||
decreases with alpha overall but is **not uniformly monotone per seed** (alpha=0.75 seed 0
|
||||
ratio 4.15 < seed 2's 4.78; alpha=0.50 seed 2 ratio 9.9 > alpha=0.25 seed 2's 7.8), so
|
||||
interpretation uses realized partitions, not alpha alone.
|
||||
All 10 clients have exactly two rare classes and are eligible in every seed; mean class-support
|
||||
CV = **1.225**. Full tables:
|
||||
`artifacts/analysis/5gnidd__alpha0.5__partition_heterogeneity_*.csv` and
|
||||
`...__partition_characterization.csv`.
|
||||
|
||||
### Comparison with X-IIoTID alpha=0.50 (descriptive)
|
||||
|
||||
| statistic | 5G-NIDD a0.50 | X-IIoTID a0.50 |
|
||||
|---|---|---|
|
||||
| pooled-selected beta | **1.00** | **0.75** |
|
||||
| pooled validation mean range (spread) | 0.645–0.692 (0.047) | 0.544–0.579 (0.035) |
|
||||
| pooled validation std range | 0.053–0.139 | 0.075–0.100 |
|
||||
| `|S_j| < M` cells (sum/27) | **4/27** | 7/27 |
|
||||
| missing client-class cells | **4** | 9 |
|
||||
| client-size ratio mean | **9.59** | 7.41 |
|
||||
| mean class-support CV | 1.225 | 1.183 |
|
||||
| union rare per seed | 6/5/6 | 6/7/7 |
|
||||
| union unseen per seed | 2/0/2 | 3/3/1 |
|
||||
| mean displacement ‖Cbar(1)−Cbar(0)‖ | 2.340 | 2.099 |
|
||||
| selected-beta client improve/degrade | 6/3, 3/5, 7/2 | 7/3, 8/1, 8/1 |
|
||||
|
||||
5G-NIDD has fewer missing class cells but more imbalanced client sizes; the selected beta differs
|
||||
descriptively. No causal or monotone claim; only two datasets, one condition each.
|
||||
|
||||
## 9. Direct prototype-effect findings (fixed round-0 snapshots, no retraining)
|
||||
|
||||
`||Cbar_j(1) - Cbar_j(0)||`, cross-seed mean, alpha=0.25 vs 0.50 vs 0.75 (mean across classes
|
||||
3.422 / 2.099 / **1.305**):
|
||||
`||Cbar_j(1) − Cbar_j(0)||`, cross-seed mean (std), 5G-NIDD alpha=0.50:
|
||||
|
||||
| class | a25 | a50 | a75 |
|
||||
|---|---|---|---|
|
||||
| Tampering | 4.248 | 2.354 | 1.934 |
|
||||
| Exfiltration | 4.316 | 2.020 | 1.448 |
|
||||
| Exploitation | 3.464 | 2.344 | 1.539 |
|
||||
| Lateral_movement | 3.726 | 2.724 | 1.369 |
|
||||
| C&C | 4.020 | 1.694 | 1.284 |
|
||||
| Weaponization | 3.124 | 2.561 | 1.472 |
|
||||
| Reconnaissance | 3.339 | 1.002 | 0.983 |
|
||||
| Normal | 1.997 | 1.557 | 0.903 |
|
||||
| RDOS | 2.566 | 2.631 | 0.814 |
|
||||
| class | mean disp | std |
|
||||
|---|---|---|
|
||||
| UDPScan | 3.461 | 2.011 |
|
||||
| HTTPFlood | 3.120 | 1.289 |
|
||||
| SYNScan | 2.875 | 0.959 |
|
||||
| SlowrateDoS | 2.763 | 1.454 |
|
||||
| SYNFlood | 2.262 | 0.748 |
|
||||
| TCPConnectScan | 2.131 | 0.213 |
|
||||
| Benign | 1.690 | 0.699 |
|
||||
| UDPFlood | 1.598 | 0.428 |
|
||||
| ICMPFlood | 1.159 | 0.093 |
|
||||
|
||||
Displacement tracks realized support imbalance and is a direct-aggregation measure only, not a
|
||||
detection result. Outputs:
|
||||
`artifacts/analysis/xiiotid__alpha0.75__multiseed__prototype_effect_pooled.csv` and per-seed
|
||||
`...__seed{0,1,2}__support_weighted__beta0__prototype_snapshot__prototype_effect.csv`.
|
||||
Mean across classes **2.340** at beta=1.0 (monotone: 0.953 / 1.567 / 2.003 / 2.340 for beta
|
||||
0.25/0.50/0.75/1.00). Seed-sensitive (e.g. UDPScan 1.42 / 3.52 / 5.44). Displacement is a
|
||||
direct-aggregation measure only, not a detection result. Outputs:
|
||||
`...__prototype_effect_pooled.csv` and per-seed `...__prototype_effect.csv` / `...__prototype_weights.csv`.
|
||||
|
||||
## 10. Client-level findings (validation, vs beta = 0)
|
||||
|
||||
Detail: `...__client_validation_detail.csv`; summary: `...__client_validation_summary.csv`; and
|
||||
`artifacts/analysis/xiiotid__cross_alpha__cross_alpha_client_summary.csv`.
|
||||
Detail: `...__client_validation_detail.csv`; summary: `...__client_validation_summary.csv`.
|
||||
|
||||
- At alpha=0.75, the response is smaller and seed-dependent. At the selected `beta=0.50`:
|
||||
6/4/0 (seed 0, mean Δ +0.008), 9/0/1 (seed 1, +0.015), 6/4/0 (seed 2, −0.007; worst client
|
||||
−0.152).
|
||||
- At a common `beta=0.75`: alpha=0.25 = 5/4/1, 5/4/1, 3/5/2 (often negative); alpha=0.50 =
|
||||
7/3/0, 8/1/1, 8/1/1 (majority improve); alpha=0.75 = 3/7/0, 9/0/1, 5/5/0.
|
||||
- `N_ij` vs ΔF1 remains non-monotonic at every alpha. Client ids are per-seed realisations and
|
||||
are not pooled.
|
||||
At the selected `beta=1.00`: **6/3/1** (seed 0, mean Δ +0.030), **3/5/2** (seed 1, mean Δ −0.101;
|
||||
worst client −0.656), **7/2/1** (seed 2, mean Δ +0.087). The response is seed-dependent and
|
||||
`N_ij` vs ΔF1 is non-monotonic. Client ids are per-seed realisations and are not pooled.
|
||||
|
||||
## 11. Three-alpha comparison (RQ3)
|
||||
## 11. Runtime / resources
|
||||
|
||||
| | alpha=0.25 | alpha=0.50 | alpha=0.75 |
|
||||
|---|---|---|---|
|
||||
| pooled-selected beta | **0.00** | **0.75** | **0.50** |
|
||||
| pooled validation curve | monotone decreasing | rises to peak at 0.75 | peaks at 0.50 |
|
||||
| pooled mean range (spread) | 0.532–0.592 (0.060) | 0.544–0.579 (0.035) | 0.469–0.477 (0.008) |
|
||||
| pooled std range | 0.045–0.090 | 0.075–0.100 | 0.045–0.055 |
|
||||
| `|S_j| < M` cells | 19/27 | 7/27 | 3/27 |
|
||||
| mean `||Cbar(1)−Cbar(0)||` | 3.422 | 2.099 | 1.305 |
|
||||
CPU only, no GPU, no swap, exit 0 on all 18 runs. Per-run wall-clock 62–68 min; CPU 734–745 %;
|
||||
peak RSS 10.36 GiB (seed 0), 8.67 GiB (seed 1), 14.04 GiB (seed 2). New compute: 18 runs,
|
||||
69,336 s ≈ **19.26 h** wall-clock, sequential (8 cores). Machine-readable:
|
||||
`artifacts/analysis/5gnidd__alpha0.5__multiseed__runtime.csv`.
|
||||
|
||||
The pooled-selected beta **changes** with the non-IID condition (0.00 → 0.75 → 0.50) but is
|
||||
**not monotone** in alpha. This is documented as a descriptive association accompanied by the
|
||||
realized partition characteristics; **no causal claim** is made. With only three alpha levels,
|
||||
three project-defined seeds, one dataset and no inferential trend test, monotonicity is not
|
||||
claimed.
|
||||
## 12. Methodological decisions / assumptions
|
||||
|
||||
> **Audited label (do not confuse).** Condition-level **pooled** selections are alpha=0.25 →
|
||||
> **0.00**, alpha=0.50 → **0.75**, alpha=0.75 → **0.50**. The value **0.50** is *not* the
|
||||
> alpha=0.50 pooled selection: it is only the Session-5 **seed-0 single-condition** choice, the
|
||||
> Session-6 alpha=0.50 **per-seed diagnostic** for seed 0, and now the alpha=0.75 pooled
|
||||
> selection. See `docs/research/SESSION7_AUDIT.md` §M and `SESSION8_AUDIT.md` §M.
|
||||
|
||||
## 12. Runtime / resources
|
||||
|
||||
CPU only, no GPU, no swap, exit 0 on all 18 runs. Per-run wall-clock 40:00–49:02 (two runs
|
||||
slower under transient host contention); CPU 714–746 %; peak RSS 6.49 GiB (seed 0), 6.91 GiB
|
||||
(seed 1), 7.20 GiB (seed 2). New compute: 18 runs, 44,371 s ≈ **12.33 h** wall-clock,
|
||||
sequential (8 cores). Machine-readable:
|
||||
`artifacts/analysis/xiiotid__alpha0.75__multiseed__runtime.csv`.
|
||||
|
||||
## 13. Methodological decisions / assumptions
|
||||
|
||||
1. **`S_j` participant/denominator rule** (`S_j = {i | N_ij > 0 and valid C_ij}`, equal weights
|
||||
in the baseline) is a **REPRODUCTION ASSUMPTION**; material in 19/27 class-condition cells at
|
||||
alpha=0.25, 7/27 at alpha=0.50, 3/27 at alpha=0.75. Carry this caveat in all reporting.
|
||||
1. **`S_j` participant/denominator rule** (`S_j = {i | N_ij > 0 and valid C_ij}`, equal weights in
|
||||
the baseline) is a **REPRODUCTION ASSUMPTION**; material in only **4/27** class-condition cells
|
||||
at 5G-NIDD alpha=0.50. Carry this caveat in all reporting.
|
||||
2. `N_ij` is class-specific sample support, not prototype quality.
|
||||
3. Beta is one value for all classes/clients; candidates frozen at `{0,.25,.5,.75,1}`.
|
||||
4. **PRIMARY selection = pooled validation across clients and seeds**; per-seed selection is
|
||||
secondary stability diagnostics; test used only after the pooled beta is fixed.
|
||||
5. Three new read-only analysis scripts were added this session (no training/aggregation code
|
||||
changed): `scripts/analyze_partition_heterogeneity.py`,
|
||||
`scripts/verify_beta0_equivalence.py`, `scripts/analyze_cross_alpha.py`.
|
||||
5. No new source scripts this session; analysis used the existing read-only
|
||||
`analyze_partition_heterogeneity.py`, `analyze_prototype_effect.py`, `analyze_multiseed_beta.py`,
|
||||
`verify_beta0_equivalence.py`. Ad-hoc Stage-A/audit helpers live only under
|
||||
`/tmp/opencode/s9/` (outside the repo).
|
||||
6. `lambda=1.0`, Adam, lr 1e-3, batch 256 remain PROJECT DECISIONS. No `equal_available` mode.
|
||||
|
||||
## 14. Unresolved questions / blockers
|
||||
## 13. Unresolved questions / blockers
|
||||
|
||||
1. `S_j` reproduction ambiguity — unresolved; no official PROTEAN code exists.
|
||||
2. **Pooled beta selection remains statistically weak at all three alphas (descriptive only):**
|
||||
the across-seed sample std is comparable to or larger than the between-beta spread. No
|
||||
inferential test was performed.
|
||||
3. `beta = 0` is not bitwise identical to `original_protean`; the documentation decision on how
|
||||
to report this remains open.
|
||||
2. Pooled beta selection remains statistically weak at all measured conditions: across-seed std is
|
||||
comparable to or larger than the between-beta spread. No inferential test was performed.
|
||||
3. `beta = 0` is not bitwise identical to `original_protean`; documentation decision still open.
|
||||
4. Session-2 manifests still store `preprocessing_hash = null` / `config_hash = null`.
|
||||
5. Only alpha=0.25/0.50/0.75, X-IIoTID, three seeds measured. No 5G-NIDD beta result.
|
||||
6. The three-alpha selected-beta sequence (0.00 / 0.75 / 0.50) is **not monotone**; no trend
|
||||
claim is made and no inference has been run.
|
||||
5. Only 5G-NIDD alpha=0.50 (one condition) measured; no 5G-NIDD alpha=0.25/0.75 result; no
|
||||
cross-alpha 5G-NIDD or cross-dataset causal claim.
|
||||
6. The selected beta differs across datasets (5G-NIDD 1.00 vs X-IIoTID 0.75) but no trend or
|
||||
causality is claimed.
|
||||
|
||||
## 15. Git status
|
||||
## 14. Git status
|
||||
|
||||
No commit or push was made. `git status --short` shows only untracked docs/analysis/run
|
||||
artifacts and the three new scripts (`results/` is gitignored; `artifacts/analysis/` is
|
||||
intentionally tracked):
|
||||
No commit or push was made. `HEAD = 2d27abc35026456de765c929dbcf0503a72ac572`, branch `main`,
|
||||
in sync with `origin/main`. `git status --short` shows only untracked docs/analysis artifacts and
|
||||
the modified handoff; `results/` is gitignored:
|
||||
|
||||
```
|
||||
M docs/handoff/SESSION_HANDOFF.md
|
||||
?? docs/research/ALPHA075_BETA_VALIDATION.md
|
||||
?? docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md
|
||||
?? docs/research/SESSION8_AUDIT.md
|
||||
?? scripts/analyze_cross_alpha.py
|
||||
?? scripts/analyze_partition_heterogeneity.py
|
||||
?? scripts/verify_beta0_equivalence.py
|
||||
?? artifacts/analysis/xiiotid__alpha0.75__* (runs' snapshots, effect CSVs, multiseed tables)
|
||||
?? artifacts/analysis/xiiotid__cross_alpha__* (heterogeneity, cross-alpha tables)
|
||||
?? (pre-existing Session-5/6/7 untracked alpha=0.25/0.5 views, unchanged)
|
||||
?? docs/research/5GNIDD_ALPHA050_BETA_VALIDATION.md
|
||||
?? docs/research/SESSION9_AUDIT.md
|
||||
?? artifacts/analysis/5gnidd__alpha0.5__* (46 files: snapshots, multiseed tables, effect/weights,
|
||||
partition characterization, beta0 equivalence, runtime,
|
||||
audit_recomputation)
|
||||
?? (pre-existing Session-5/6/7/8 untracked views, unchanged)
|
||||
```
|
||||
|
||||
No tracked source/test/config file differs from HEAD. `planningawal/` and both raw datasets are
|
||||
unchanged (audit re-verified SHA-256: X-IIoTID `7b929005…`, 5G-NIDD `fa36f808…`).
|
||||
`artifacts/partitions/` unmodified. No 5G-NIDD `support_weighted` run exists. `equal_available`
|
||||
absent. `HEAD = ff68d380e23859d21c07c8a3a7b56b1b7f06c91b`. No commit or push was made.
|
||||
unchanged. `artifacts/partitions/` unmodified. `equal_available` absent. No 5G-NIDD
|
||||
`support_weighted` run before this session; no 5G-NIDD alpha=0.25/0.75 run. The pre-existing
|
||||
`5gnidd__alpha0.5__seed0__original_protean` was overwritten in place by the fresh rerun (backed up
|
||||
to `/tmp/opencode/s9/old_baseline_backup/`).
|
||||
|
||||
## 16. Session 8 audit result
|
||||
## 15. Session 9 audit result
|
||||
|
||||
- Verdict: **PASS** (`docs/research/SESSION8_AUDIT.md`).
|
||||
- Verdict: **PASS** (`docs/research/SESSION9_AUDIT.md`).
|
||||
- Run inventory 18/18, pairing hash-identical, manifests reproduced from raw data.
|
||||
- Validation recomputed exactly; **pooled `beta_star(alpha=0.75) = 0.50`** (validation only).
|
||||
`beta_star` test and `original_protean` test means/stds exact.
|
||||
- Validation recomputed exactly; **pooled `beta_star(5gnidd, alpha=0.50) = 1.00`** (validation
|
||||
only). Selected-beta test and `original_protean` test means/stds exact.
|
||||
- Leakage audit PASS; `test_used_for_selection=false` upheld.
|
||||
- `beta=0` equivalence: class **B** for all three seeds (no semantic discrepancy).
|
||||
- Direct-effect, client-level, rare/unseen, cross-alpha labels: PASS.
|
||||
- Rare/unseen integrity PASS (0 mismatches / 270 cells). `beta=0` equivalence class **B**.
|
||||
- Reuse decision PASS (not reused; rerun reproduces old baseline exactly).
|
||||
- Direct-effect, client-level, cross-dataset labels: PASS.
|
||||
- Source/data integrity PASS. pytest: **75 passed, 0 failed, 0 skipped, 0 warnings**.
|
||||
|
||||
## 17. Files created / modified this session
|
||||
## 16. Files created / modified this session
|
||||
|
||||
Created:
|
||||
|
||||
- `docs/research/ALPHA075_BETA_VALIDATION.md`
|
||||
- `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md`
|
||||
- `docs/research/SESSION8_AUDIT.md`
|
||||
- `scripts/analyze_partition_heterogeneity.py`
|
||||
- `scripts/verify_beta0_equivalence.py`
|
||||
- `scripts/analyze_cross_alpha.py`
|
||||
- `artifacts/analysis/xiiotid__alpha0.75__*` (run snapshots, effect/weights CSVs, multiseed
|
||||
tables, `beta0_equivalence.csv`, `runtime.csv`)
|
||||
- `artifacts/analysis/xiiotid__cross_alpha__*` (partition heterogeneity per-seed/per-class/summary,
|
||||
cross-alpha beta table/displacement/client summary/summary)
|
||||
- `results/xiiotid__alpha0.75__seed{0,1,2}__*` (18 run dirs, gitignored)
|
||||
- `docs/research/5GNIDD_ALPHA050_BETA_VALIDATION.md`
|
||||
- `docs/research/SESSION9_AUDIT.md`
|
||||
- `artifacts/analysis/5gnidd__alpha0.5__*` (46 files: 18 prototype snapshots, 6 prototype
|
||||
effect/weights CSVs, multiseed validation/client/test/effect/beta0/runtime tables,
|
||||
partition heterogeneity + characterization, audit recomputation)
|
||||
- `results/5gnidd__alpha0.5__seed{0,1,2}__*` (18 run dirs, gitignored)
|
||||
|
||||
Modified:
|
||||
|
||||
@@ -337,86 +317,81 @@ Modified:
|
||||
|
||||
No source or test code was changed.
|
||||
|
||||
## 18. Exact recommended next phase
|
||||
## 17. Exact recommended next phase
|
||||
|
||||
1. **Session 8 is audited and safe to checkpoint** (verdict PASS; see §16 and
|
||||
`docs/research/SESSION8_AUDIT.md`). X-IIoTID alpha=0.75 behaved cleanly. Before
|
||||
checkpointing, stage only the intended docs/scripts/analysis artifacts; do not commit
|
||||
`results/` or the raw datasets.
|
||||
2. **X-IIoTID three-alpha study is complete** (0.25, 0.50, 0.75 × seeds `{0,1,2}`), with
|
||||
audited condition-level pooled selections 0.00 / 0.75 / 0.50 and a cross-alpha report.
|
||||
3. **Next major phase: 5G-NIDD.** Follow the same protocol (the planned alphas `{0.75, 0.50,
|
||||
0.25}`, seeds `{0,1,2}`, five betas + `original_protean`, frozen manifests, pooled
|
||||
validation-only selection). Do **not** launch the full grid blindly: verify 5G-NIDD manifests
|
||||
exist and reproduce, run the same one-condition-at-a-time pacing, and measure runtime first.
|
||||
A single smoke/one-condition run before scaling is recommended.
|
||||
4. Do **not** treat any selected beta as an optimum. Report each as "best among tested
|
||||
candidates under pooled validation for that condition".
|
||||
5. Do **not** infer a monotone alpha→beta relation from three X-IIoTID conditions.
|
||||
6. Update `EXPERIMENT_PROTOCOL.md` only if a real ambiguity appears; update this handoff.
|
||||
7. Do not commit or push unless asked.
|
||||
1. **Session 9 is audited and safe to checkpoint** (verdict PASS; see §15 and
|
||||
`docs/research/SESSION9_AUDIT.md`). Before checkpointing, stage only the intended
|
||||
docs/analysis artifacts; do not commit `results/` or the raw datasets.
|
||||
2. **5G-NIDD alpha=0.50 is complete.** Next 5G-NIDD condition (alpha=0.75 or alpha=0.25) is
|
||||
**safe to run** — its manifest already exists and reproduces from raw data, the pipeline is
|
||||
stable, and the same frozen protocol applies. Budget **≈19 h per condition** (18 runs × ~64 min)
|
||||
and run **one condition at a time**; do not launch the full grid blindly. A single condition,
|
||||
then inspect, then proceed is recommended.
|
||||
3. Follow the same pooled validation-only selection rule; report each selected beta as "best among
|
||||
the tested candidates under pooled validation for that condition" — never a global optimum.
|
||||
4. Do not infer a monotone alpha→beta relation from three X-IIoTID conditions or from a single
|
||||
5G-NIDD condition; do not claim dataset causality from two datasets.
|
||||
5. Update `EXPERIMENT_PROTOCOL.md` only if a real ambiguity appears; update this handoff.
|
||||
6. Do not commit or push unless asked.
|
||||
|
||||
## 19. Important commands
|
||||
## 18. Important commands
|
||||
|
||||
```bash
|
||||
# Tests (75 expected)
|
||||
.venv/bin/python -m pytest
|
||||
|
||||
# One full paired beta run (CPU, ~40 min, ~6.5-7.2 GiB)
|
||||
.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.75 --seed 1 \
|
||||
# One full paired beta run (CPU, ~63-68 min, ~8.7-14.1 GiB for 5G-NIDD)
|
||||
.venv/bin/python scripts/run_baseline.py --dataset 5gnidd --alpha 0.5 --seed 1 \
|
||||
--clients 10 --rounds 10 --local-epochs 3 \
|
||||
--aggregation-mode support_weighted --beta 0.5
|
||||
--aggregation-mode support_weighted --beta 1.0
|
||||
|
||||
# Pooled multi-seed validation analysis (read-only, no retraining)
|
||||
.venv/bin/python scripts/analyze_multiseed_beta.py --dataset xiiotid --alpha 0.75 --seeds 0 1 2
|
||||
.venv/bin/python scripts/analyze_multiseed_beta.py --dataset 5gnidd --alpha 0.5 --seeds 0 1 2
|
||||
|
||||
# Direct prototype-effect analysis (no retraining)
|
||||
.venv/bin/python scripts/analyze_prototype_effect.py \
|
||||
--snapshot artifacts/analysis/xiiotid__alpha0.75__seed0__support_weighted__beta0__prototype_snapshot.json \
|
||||
--class-names "C&C,Exfiltration,Exploitation,Lateral_movement,Normal,RDOS,Reconnaissance,Tampering,Weaponization" \
|
||||
--snapshot artifacts/analysis/5gnidd__alpha0.5__seed0__support_weighted__beta0__prototype_snapshot.json \
|
||||
--class-names "Benign,HTTPFlood,ICMPFlood,SYNFlood,SYNScan,SlowrateDoS,TCPConnectScan,UDPFlood,UDPScan" \
|
||||
--output-dir artifacts/analysis
|
||||
|
||||
# Realized partition characterization (all alphas)
|
||||
.venv/bin/python scripts/analyze_partition_heterogeneity.py --dataset xiiotid --alphas 0.75 0.5 0.25
|
||||
# Realized partition characterization
|
||||
.venv/bin/python scripts/analyze_partition_heterogeneity.py --dataset 5gnidd --alphas 0.5
|
||||
|
||||
# beta=0 equivalence audit
|
||||
.venv/bin/python scripts/verify_beta0_equivalence.py --dataset xiiotid --alpha 0.75 --seeds 0 1 2
|
||||
|
||||
# Cross-alpha comparison
|
||||
.venv/bin/python scripts/analyze_cross_alpha.py --dataset xiiotid --alphas 0.25 0.5 0.75
|
||||
.venv/bin/python scripts/verify_beta0_equivalence.py --dataset 5gnidd --alpha 0.5 --seeds 0 1 2
|
||||
```
|
||||
|
||||
## 20. Recommended next-session prompt
|
||||
## 19. Recommended next-session prompt
|
||||
|
||||
```
|
||||
You are continuing the perdalam-protean research repository (Session 9).
|
||||
You are continuing the perdalam-protean research repository (Session 10).
|
||||
|
||||
FIRST, read completely:
|
||||
1. docs/handoff/SESSION_HANDOFF.md
|
||||
2. docs/research/ALPHA075_BETA_VALIDATION.md
|
||||
3. docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md
|
||||
4. docs/research/SESSION8_AUDIT.md
|
||||
5. docs/research/EXPERIMENT_PROTOCOL.md
|
||||
6. docs/baseline/BASELINE_VALIDATION.md
|
||||
2. docs/research/5GNIDD_ALPHA050_BETA_VALIDATION.md
|
||||
3. docs/research/SESSION9_AUDIT.md
|
||||
4. docs/research/EXPERIMENT_PROTOCOL.md
|
||||
5. docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md
|
||||
6. docs/research/MULTISEED_BETA_VALIDATION.md
|
||||
|
||||
Critical rules:
|
||||
- planningawal/ is immutable.
|
||||
- ~/datasets/X-IIoTID/X-IIoTID-dataset.csv and ~/datasets/5G-NIDD/5G-NIDD.csv are read-only.
|
||||
- The S_j participant rule is a REPRODUCTION ASSUMPTION (material in 19/27 cells at alpha=0.25).
|
||||
- ~/datasets/5G-NIDD/5G-NIDD.csv and ~/datasets/X-IIoTID/X-IIoTID-dataset.csv are read-only.
|
||||
- The S_j participant rule is a REPRODUCTION ASSUMPTION.
|
||||
- Do NOT add an executable equal_available mode.
|
||||
- Beta selection uses validation only; PRIMARY is pooled across clients and seeds.
|
||||
- Do NOT launch the full grid blindly; verify manifests and pace one condition at a time.
|
||||
- Do NOT launch the full grid; run one condition at a time (~19 h per 5G-NIDD condition).
|
||||
|
||||
Context: Session 8 completed X-IIoTID alpha=0.75 for seeds {0,1,2} (18 runs, ~12.33 h) and was
|
||||
independently audited (PASS; docs/research/SESSION8_AUDIT.md). Pooled validation selected
|
||||
beta=0.50. Three-alpha X-IIoTID pooled selections: alpha=0.25 -> 0.00, alpha=0.50 -> 0.75,
|
||||
alpha=0.75 -> 0.50 (NOT monotone). beta=0 equivalence class B. Realized heterogeneity: 19/27,
|
||||
7/27, 3/27 cells |S_j|<M; mean displacement 3.42 / 2.10 / 1.31.
|
||||
Context: Session 9 completed 5G-NIDD alpha=0.50 for seeds {0,1,2} (18 runs, ~19.26 h) and was
|
||||
independently audited (PASS; docs/research/SESSION9_AUDIT.md). Pooled validation selected
|
||||
beta=1.00 (per-seed diagnostics 1.00 / 0.00 / 1.00). beta=0 equivalence class B. Realized
|
||||
heterogeneity: 4/27 cells |S_j|<M, 4 missing cells, client ratio mean 9.59, mean displacement
|
||||
2.340. X-IIoTID alpha=0.50 selected 0.75. The pre-existing seed-0 baseline was not reusable and
|
||||
was re-run; it reproduces exactly.
|
||||
|
||||
Task: X-IIoTID three-alpha study is complete. Recommended next phase is 5G-NIDD under the same
|
||||
frozen protocol (alphas {0.75, 0.50, 0.25}, seeds {0,1,2}, five betas + original_protean,
|
||||
frozen manifests, pooled validation-only selection). Before scaling, verify 5G-NIDD manifests
|
||||
exist and reproduce, run one paired condition, inspect it, then proceed. Produce a 5G-NIDD beta
|
||||
validation report and a cross-dataset comparison, update the handoff. Use .venv/bin/python. Do
|
||||
not commit or push unless asked.
|
||||
Task: run the next single 5G-NIDD condition (recommended alpha=0.75, then alpha=0.25) under the
|
||||
same frozen protocol (seeds {0,1,2}, five betas + original_protean, frozen manifests, pooled
|
||||
validation-only selection). Verify the manifest reproduces, run the 18 runs sequentially with
|
||||
runtime/status files, produce a condition report + audit, and extend the cross-dataset/cross-alpha
|
||||
comparison. Use .venv/bin/python. Do not commit or push unless asked.
|
||||
```
|
||||
@@ -0,0 +1,495 @@
|
||||
# 5G-NIDD Alpha-0.50 Beta Validation (Session 9, Stage B)
|
||||
|
||||
Status: **first cross-dataset beta-validation condition**. This report covers a single 5G-NIDD
|
||||
condition — `alpha = 0.50`, seeds `{0,1,2}`, the frozen `original_protean` baseline plus the five
|
||||
`support_weighted` beta candidates — using the same pooled-across-seed validation-only selection
|
||||
rule already applied to X-IIoTID. It is **not** a sweep, not a hyper-parameter search, and not
|
||||
evidence of a global optimum. No other 5G-NIDD alpha (0.25 / 0.75) was run.
|
||||
|
||||
Companion documents:
|
||||
|
||||
- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen protocol and selection hierarchy.
|
||||
- `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md` — X-IIoTID three-condition reference.
|
||||
- `docs/research/MULTISEED_BETA_VALIDATION.md` — X-IIoTID alpha=0.50 (the matched condition).
|
||||
- `docs/research/SESSION9_AUDIT.md` — independent audit of this condition.
|
||||
- `docs/handoff/SESSION_HANDOFF.md` — continuation point.
|
||||
|
||||
---
|
||||
|
||||
## A. Scope
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| dataset | `5gnidd` (5G-NIDD) |
|
||||
| alpha | 0.50 |
|
||||
| seeds | 0, 1, 2 |
|
||||
| aggregation | `original_protean` (per seed) + `support_weighted` for `beta ∈ {0, 0.25, 0.50, 0.75, 1.00}` |
|
||||
| runs | 18: 15 `support_weighted` + 3 `original_protean`, **all run new this session** |
|
||||
| compute | sequential CPU, one run at a time, 69,336 s ≈ **19.26 h** wall-clock |
|
||||
| selection | pooled validation-only across seeds `{0,1,2}` |
|
||||
| test | read only after the pooled beta was fixed |
|
||||
|
||||
No 5G-NIDD `alpha=0.25` or `alpha=0.75` run was performed; the full grid was deliberately not
|
||||
launched. `equal_available` was not added. No training/aggregation/preprocessing source code was
|
||||
changed.
|
||||
|
||||
---
|
||||
|
||||
## B. Dataset and manifests
|
||||
|
||||
### B.1 Raw-data provenance (Stage A)
|
||||
|
||||
- Path `~/datasets/5G-NIDD/5G-NIDD.csv`; SHA-256
|
||||
`fa36f80859585f474504ca69eb951a079b35145537976c86b00aae3aab46ee59` — matches the audited value.
|
||||
- Raw shape 1,215,890 rows × 52 columns; **1** exact duplicate row dropped → 1,215,889; **0**
|
||||
label rows dropped (`drop_labels` empty). Missing tokens: empty string only.
|
||||
- **Label semantics (canonicalised):** label column `Attack Type`; **9 classes** — benign
|
||||
`Benign` (index 0) plus **8 attack classes** `HTTPFlood, ICMPFlood, SYNFlood, SYNScan,
|
||||
SlowrateDoS, TCPConnectScan, UDPFlood, UDPScan`.
|
||||
|
||||
| class | full-dataset count |
|
||||
|---|---|
|
||||
| Benign | 477,736 |
|
||||
| UDPFlood | 457,340 |
|
||||
| HTTPFlood | 140,812 |
|
||||
| SlowrateDoS | 73,124 |
|
||||
| TCPConnectScan | 20,052 |
|
||||
| SYNScan | 20,043 |
|
||||
| UDPScan | 15,906 |
|
||||
| SYNFlood | 9,721 |
|
||||
| ICMPFlood | 1,155 |
|
||||
|
||||
- **46 feature columns** after excluding identifiers/targets: 41 numeric (median-imputed and
|
||||
train-standardized) and 5 categorical one-hot (`Proto, sDSb, dDSb, Cause, State`), with an
|
||||
explicit `__MISSING__` category. Preprocessing is spec-driven, fits on the train split only,
|
||||
and contains **no X-IIoTID-specific assumptions**.
|
||||
- Rare/unseen logic verified against this class list: attack classes only, `N_ij > 0`, two
|
||||
smallest positive training supports per client, `N_ij = 0` is unseen (never rare), benign never
|
||||
rare/unseen. All 10 clients are eligible in all three seeds (each has exactly two rare classes).
|
||||
|
||||
### B.2 Manifest identity (reused, never regenerated)
|
||||
|
||||
Train/validation/test sizes are 851,123 / 121,589 / 243,177 for every seed (10 clients, 9
|
||||
classes, `Benign`). Each manifest was independently reproduced from the raw dataset during Stage A
|
||||
(`prepare_dataset` → `reproduce_train_assignments`): exact `N_ij` match and partition-hash match
|
||||
for all three seeds (and, as a side check, for alpha=0.25/0.75).
|
||||
|
||||
| seed | manifest | content hash | partition hash | split hash | rng seed |
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | `artifacts/partitions/5gnidd__alpha0.5__seed0.json` | `0732582c…` | `8352d4ff…` | `18f89f48…` | 673621352 |
|
||||
| 1 | `artifacts/partitions/5gnidd__alpha0.5__seed1.json` | `f43906e1…` | `0815c91e…` | `3d05b7fe…` | 1403944623 |
|
||||
| 2 | `artifacts/partitions/5gnidd__alpha0.5__seed2.json` | `fe8b4fc6…` | `fdcf388e…` | `fc758d06…` | 633461056 |
|
||||
|
||||
`preprocessing_hash` / `config_hash` remain `null` in these Session-2 manifests (known limitation,
|
||||
carry-over from X-IIoTID).
|
||||
|
||||
### B.3 Pairing (from content hashes, not filenames)
|
||||
|
||||
Within each seed all six runs share one manifest content hash and a **byte-identical round-0
|
||||
snapshot**; `train_support_Nij.csv`, `characterization.csv` and `rare_unseen.csv` are identical
|
||||
across the six runs; `manifest_reload_ok = true` and `smoke = false` everywhere.
|
||||
|
||||
| seed | manifest content hash | round-0 snapshot SHA-256 (all 6 runs) |
|
||||
|---|---|---|
|
||||
| 0 | `0732582cf4510089…` | `eb0839f25957274a6bc42532c07b89826cf66c581ac05d197a10333a21bc467f` |
|
||||
| 1 | `f43906e111a8342c…` | `6cbd04e0dd97bbe0dad733f654962a115cef54e126ab2c4e58d853ac0b249afb` |
|
||||
| 2 | `fe8b4fc6f0486cf4…` | `f012a02b543b926a9df31134e06bf442ed832b35cdeb9aba1f20d404e8b93687` |
|
||||
|
||||
---
|
||||
|
||||
## C. Structural characterization (realized, before interpretation)
|
||||
|
||||
All statistics are from the training-split `N_ij`; `M = 10` clients; `|S_j| < M` means fewer than
|
||||
10 clients contribute a prototype for a class.
|
||||
|
||||
### C.1 Per-seed summary
|
||||
|
||||
| seed | `|S_j| < M` cells | missing client-class cells | client-size min/max (ratio) | union rare | union unseen | max unseen on a client |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0 | 2 / 9 | 2 | 15,388 / 178,993 (11.63) | 6 | 2 | 1 |
|
||||
| 1 | 0 / 9 | 0 | 31,613 / 140,144 (4.43) | 5 | 0 | 0 |
|
||||
| 2 | 2 / 9 | 2 | 20,087 / 255,463 (12.72) | 6 | 2 | 1 |
|
||||
| **total / mean** | **4 / 27** | **4** | ratio mean **9.59** | 6/5/6 | 2/0/2 | 1 |
|
||||
|
||||
All 10 clients have exactly two rare classes in every seed, so all 10 are eligible for rare-class
|
||||
evaluation. Missing cells are only ever single-client gaps in the two smallest attack classes
|
||||
(e.g. seed 0: `ICMPFlood` absent on client 3, `SYNFlood` absent on client 4). Rare classes are
|
||||
dominated by `ICMPFlood` (always rare) and one of `SYNFlood`/`TCPConnectScan`/`UDPScan`/
|
||||
`HTTPFlood`/`SYNScan`.
|
||||
|
||||
### C.2 Per-class `N_ij` inequality (training split)
|
||||
|
||||
`CV` is the population coefficient of variation of the positive supports within a class;
|
||||
`max/min` is over contributing clients only. Machine-readable:
|
||||
`artifacts/analysis/5gnidd__alpha0.5__partition_heterogeneity_per_class.csv`.
|
||||
|
||||
**Seed 0**
|
||||
|
||||
| class | total | #clients | min+ | max | CV | max/min |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Benign | 334,415 | 10 | 226 | 108,257 | 1.069 | 479 |
|
||||
| HTTPFlood | 98,569 | 10 | 185 | 61,393 | 1.779 | 332 |
|
||||
| ICMPFlood | 808 | 9 | 8 | 243 | 0.809 | 30.4 |
|
||||
| SYNFlood | 6,805 | 9 | 2 | 3,250 | 1.318 | 1,625 |
|
||||
| SYNScan | 14,030 | 10 | 11 | 6,713 | 1.414 | 610 |
|
||||
| SlowrateDoS | 51,187 | 10 | 80 | 28,363 | 1.783 | 355 |
|
||||
| TCPConnectScan | 14,037 | 10 | 1 | 4,175 | 0.982 | 4,175 |
|
||||
| UDPFlood | 320,138 | 10 | 1 | 88,943 | 0.896 | 88,943 |
|
||||
| UDPScan | 11,134 | 10 | 14 | 3,420 | 0.966 | 244 |
|
||||
|
||||
Seeds 1 and 2 are in the machine-readable file; the notable pattern is that even though the
|
||||
number of *missing* cells is tiny, the per-class support is strongly imbalanced for the rare
|
||||
classes (`TCPConnectScan`, `SYNFlood`, `UDPFlood` max/min in the hundreds–thousands). The mean
|
||||
class-support CV across seeds is **1.225** (1.224 / 1.051 / 1.401).
|
||||
|
||||
### C.3 Rare/unseen identity
|
||||
|
||||
Per-seed union rare classes (ids in brackets): seed 0 = `{ICMPFlood, SYNFlood, SYNScan,
|
||||
TCPConnectScan, UDPFlood, UDPScan}`; seed 1 = `{ICMPFlood, SYNFlood, SYNScan, TCPConnectScan,
|
||||
UDPScan}`; seed 2 = `{ICMPFlood, HTTPFlood, SYNFlood, SYNScan, UDPFlood, UDPScan}`. Unseen unions:
|
||||
seed 0 = `{ICMPFlood, SYNFlood}` (one client each), seed 1 = `{}`, seed 2 =
|
||||
`{TCPConnectScan, UDPScan}`. Full per-client detail:
|
||||
`artifacts/analysis/5gnidd__alpha0.5__partition_characterization.csv`.
|
||||
|
||||
---
|
||||
|
||||
## D. Existing seed-0 baseline reuse decision
|
||||
|
||||
A pre-existing artifact `results/5gnidd__alpha0.5__seed0__original_protean` (Session-5 era) was
|
||||
inspected before training and **was not reused**. It contains the frozen training outputs
|
||||
(`history.json`, `test_*`, `per_client.csv`, `characterization.csv`, `rare_unseen.csv`,
|
||||
`train_support_Nij.csv`, `metadata.json`) and a round-0 snapshot, but **no validation artifacts**
|
||||
(`val_selection.json`, `val_per_client.csv`, `val_global_metrics.json`, `val_confusion_matrix.json`,
|
||||
`val_per_class.csv`). Under the frozen protocol the pooled selection metric requires
|
||||
`val_selection.json` / `val_per_client.csv`, and pairing/`beta=0` equivalence require re-runnable
|
||||
validation artifacts; the old directory therefore fails the "complete artifacts" reuse criterion.
|
||||
|
||||
The directory was backed up to `/tmp/opencode/s9/old_baseline_backup/` and **all 18 runs were
|
||||
re-executed**. As a determinism check, the fresh seed-0 `original_protean` run reproduces the old
|
||||
artifact **exactly**: `history.json` is byte-identical and every test metric is identical
|
||||
(e.g. rare-class macro-F1 `0.8719637151922778`, accuracy `0.7344156725348203`). The old baseline
|
||||
was therefore methodologically sound but structurally incomplete for the validation protocol.
|
||||
|
||||
---
|
||||
|
||||
## E. Frozen protocol
|
||||
|
||||
Identical to `EXPERIMENT_PROTOCOL.md` §14: 10 clients, 10 rounds, 3 local epochs, batch 256, Adam
|
||||
`lr=1e-3`, `mu=0.1`, `lambda=1.0`; unchanged architecture, preprocessing, split recipe, Dirichlet
|
||||
partition recipe (`per_class_dirichlet_multinomial`), loss and nearest-prototype inference. All 18
|
||||
runs used `scripts/run_baseline.py` at HEAD `2d27abc35026456de765c929dbcf0503a72ac572`.
|
||||
|
||||
Selection rule (unchanged):
|
||||
|
||||
- **PRIMARY:** pooled-across-seed validation per-client-mean rare-class macro-F1, validation only.
|
||||
- **SECONDARY (diagnostic):** per-seed validation selection.
|
||||
- **TEST:** read only after the pooled beta was fixed; `test_used_for_selection = false` for every
|
||||
`support_weighted` run.
|
||||
|
||||
---
|
||||
|
||||
## F. Per-seed validation results
|
||||
|
||||
`val rareMacroF1 (pc mean)` is the primary metric; all 10 clients eligible in every seed.
|
||||
Machine-readable: `artifacts/analysis/5gnidd__alpha0.5__multiseed__validation_per_seed.csv`.
|
||||
|
||||
### Seed 0
|
||||
|
||||
| beta | eligible | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | 10 | 0.738597 | 0.869471 | 0.971734 | 0.835222 | 0.735708 |
|
||||
| 0.25 | 10 | 0.661461 | 0.826591 | 0.971734 | 0.804446 | 0.733586 |
|
||||
| 0.50 | 10 | 0.764487 | 0.877274 | 0.971734 | 0.844008 | 0.738932 |
|
||||
| 0.75 | 10 | 0.750661 | 0.866155 | 0.971734 | 0.846984 | 0.747313 |
|
||||
| **1.00** | 10 | **0.768877** | 0.878253 | 0.971734 | 0.847921 | 0.743390 |
|
||||
|
||||
### Seed 1
|
||||
|
||||
| beta | eligible | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **0.00** | 10 | **0.633243** | 0.923637 | 0.970682 | 0.866358 | 0.746638 |
|
||||
| 0.25 | 10 | 0.573717 | 0.903669 | 0.970682 | 0.854134 | 0.745594 |
|
||||
| 0.50 | 10 | 0.563388 | 0.897248 | 0.970682 | 0.842769 | 0.739434 |
|
||||
| 0.75 | 10 | 0.543319 | 0.875983 | 0.970682 | 0.830737 | 0.739360 |
|
||||
| 1.00 | 10 | 0.531764 | 0.849605 | 0.970682 | 0.820473 | 0.743052 |
|
||||
|
||||
### Seed 2
|
||||
|
||||
| beta | eligible | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | 10 | 0.689849 | 0.837430 | 0.971518 | 0.817547 | 0.750010 |
|
||||
| 0.25 | 10 | 0.700055 | 0.841150 | 0.968867 | 0.817724 | 0.749459 |
|
||||
| 0.50 | 10 | 0.696131 | 0.838533 | 0.958948 | 0.813338 | 0.745158 |
|
||||
| 0.75 | 10 | 0.771124 | 0.884167 | 0.953065 | 0.841394 | 0.742304 |
|
||||
| **1.00** | 10 | **0.776468** | 0.887075 | 0.954616 | 0.843904 | 0.743431 |
|
||||
|
||||
The per-seed curves are **non-monotone and seed-dependent**: seed 0 rises to a peak at 1.00 with a
|
||||
sharp dip at 0.25; seed 1 is monotone decreasing (best at 0.00); seed 2 rises then is nearly flat
|
||||
at 0.75–1.00.
|
||||
|
||||
---
|
||||
|
||||
## G. Per-seed diagnostic beta choices (SECONDARY, stability only)
|
||||
|
||||
| seed | selected beta | val rareMacroF1 (pc mean) | tie-break used | test_used_for_selection |
|
||||
|---|---|---|---|---|
|
||||
| 0 | 1.00 | 0.768877 | no | false |
|
||||
| 1 | 0.00 | 0.633243 | no | false |
|
||||
| 2 | 1.00 | 0.776468 | no | false |
|
||||
|
||||
These per-seed choices are **diagnostic only**; they are not the condition-level selection rule.
|
||||
`val_selection.json` records `"selection_split": "validation"` and
|
||||
`"test_used_for_selection": false` for all 15 `support_weighted` runs.
|
||||
|
||||
---
|
||||
|
||||
## H. PRIMARY pooled-across-seed validation selection
|
||||
|
||||
| beta | pooled mean | std (sample, ddof=1) | min | max | pooled val rareRecall (pc mean) | selected |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | 0.687230 | 0.052726 | 0.633243 | 0.738597 | 0.947737 | |
|
||||
| 0.25 | 0.645078 | 0.064743 | 0.573717 | 0.700055 | 0.946862 | |
|
||||
| 0.50 | 0.674669 | 0.102253 | 0.563388 | 0.764487 | 0.943659 | |
|
||||
| 0.75 | 0.688368 | 0.126032 | 0.543319 | 0.771124 | 0.941915 | |
|
||||
| **1.00** | **0.692370** | 0.139140 | 0.531764 | 0.776468 | 0.942301 | **yes** |
|
||||
|
||||
- **PRIMARY selected beta (`beta_star_5gnidd_alpha050`) = 1.00.**
|
||||
- **Criterion:** maximum pooled validation per-client-mean rare-class macro-F1 (validation only).
|
||||
- **Tie-break used:** no. Top-two beta = 1.00 (0.692370) vs 0.75 (0.688368); gap ≈ **4.0e-3**.
|
||||
- **`test_used_for_selection = false`.** Test metrics were not inspected until the pooled beta was
|
||||
fixed.
|
||||
- **Interpretation:** `beta = 1.00` is the *best among the tested candidates under pooled
|
||||
validation across seeds `{0,1,2}` for 5G-NIDD alpha=0.50*. It is **not** a global optimum. The
|
||||
pooled validation curve is non-monotone (0.687 → 0.645 → 0.675 → 0.688 → 0.692), the total
|
||||
between-beta spread is ≈ **0.047**, and the across-seed std is **0.053–0.139** — comparable to or
|
||||
larger than the between-beta spread, so the selection is statistically weak. The pooled win is
|
||||
driven by seeds 0 and 2 (+0.030 and +0.087 at beta=1 vs beta=0) outweighing seed 1's large loss
|
||||
(−0.101), for a net +0.005 over beta=0.
|
||||
|
||||
Machine-readable: `artifacts/analysis/5gnidd__alpha0.5__multiseed__validation_pooled.csv`,
|
||||
`...__selection_summary.json`.
|
||||
|
||||
---
|
||||
|
||||
## I. Final selected-beta test metrics (descriptive only)
|
||||
|
||||
`beta_star = 1.00` was fixed from pooled validation; only then were test metrics read.
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.880962 | 0.840075 | 0.894914 | 0.871984 ± 0.028501 |
|
||||
| rare-class recall | 0.974355 | 0.968341 | 0.958616 | 0.967104 ± 0.007942 |
|
||||
| macro-F1 | 0.850470 | 0.815941 | 0.849378 | 0.838596 ± 0.019627 |
|
||||
| accuracy | 0.742838 | 0.744026 | 0.742907 | 0.743257 ± 0.000667 |
|
||||
| weighted-F1 | 0.716950 | 0.720106 | 0.717530 | 0.718195 ± 0.001680 |
|
||||
|
||||
Note the global rare support is very different per seed (104,843 / 13,375 / 128,995), because the
|
||||
training-derived union of rare classes differs by seed; cross-seed absolute levels are therefore
|
||||
not directly comparable.
|
||||
|
||||
---
|
||||
|
||||
## J. `original_protean` comparison
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.871964 | 0.921136 | 0.848525 | 0.880542 ± 0.037058 |
|
||||
| rare-class recall | 0.974314 | 0.968566 | 0.975859 | 0.972913 ± 0.003843 |
|
||||
| macro-F1 | 0.836881 | 0.865437 | 0.824867 | 0.842395 ± 0.020840 |
|
||||
| accuracy | 0.734416 | 0.747377 | 0.750133 | 0.743975 ± 0.008393 |
|
||||
| weighted-F1 | 0.708765 | 0.722802 | 0.724622 | 0.718730 ± 0.008678 |
|
||||
|
||||
Descriptive difference (`beta_star` − `original_protean`) for test rare-class macro-F1 is
|
||||
**+0.008998** (seed 0), **−0.081061** (seed 1), **+0.046389** (seed 2); mean **−0.008558**. Seed 1
|
||||
moves strongly the opposite way to seeds 0 and 2, and the magnitude is far smaller than the
|
||||
between-seed std (≈ 0.03–0.04). No superiority or generalisation is claimed; this is one dataset,
|
||||
one alpha, three project-defined seeds. Test columns:
|
||||
`artifacts/analysis/5gnidd__alpha0.5__multiseed__test_descriptive.csv`.
|
||||
|
||||
A descriptive dataset-level observation: on 5G-NIDD, test weighted-F1 (`≈0.718`) is **below**
|
||||
accuracy (`≈0.744`), i.e. the high-support classes (`Benign`, `UDPFlood`) are the harder ones for
|
||||
nearest-prototype inference. This is the opposite of X-IIoTID (weighted-F1 > accuracy) and is a
|
||||
dataset characteristic, not a beta effect.
|
||||
|
||||
---
|
||||
|
||||
## K. `beta = 0` equivalence vs `original_protean`
|
||||
|
||||
| property | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| same manifest / split / hashes | yes | yes | yes |
|
||||
| round-0 snapshot byte-identical | yes (`eb0839f2…`) | yes (`6cbd04e0…`) | yes (`f012a02b…`) |
|
||||
| support / characterization / rare_unseen identical | yes | yes | yes |
|
||||
| independent aggregation diff (`ΣC/n` vs `(1/n)ΣC`) | 4.4e-16 | 4.4e-16 | 4.4e-16 |
|
||||
| history rare-macro-F1 round-0 diff | 0.0 | 0.0 | 0.0 |
|
||||
| history rare-macro-F1 max diff (rounds ≥ 1) | 1.93e-2 | 6.01e-2 | 2.60e-2 |
|
||||
| test metric max abs diff | 2.33e-3 | 1.86e-4 | 8.70e-3 |
|
||||
| **classification** | **B** | **B** | **B** |
|
||||
|
||||
Class **B**: semantic equivalence + floating-point numerical divergence; **no semantic
|
||||
discrepancy**. Round 0 is byte-identical; a ~1 ULP ordering difference enters round-1
|
||||
prototype-alignment training and the trajectories then diverge. The downstream divergence is
|
||||
somewhat larger than at X-IIoTID (5G-NIDD training is more sensitive to the tiny round-1
|
||||
difference), but it remains pure floating-point divergence, not a semantic difference. Equivalence
|
||||
remains conditional on the shared `S_j` reproduction assumption and is **not** proof that
|
||||
`beta = 0` reproduces original PROTEAN. Machine-readable:
|
||||
`artifacts/analysis/5gnidd__alpha0.5__multiseed__beta0_equivalence.csv`.
|
||||
|
||||
---
|
||||
|
||||
## L. Direct prototype-effect findings (fixed round-0 snapshots, no retraining)
|
||||
|
||||
Computed from each seed's fixed `{C_ij, N_ij}` round-0 snapshot via the production aggregation.
|
||||
Cross-seed mean `||Cbar_j(beta) − Cbar_j(0)||` at `beta = 1.0` (sample std across seeds):
|
||||
|
||||
| class | mean disp | std | mean rel disp | #contributors (seed 0) |
|
||||
|---|---|---|---|---|
|
||||
| UDPScan | 3.461 | 2.011 | 0.424 | 10 |
|
||||
| HTTPFlood | 3.120 | 1.289 | 0.433 | 10 |
|
||||
| SYNScan | 2.875 | 0.959 | 0.396 | 10 |
|
||||
| SlowrateDoS | 2.763 | 1.454 | 0.454 | 10 |
|
||||
| SYNFlood | 2.262 | 0.748 | 0.658 | 9 |
|
||||
| TCPConnectScan | 2.131 | 0.213 | 0.575 | 10 |
|
||||
| Benign | 1.690 | 0.699 | 0.296 | 10 |
|
||||
| UDPFlood | 1.598 | 0.428 | 0.300 | 10 |
|
||||
| ICMPFlood | 1.159 | 0.093 | 0.321 | 9 |
|
||||
|
||||
- Mean displacement across the nine classes is **2.340** at beta=1.0, monotone in beta
|
||||
(0.25 → 0.953, 0.50 → 1.567, 0.75 → 2.003, 1.00 → 2.340).
|
||||
- Displacement is **seed-sensitive**: e.g. `UDPScan` 1.42 (seed 0) / 3.52 (seed 1) / 5.44
|
||||
(seed 2); `HTTPFlood` 2.40 / 2.36 / 4.61. The largest displacements are the rare, support-
|
||||
imbalanced classes; `Benign`/`UDPFlood` move least.
|
||||
- Even classes with a single missing contributor (`ICMPFlood`, `SYNFlood`) move, because the
|
||||
weighting changes among the 9–10 contributors, not because of the missing one.
|
||||
- Displacement is a **direct-aggregation measure only** and is **not** interpreted as improved or
|
||||
degraded detection. The end-to-end effect is reported separately in §H/§I.
|
||||
|
||||
Outputs: `artifacts/analysis/5gnidd__alpha0.5__seed{0,1,2}__support_weighted__beta0__prototype_snapshot__prototype_effect.csv`
|
||||
and `...__prototype_weights.csv`; cross-seed aggregate
|
||||
`artifacts/analysis/5gnidd__alpha0.5__multiseed__prototype_effect_pooled.csv`.
|
||||
|
||||
---
|
||||
|
||||
## M. Client-level effect (validation, relative to beta = 0)
|
||||
|
||||
Detail: `...__client_validation_detail.csv`; summary: `...__client_validation_summary.csv`. Client
|
||||
ids are per-seed realisations and are **not** pooled across seeds.
|
||||
|
||||
Improve / degrade / unchanged counts (tolerance 1e-9):
|
||||
|
||||
| seed | beta | improve | degrade | unchanged | mean Δ | min Δ | max Δ |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 0 | 0.25 | 2 | 7 | 1 | −0.0771 | −0.2036 | +0.0304 |
|
||||
| 0 | 0.50 | 7 | 2 | 1 | +0.0259 | −0.0343 | +0.0699 |
|
||||
| 0 | 0.75 | 5 | 4 | 1 | +0.0121 | −0.0679 | +0.0713 |
|
||||
| 0 | **1.00** | **6** | **3** | **1** | **+0.0303** | −0.0884 | +0.1034 |
|
||||
| 1 | 0.25 | 3 | 3 | 4 | −0.0595 | −0.3231 | +0.0125 |
|
||||
| 1 | 0.50 | 3 | 5 | 2 | −0.0699 | −0.4231 | +0.0181 |
|
||||
| 1 | 0.75 | 3 | 5 | 2 | −0.0899 | −0.5481 | +0.0240 |
|
||||
| 1 | **1.00** | **3** | **5** | **2** | **−0.1015** | −0.6564 | +0.0631 |
|
||||
| 2 | 0.25 | 5 | 4 | 1 | +0.0102 | −0.0262 | +0.0726 |
|
||||
| 2 | 0.50 | 5 | 4 | 1 | +0.0063 | −0.0442 | +0.0738 |
|
||||
| 2 | 0.75 | 7 | 2 | 1 | +0.0813 | −0.0305 | +0.1974 |
|
||||
| 2 | **1.00** | **7** | **2** | **1** | **+0.0866** | −0.0337 | +0.2016 |
|
||||
|
||||
- The response is strongly seed-dependent. At the selected `beta = 1.00`, seeds 0 and 2 improve
|
||||
on 6/10 and 7/10 clients with positive mean Δ (+0.030, +0.087), while seed 1 degrades 5/10 with
|
||||
mean Δ −0.101, dominated by a single client at **−0.656**.
|
||||
- `N_ij` vs ΔF1 is **non-monotonic**: the worst negative delta in seed 1 (client 2) has
|
||||
`min positive attack N_ij = 45` (not the lowest-support client), while some very low-support
|
||||
clients (`min N_ij = 1`) improve strongly.
|
||||
- This analysis is descriptive and non-causal.
|
||||
|
||||
---
|
||||
|
||||
## N. Runtime and resources
|
||||
|
||||
CPU only, no GPU, no swap, exit status 0 for every run (`/usr/bin/time -v`). Full table:
|
||||
`artifacts/analysis/5gnidd__alpha0.5__multiseed__runtime.csv`.
|
||||
|
||||
| seed | aggregation | wall | CPU % | peak RSS |
|
||||
|---|---|---|---|---|
|
||||
| 0 | `original_protean` | 62:28 | 745 | 10.36 GiB |
|
||||
| 0 | beta 0.00 | 63:37 | 745 | 10.37 GiB |
|
||||
| 0 | beta 0.25 | 62:56 | 744 | 10.36 GiB |
|
||||
| 0 | beta 0.50 | 63:26 | 743 | 10.36 GiB |
|
||||
| 0 | beta 0.75 | 63:35 | 742 | 10.39 GiB |
|
||||
| 0 | beta 1.00 | 64:25 | 741 | 10.36 GiB |
|
||||
| 1 | `original_protean` | 63:57 | 742 | 8.67 GiB |
|
||||
| 1 | beta 0.00 | 66:26 | 738 | 8.67 GiB |
|
||||
| 1 | beta 0.25 | 68:25 | 734 | 8.69 GiB |
|
||||
| 1 | beta 0.50 | 64:57 | 740 | 8.67 GiB |
|
||||
| 1 | beta 0.75 | 65:30 | 740 | 8.67 GiB |
|
||||
| 1 | beta 1.00 | 64:41 | 741 | 8.67 GiB |
|
||||
| 2 | `original_protean` | 63:44 | 740 | 14.04 GiB |
|
||||
| 2 | beta 0.00 | 63:39 | 739 | 14.05 GiB |
|
||||
| 2 | beta 0.25 | 63:29 | 740 | 14.03 GiB |
|
||||
| 2 | beta 0.50 | 63:14 | 739 | 14.06 GiB |
|
||||
| 2 | beta 0.75 | 63:38 | 739 | 14.06 GiB |
|
||||
| 2 | beta 1.00 | 63:29 | 739 | 14.07 GiB |
|
||||
|
||||
- New compute this session: **18 runs, 69,336 s ≈ 19.26 h** wall-clock, sequential on an 8-core
|
||||
host.
|
||||
- Per-run wall time is homogeneous (62–68 min). Peak RSS is partition-dependent, not beta-
|
||||
dependent: ~10.4 GiB (seed 0), ~8.67 GiB (seed 1), ~14.04 GiB (seed 2), all within the 29 GiB
|
||||
host. 5G-NIDD uses more memory than X-IIoTID (≈8.7–14.0 GiB vs ≈6.5–7.2 GiB) because the
|
||||
training split is ~1.5× larger.
|
||||
- No warnings, no client failures, 9/9 global prototypes in every round, no retries.
|
||||
|
||||
---
|
||||
|
||||
## O. Comparison with X-IIoTID alpha = 0.50 (descriptive only)
|
||||
|
||||
X-IIoTID alpha=0.50 reference: pooled-selected `beta = 0.75`
|
||||
(`docs/research/MULTISEED_BETA_VALIDATION.md`, `XIIOTID_CROSS_ALPHA_ANALYSIS.md`).
|
||||
|
||||
| statistic | 5G-NIDD alpha=0.50 | X-IIoTID alpha=0.50 |
|
||||
|---|---|---|
|
||||
| classes (attack) | 9 (8) | 9 (8) |
|
||||
| training rows | 851,123 | 569,909 |
|
||||
| pooled-selected beta | **1.00** | **0.75** |
|
||||
| pooled validation mean range (spread) | 0.645–0.692 (**0.047**) | 0.544–0.579 (0.035) |
|
||||
| pooled validation std range | 0.053–0.139 | 0.075–0.100 |
|
||||
| `|S_j| < M` cells (sum/27) | **4/27** | 7/27 |
|
||||
| missing client-class cells | **4** | 9 |
|
||||
| client-size ratio (seeds / mean) | 11.63 / 4.43 / 12.72 (**9.59**) | 5.11 / 7.22 / 9.89 (7.41) |
|
||||
| mean class-support CV | 1.225 | 1.183 |
|
||||
| union rare per seed | 6 / 5 / 6 | 6 / 7 / 7 |
|
||||
| union unseen per seed | 2 / 0 / 2 | 3 / 3 / 1 |
|
||||
| max unseen on a client | 1 | 2 |
|
||||
| mean displacement `‖Cbar(1)−Cbar(0)‖` | 2.340 | 2.099 |
|
||||
| selected-beta client improve/degrade (per seed) | 6/3, 3/5, 7/2 | 7/3, 8/1, 8/1 |
|
||||
|
||||
Observations (descriptive, two datasets, one condition each, no causal claim):
|
||||
|
||||
- 5G-NIDD alpha=0.50 has **fewer missing class cells** (4 vs 9) but **more imbalanced client
|
||||
sizes** (mean ratio 9.59 vs 7.41); its mean support CV and mean prototype displacement are
|
||||
slightly higher, and it has a single-client unseen-class structure (max 1 vs 2).
|
||||
- The pooled validation selection differs across datasets (5G-NIDD → 1.00, X-IIoTID → 0.75) and
|
||||
the 5G-NIDD curve is non-monotone. At both datasets the between-seed std is comparable to or
|
||||
larger than the between-beta spread, so both selections are statistically weak.
|
||||
- The client-level response is seed-dependent on both datasets; X-IIoTID alpha=0.50 improved on a
|
||||
majority of clients in all three seeds, whereas 5G-NIDD's seed 1 degrades.
|
||||
- **No claim** is made that the dataset explains the selected beta, and no generalisation beyond
|
||||
these two observed conditions is offered.
|
||||
|
||||
---
|
||||
|
||||
## P. Limitations
|
||||
|
||||
- **One new condition** (5G-NIDD alpha=0.50). No 5G-NIDD alpha=0.25/0.75 result exists; no
|
||||
cross-alpha 5G-NIDD claim is possible. No X-IIoTID/5G-NIDD causal or monotone claim.
|
||||
- **Three project-defined seeds** (`{0,1,2}`); the pooled mean's variation across seeds is
|
||||
comparable to or larger than the between-beta spread, so `beta_star = 1.00` is statistically
|
||||
weak and is only "best among the five tested candidates under pooled validation for this
|
||||
condition".
|
||||
- The `S_j` participant/denominator rule (`S_j = {i | N_ij > 0 and valid C_ij}`, equal weights in
|
||||
the baseline) remains a **REPRODUCTION ASSUMPTION**. At 5G-NIDD alpha=0.50 it is material in
|
||||
only **4/27** class-condition cells, but the `beta=0` comparison is still conditional on both
|
||||
modes sharing this rule and does **not** prove `beta=0` reproduces original PROTEAN.
|
||||
- Train/validation/test adaptation (70/10/20 stratified) differs from the paper's literal 80/20;
|
||||
the validation eval partition is a project decision.
|
||||
- The union of rare classes (and hence global rare support) differs by seed; only the per-client
|
||||
mean metric is directly comparable within the condition.
|
||||
- `preprocessing_hash`/`config_hash` remain `null` in the 5G-NIDD manifests.
|
||||
- **No global beta optimum claim**; test metrics did not influence selection.
|
||||
@@ -0,0 +1,284 @@
|
||||
# Session 9 Post-Experiment Audit (5G-NIDD alpha=0.50)
|
||||
|
||||
Status: **PASS.** Independent recomputation from machine-readable artifacts and the raw dataset
|
||||
confirms every Session-9 Stage-B claim for 5G-NIDD `alpha=0.50`. No discovered issue changes
|
||||
`beta_star` or the scientific interpretation. No corrections were required. **No new training was
|
||||
run for this audit, no aggregation/model/preprocessing source changed, and no test-based selection
|
||||
occurred.**
|
||||
|
||||
Audit performed: 2026-09-28. Audited HEAD: `2d27abc35026456de765c929dbcf0503a72ac572`.
|
||||
Machine-readable audit record:
|
||||
`artifacts/analysis/5gnidd__alpha0.5__multiseed__audit_recomputation.json`.
|
||||
|
||||
Companion documents: `docs/research/5GNIDD_ALPHA050_BETA_VALIDATION.md`,
|
||||
`docs/research/EXPERIMENT_PROTOCOL.md`, `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md`,
|
||||
`docs/handoff/SESSION_HANDOFF.md`.
|
||||
|
||||
---
|
||||
|
||||
## A. Scope
|
||||
|
||||
Independently verify the Session-9 claim for:
|
||||
|
||||
- dataset `5gnidd`, alpha `0.50`, seeds `{0,1,2}`, beta `{0, 0.25, 0.5, 0.75, 1.0}` plus
|
||||
`original_protean` (18 runs);
|
||||
- PRIMARY pooled-validation selection `beta_star = 1.00` using validation per-client
|
||||
rare-class macro-F1 averaged within seed then across seeds;
|
||||
- reuse decision for the pre-existing seed-0 baseline;
|
||||
- `beta=0` equivalence class;
|
||||
- cross-dataset label (5G-NIDD 1.00 vs X-IIoTID alpha=0.50 0.75).
|
||||
|
||||
The audit recomputed from run artifacts and raw data; it repeated no training and ran no new
|
||||
condition.
|
||||
|
||||
---
|
||||
|
||||
## B. Run inventory
|
||||
|
||||
18 full-data runs enumerated from `results/*/metadata.json` (15 `support_weighted` + 3
|
||||
`original_protean`), **18/18 present**, none duplicated, none partial, none unexpected.
|
||||
|
||||
- Each run contains all 14 expected artifacts; the round-0 prototype snapshot exists for all 18
|
||||
runs (`artifacts/analysis/5gnidd__alpha0.5__seed*__*__prototype_snapshot.json`).
|
||||
- Metadata verified per run: dataset=5gnidd, alpha=0.5, seed, mode, clients=10, rounds=10,
|
||||
local_epochs=3, batch=256, lr=1e-3, mu=0.1, lambda=1.0, `manifest_reload_ok=true`,
|
||||
`smoke=false`. All 18 exit status 0.
|
||||
- No 5G-NIDD run exists for alpha=0.25 or alpha=0.75; no `support_weighted` 5G-NIDD run older than
|
||||
this session exists; `equal_available` is absent from `src/`, `scripts/`, `tests/`.
|
||||
|
||||
---
|
||||
|
||||
## C. Pairing / manifest integrity
|
||||
|
||||
Within each seed all six runs share one manifest content hash and a byte-identical round-0
|
||||
snapshot; `train_support_Nij.csv`, `characterization.csv` and `rare_unseen.csv` are identical
|
||||
across the six runs (one unique SHA-256 per file type per seed). `manifest_reload_ok = true` and
|
||||
`smoke = false` everywhere.
|
||||
|
||||
| seed | manifest content hash | round-0 snapshot SHA-256 | support/char/rare unique counts |
|
||||
|---|---|---|---|
|
||||
| 0 | `0732582cf4510089…` | `eb0839f2595727…` | 1 / 1 / 1 |
|
||||
| 1 | `f43906e111a8342c…` | `6cbd04e0dd97bb…` | 1 / 1 / 1 |
|
||||
| 2 | `fe8b4fc6f0486cf4…` | `f012a02b543b92…` | 1 / 1 / 1 |
|
||||
|
||||
Each alpha=0.5 manifest was independently reproduced from the raw dataset
|
||||
(`prepare_dataset` → `reproduce_train_assignments`): exact `N_ij` match and partition-hash match
|
||||
for all three seeds; split sizes 851,123 / 121,589 / 243,177; `N_ij` sum 851,123; 10 clients.
|
||||
Pairing is proven by content hashes, not filenames. As a side check, alpha=0.25/0.75 5G-NIDD
|
||||
manifests also reproduced exactly.
|
||||
|
||||
---
|
||||
|
||||
## D. Partition characterization audit
|
||||
|
||||
Independently recomputed from the manifests; matches the report exactly:
|
||||
|
||||
| statistic | seed 0 / 1 / 2 | reality (sum) |
|
||||
|---|---|---|
|
||||
| `|S_j| < M` cells | 2 / 0 / 2 | **4 / 27** |
|
||||
| missing client-class cells | 2 / 0 / 2 | **4** |
|
||||
| client-size ratio | 11.63 / 4.43 / 12.72 | mean **9.59** |
|
||||
| union rare classes | 6 / 5 / 6 | 2 rare per client (all) |
|
||||
| union unseen classes | 2 / 0 / 2 | max 1 on a client |
|
||||
|
||||
---
|
||||
|
||||
## E. Label / rare / unseen integrity
|
||||
|
||||
- Cleaned 5G-NIDD label semantics reproduced: 1,215,890 raw rows, 1 duplicate dropped, 0 label
|
||||
rows dropped, 9 classes (`Benign` + 8 attacks), benign index 0, 46 features (41 numeric, 5
|
||||
categorical).
|
||||
- Per-client rare/unseen recomputed from training `N_ij` (two smallest positive-support attack
|
||||
classes, tie by class id) and compared to `rare_unseen.csv` across all 18 runs: **270 cells
|
||||
checked, 0 mismatches**; benign class never flagged rare/unseen; rare and unseen never overlap;
|
||||
every client has exactly two rare classes.
|
||||
- `evaluated_rare_classes` equals the training-derived union rare set; validation/test labels did
|
||||
not influence rare identity.
|
||||
|
||||
---
|
||||
|
||||
## F. Validation metric recomputation
|
||||
|
||||
Recomputed from raw artifacts (no report numbers copied):
|
||||
|
||||
- seed-level primary score = mean over eligible clients of `val_per_client.csv`
|
||||
`rare_class_macro_f1`; equals `val_selection.json` per-client-mean to **max diff 0.0** for all
|
||||
15 runs; all 10 clients eligible in every seed.
|
||||
- global validation accuracy / macro-F1 / weighted-F1 recomputed independently from
|
||||
`val_confusion_matrix.json` with the production formulas: **max diff 0.0** versus
|
||||
`val_global_metrics.json`.
|
||||
|
||||
---
|
||||
|
||||
## G. Pooled-selection recomputation
|
||||
|
||||
Computed from the independently recomputed seed-level scores (not from the report). Pooled = mean
|
||||
of the three seed means; std sample (`ddof=1`).
|
||||
|
||||
| beta | pooled mean | sample std |
|
||||
|---|---|---|
|
||||
| 0.00 | 0.687230 | 0.052726 |
|
||||
| 0.25 | 0.645078 | 0.064743 |
|
||||
| 0.50 | 0.674669 | 0.102253 |
|
||||
| 0.75 | 0.688368 | 0.126032 |
|
||||
| **1.00** | **0.692370** | 0.139140 |
|
||||
|
||||
- Recompute **matches the report** to formatting precision.
|
||||
- `argmax` = **1.00** (no tie; top-two gap ≈ 4.0e-3). `tie_break_used=false`.
|
||||
- Per-seed diagnostics reproduce **1.00 / 0.00 / 1.00**.
|
||||
|
||||
---
|
||||
|
||||
## H. Validation / test leakage audit
|
||||
|
||||
- `run_baseline.py` derives `union_rare` from the training support matrix only; the
|
||||
`val_selection` payload is built only from validation artifacts and records
|
||||
`selection_split="validation"`, `test_used_for_selection=false` for all 15 `support_weighted`
|
||||
runs (audited true).
|
||||
- `analyze_multiseed_beta.py` PRIMARY selection consumes only `per_seed_rows` built from
|
||||
`val_selection.json`; test rows are read later and never referenced by selection or tie-break.
|
||||
No code path sorts/filters candidates by a test field.
|
||||
- The audit's own `beta_star = 1.00` was reproduced from validation artifacts alone.
|
||||
|
||||
Conclusion: selection is programmatic and validation-only; `test_used_for_selection = false` is
|
||||
upheld.
|
||||
|
||||
---
|
||||
|
||||
## I. Final test recomputation
|
||||
|
||||
`beta_star = 1.00` fixed from pooled validation; only then test metrics read. Recomputed from
|
||||
`test_descriptive.csv` (which is itself derived from `test_global_metrics.json`), averaged across
|
||||
seeds with sample std:
|
||||
|
||||
| metric | selected beta 1.00 (mean ± std) | original_protean (mean ± std) |
|
||||
|---|---|---|
|
||||
| rare-class macro-F1 | 0.871984 ± 0.028501 | 0.880542 ± 0.037058 |
|
||||
| rare-class recall | 0.967104 ± 0.007942 | 0.972913 ± 0.003843 |
|
||||
| macro-F1 | 0.838596 ± 0.019627 | 0.842395 ± 0.020840 |
|
||||
| accuracy | 0.743257 ± 0.000667 | 0.743975 ± 0.008393 |
|
||||
| weighted-F1 | 0.718195 ± 0.001680 | 0.718730 ± 0.008678 |
|
||||
|
||||
Matches the report exactly. No superiority claim is made.
|
||||
|
||||
---
|
||||
|
||||
## J. `beta = 0` equivalence audit
|
||||
|
||||
For seeds 0, 1, 2, `original_protean` vs `support_weighted beta=0`:
|
||||
|
||||
| property | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| same manifest / split / hashes | yes | yes | yes |
|
||||
| round-0 snapshot byte-identical | yes | yes | yes |
|
||||
| support/characterization/rare_unseen identical | yes | yes | yes |
|
||||
| independent aggregation diff | 4.4e-16 | 4.4e-16 | 4.4e-16 |
|
||||
| history rare-macro-F1 round-0 diff | 0.0 | 0.0 | 0.0 |
|
||||
| history rare-macro-F1 max diff (rounds ≥ 1) | 1.93e-2 | 6.01e-2 | 2.60e-2 |
|
||||
| test metric max diff | 2.33e-3 | 1.86e-4 | 8.70e-3 |
|
||||
| classification | **B** | **B** | **B** |
|
||||
|
||||
Class **B** (semantically equivalent + floating-point divergence); **no semantic discrepancy**.
|
||||
Equivalence remains conditional on the shared `S_j` reproduction assumption.
|
||||
|
||||
---
|
||||
|
||||
## K. Direct-effect audit
|
||||
|
||||
Independently recomputed `||Cbar_j(beta) − Cbar_j(0)||` from the fixed round-0 snapshots with the
|
||||
production aggregation: the seed-0 `...__prototype_effect.csv` is produced by
|
||||
`analyze_prototype_effect.py` from the snapshot and the cross-seed pooled file aggregates it.
|
||||
Mean displacement at `beta=1.0` is **2.340** (monotone: 0.953 / 1.567 / 2.003 / 2.340 for
|
||||
beta 0.25/0.50/0.75/1.00). Displacement is a direct-aggregation measure only; no detection
|
||||
causality is inferred.
|
||||
|
||||
---
|
||||
|
||||
## L. Client-level audit
|
||||
|
||||
Deltas recomputed from `val_per_client.csv` vs `client_validation_detail.csv` (via the
|
||||
`beta=0` base): improve/degrade/unchanged at the selected `beta=1.00` = 6/3/1 (seed 0,
|
||||
mean +0.030), 3/5/2 (seed 1, mean −0.101), 7/2/1 (seed 2, mean +0.087). `N_ij` vs ΔF1 is
|
||||
non-monotonic. Client ids are per-seed realisations, not pooled.
|
||||
|
||||
---
|
||||
|
||||
## M. Cross-dataset labels
|
||||
|
||||
Condition-level pooled selections confirmed from `selection_summary.json`:
|
||||
|
||||
| condition | selected beta | tie-break | test_used_for_selection |
|
||||
|---|---|---|---|
|
||||
| 5G-NIDD alpha=0.50 | 1.00 | no | false |
|
||||
| X-IIoTID alpha=0.50 | 0.75 | no | false |
|
||||
|
||||
The report makes no causal or generalising claim and explicitly states the selections differ
|
||||
descriptively across the two datasets.
|
||||
|
||||
---
|
||||
|
||||
## N. Reuse decision audit
|
||||
|
||||
The pre-existing `results/5gnidd__alpha0.5__seed0__original_protean` (backed up to
|
||||
`/tmp/opencode/s9/old_baseline_backup/`) lacks all validation artifacts and therefore failed the
|
||||
reuse criterion; all 18 runs were re-executed. The fresh seed-0 `original_protean` reproduces the
|
||||
old artifact **exactly**: byte-identical `history.json` and identical test metrics (rare-class
|
||||
macro-F1 `0.8719637151922778`, accuracy `0.7344156725348203`). Reuse decision confirmed.
|
||||
|
||||
---
|
||||
|
||||
## O. Artifact provenance
|
||||
|
||||
| Reported quantity | Source artifact(s) | Derivation |
|
||||
|---|---|---|
|
||||
| run existence / config | `results/<run>/metadata.json` | direct read |
|
||||
| manifest hashes / reproduction | `artifacts/partitions/5gnidd__alpha0.5__seed{0,1,2}.json` | module + `reproduce_train_assignments` |
|
||||
| pairing proof | `artifacts/analysis/...__prototype_snapshot.json` | sha256 across 6 runs/seed |
|
||||
| seed-level validation score | `results/.../val_per_client.csv` + `val_selection.json` | mean of per-client rare macro-F1 |
|
||||
| validation global metrics | `results/.../val_confusion_matrix.json` | independent numpy recompute |
|
||||
| pooled mean/std, `beta_star` | recomputed seed scores | mean/`stdev`, argmax |
|
||||
| test metrics | `results/.../test_global_metrics.json` + `...__test_descriptive.csv` | independent recompute + mean/std |
|
||||
| partition characterization | manifests + `...__partition_heterogeneity_*.csv` + `...__partition_characterization.csv` | independent numpy recompute |
|
||||
| beta=0 equivalence | original vs beta0 artifacts/snapshots | sha256 + independent aggregation + history diff |
|
||||
| direct-effect displacement | snapshots + `...__prototype_effect_pooled.csv` | production aggregation |
|
||||
| client improve/degrade | `results/.../val_per_client.csv` + `...__client_validation_detail.csv` | deltas vs beta=0 |
|
||||
| rare-class identity | `results/.../rare_unseen.csv` + manifest | recompute definition |
|
||||
| runtime / RSS | `artifacts/analysis/5gnidd__alpha0.5__multiseed__runtime.csv` (source `/tmp/opencode/5gnidd_alpha050_runtime.csv`) | parse `/usr/bin/time -v`; sum 69,336 s = 19.26 h |
|
||||
| cross-dataset table | `xiiotid__alpha0.5__multiseed__*` + `5gnidd__alpha0.5__multiseed__*` | read per-condition multiseed artifacts |
|
||||
|
||||
---
|
||||
|
||||
## P. Deviations / corrections found
|
||||
|
||||
**None.** No arithmetic, metric-label, std-convention, provenance or pairing defect was found in
|
||||
the Session 9 report or analysis artifacts. No training/aggregation/preprocessing source file was
|
||||
modified.
|
||||
|
||||
Informational note (not a defect): the pre-existing seed-0 baseline was structurally incomplete
|
||||
(no validation artifacts) and was correctly not reused; the fresh rerun is byte-identical for the
|
||||
training outputs.
|
||||
|
||||
---
|
||||
|
||||
## Q. Final audit verdict
|
||||
|
||||
**PASS.**
|
||||
|
||||
- Run inventory: **PASS** (18/18; no duplicates/partials; no alpha≠0.50 or legacy 5G-NIDD run).
|
||||
- Pairing / manifest reproduction: **PASS** (hash-identical; raw-data reproduction exact).
|
||||
- Partition characterization: **PASS** (4/27, 4 missing, ratio 11.63/4.43/12.72, unseen 2/0/2).
|
||||
- Rare/unseen integrity: **PASS** (0 mismatches over 270 cells).
|
||||
- Validation recomputation (seed-level and global): **PASS** (exact).
|
||||
- Pooled selection: **PASS**; independent `beta_star = 1.00` using validation only.
|
||||
- Validation/test leakage: **PASS**.
|
||||
- Test summaries: **PASS** (exact).
|
||||
- beta=0 equivalence: **PASS**, class **B** for all seeds.
|
||||
- Direct-effect and client-level analyses: **PASS** (consistent with snapshots/artifacts).
|
||||
- Reuse decision: **PASS** (not reused; rerun reproduces the old baseline exactly).
|
||||
- Cross-dataset labels: **PASS** (5G-NIDD 1.00 vs X-IIoTID 0.75; no causal claim).
|
||||
- Source/data integrity: **PASS**; `planningawal/` unchanged; raw SHA-256 unchanged;
|
||||
`equal_available` absent; pytest 75 passed.
|
||||
|
||||
Session 9 is **safe to checkpoint** with the documented limitations (single new condition, three
|
||||
project-defined seeds, weak effect relative to seed variance, `S_j` reproduction assumption, no
|
||||
global-optimum claim).
|
||||
Reference in new issue
Block a user