experiment: complete xiiotid three-alpha beta study
This commit is contained in:
willtanoe committed
1 parent
ff68d380e2
commit
2d27abc350
34 files changed
+39475
-221
No files matched your search
+242
-221
@@ -3,34 +3,33 @@
|
||||
This is the canonical continuation point for the next agent session. It is written to be
|
||||
readable without prior chat history, terminal scrollback, or git history.
|
||||
|
||||
Last updated: 2026-09-26 (Session 7, post-experiment audit complete). Session 7 type:
|
||||
**protocol-scaling / non-IID-severity experiment**. Session 6 completed X-IIoTID `alpha=0.50`
|
||||
for seeds `{0,1,2}`. Session 7 completed X-IIoTID **`alpha=0.25`** for seeds `{0,1,2}` across
|
||||
the five frozen beta candidates plus `original_protean`, using the **same pooled-across-seed
|
||||
validation-only** selection rule. **alpha=0.75 and 5G-NIDD were not run.** No commit or push
|
||||
was made.
|
||||
Last updated: 2026-09-26 (Session 8). Session 8 type: **protocol-scaling /
|
||||
non-IID-severity experiment**. Session 7 completed X-IIoTID `alpha=0.25`; Session 6 completed
|
||||
`alpha=0.50`. Session 8 completed X-IIoTID **`alpha=0.75`** for seeds `{0,1,2}` across the five
|
||||
frozen beta candidates plus `original_protean`, using the **same pooled-across-seed
|
||||
validation-only** selection rule, and produced the first **three-alpha (0.25/0.50/0.75)
|
||||
comparison** for X-IIoTID. **5G-NIDD was not run.** No commit or push was made.
|
||||
|
||||
> **Audit status (Session 7).** Independent audit verdict: **PASS WITH MINOR CORRECTIONS**
|
||||
> (`docs/research/SESSION7_AUDIT.md`). `beta_star(alpha=0.25) = 0.00` confirmed from validation
|
||||
> only; leakage audit PASS; `beta=0` class **B**; source/data integrity PASS; pytest 75 passed,
|
||||
> 0 failed, 0 skipped, 0 warnings. Corrections were documentation-only (cross-alpha label
|
||||
> disambiguation + one wording nuance); no `beta_star` or scientific interpretation changed.
|
||||
> **Session 7 is safe to checkpoint.**
|
||||
> **Audit status (Session 8).** Independent audit verdict: **PASS** (`docs/research/SESSION8_AUDIT.md`).
|
||||
> `beta_star(alpha=0.75) = 0.50` confirmed from validation only; no corrections required; run
|
||||
> inventory 18/18; pairing hash-identical and reproduced from raw data; leakage audit PASS;
|
||||
> `beta=0` class **B**; source/data integrity PASS; pytest 75 passed. **Session 8 is safe to
|
||||
> checkpoint.**
|
||||
|
||||
> **Headline (Session 7).** PRIMARY pooled-validation selected beta for X-IIoTID
|
||||
> `alpha=0.25` = **0.00** (validation only). The pooled validation curve is monotonically
|
||||
> **decreasing** in beta at alpha=0.25, whereas at alpha=0.50 it peaked at 0.75. Per-seed
|
||||
> diagnostics were 0.75 / 0.50 / 0.00. beta=0 is class **B** (semantically equivalent +
|
||||
> floating-point divergence) for all three seeds; no semantic discrepancy. The realized
|
||||
> alpha=0.25 partitions are materially more heterogeneous (19/27 class-condition cells with
|
||||
> `|S_j| < M` vs 7/27 at alpha=0.50). Effect remains small relative to between-seed variance.
|
||||
> **Headline (Session 8).** PRIMARY pooled-validation selected beta for X-IIoTID `alpha=0.75`
|
||||
> = **0.50** (validation only; per-seed diagnostics 0.50 / 0.50 / 0.00). The pooled validation
|
||||
> curve peaks at 0.50 and declines both sides, but the total between-beta spread (≈ 0.0081) is
|
||||
> much smaller than the between-seed std (≈ 0.045). `beta=0` is class **B** for all three seeds.
|
||||
> Realized alpha=0.75 partitions are the mildest of the three (3/27 cells with `|S_j| < M`,
|
||||
> 4 missing cells, client-ratio mean 4.67). **Three-alpha pooled selections: 0.25 → 0.00,
|
||||
> 0.50 → 0.75, 0.75 → 0.50 — not monotone in alpha.**
|
||||
|
||||
---
|
||||
|
||||
## 1. Project objective
|
||||
|
||||
Build the code and experiment infrastructure for a thesis study that modifies **only the
|
||||
server-side prototype aggregation** of PROTEAN (Chennoufi et al., 2025), a federated
|
||||
Build the code and experiment infrastructure for a thesis study that modifies **only** the
|
||||
server-side prototype aggregation of PROTEAN (Chennoufi et al., 2025), a federated
|
||||
intrusion-detection framework. The mechanism weights each contributing client's class
|
||||
prototype by its class sample support:
|
||||
|
||||
@@ -55,327 +54,350 @@ bitwise identical to the baseline (see §7).
|
||||
`~/datasets/5G-NIDD/5G-NIDD.csv`
|
||||
`fa36f80859585f474504ca69eb951a079b35145537976c86b00aae3aab46ee59`.
|
||||
- Frozen baseline (`original_protean`) source unchanged; no `equal_available` mode added;
|
||||
beta candidate set unchanged (`{0, .25, .5, .75, 1}`).
|
||||
beta candidate set unchanged (`{0, .25, .5, .75, 1}`). No tracked source/test/config file
|
||||
differs from HEAD.
|
||||
|
||||
## 3. Session 7 objective and outcome
|
||||
## 3. Session 8 objective and outcome
|
||||
|
||||
Objective: evaluate one additional non-IID severity (X-IIoTID `alpha=0.25`, seeds `{0,1,2}`)
|
||||
across the five beta candidates using the exact Session-6 three-seed pooled-validation
|
||||
protocol, and test whether the pooled-selected beta changes under a more heterogeneous
|
||||
condition. All 18 runs (15 `support_weighted` + 3 `original_protean`) completed with rc=0.
|
||||
Report: `docs/research/ALPHA025_BETA_VALIDATION.md`.
|
||||
Objective: evaluate the final X-IIoTID non-IID severity (`alpha=0.75`, seeds `{0,1,2}`) across
|
||||
the five beta candidates using the exact pooled-validation protocol, then compare alpha=0.25,
|
||||
0.50 and 0.75 using realized partition characteristics. All 18 runs (15 `support_weighted` +
|
||||
3 `original_protean`) completed with rc=0.
|
||||
|
||||
Outcome: **PRIMARY pooled-validation selected beta = 0.00** (validation only);
|
||||
`test_used_for_selection = false` for all 15 support_weighted runs. Pooled means: 0.00 →
|
||||
0.591704, 0.25 → 0.588691, 0.50 → 0.585851, 0.75 → 0.579463, 1.00 → 0.532099 (monotone
|
||||
decreasing). Between-seed std 0.045–0.090; the effect is small relative to seed variance.
|
||||
Outcome: **PRIMARY pooled-validation selected beta = 0.50** (validation only);
|
||||
`test_used_for_selection = false` for all 15 runs. Pooled means: 0.00 → 0.471706,
|
||||
0.25 → 0.474368, 0.50 → 0.476749, 0.75 → 0.472541, 1.00 → 0.468605 (peak at 0.50). Between-seed
|
||||
std 0.0447–0.0553; between-beta spread ≈ 0.0081, so the effect is small relative to seed
|
||||
variance.
|
||||
|
||||
Reports: `docs/research/ALPHA075_BETA_VALIDATION.md`,
|
||||
`docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md`, audit `docs/research/SESSION8_AUDIT.md`.
|
||||
|
||||
## 4. Condition and pairing
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| dataset / alpha / seeds | `xiiotid` / 0.25 / `{0,1,2}` |
|
||||
| dataset / alpha / seeds | `xiiotid` / 0.75 / `{0,1,2}` |
|
||||
| runs | 15 `support_weighted` + 3 `original_protean`, all new this session |
|
||||
| training | 10 clients, 10 rounds, 3 local epochs, batch 256, lr 1e-3, mu 0.1, lambda 1.0, Adam |
|
||||
| code revision | unchanged from HEAD `6b97a0d38b839c737c49f41006817503d0672871`; only docs/analysis artifacts added |
|
||||
| code revision | unchanged from HEAD `ff68d380e23859d21c07c8a3a7b56b1b7f06c91b`; only docs/analysis/scripts added |
|
||||
|
||||
Manifests (reused, never regenerated; one content hash per seed across all six runs; each was
|
||||
reproduced from the raw dataset with split/`N_ij`/partition hashes verified):
|
||||
Manifests (reused, never regenerated; one content hash per seed across all six runs; each
|
||||
reproduced from the raw dataset with split/`N_ij`/partition hash verified):
|
||||
|
||||
| seed | manifest | content hash | partition hash | split hash | rng seed |
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | `artifacts/partitions/xiiotid__alpha0.25__seed0.json` | `839ba18e…` | `60faf299…` | `d0acc0c1…` | 1617371214 |
|
||||
| 1 | `artifacts/partitions/xiiotid__alpha0.25__seed1.json` | `84c20b4b…` | `6ffb10ae…` | `880a9f82…` | 1670706564 |
|
||||
| 2 | `artifacts/partitions/xiiotid__alpha0.25__seed2.json` | `d6e68ee1…` | `c8e1ad31…` | `8e0db0be…` | 795318573 |
|
||||
| 0 | `artifacts/partitions/xiiotid__alpha0.75__seed0.json` | `759f7e50…` | `473b1c4a…` | `d0acc0c1…` | 957653162 |
|
||||
| 1 | `artifacts/partitions/xiiotid__alpha0.75__seed1.json` | `4a50ba21…` | `a0ec9e9c…` | `880a9f82…` | 762050403 |
|
||||
| 2 | `artifacts/partitions/xiiotid__alpha0.75__seed2.json` | `922b7bb1…` | `00345255…` | `8e0db0be…` | 843543396 |
|
||||
|
||||
Pairing proof: round-0 snapshot byte-identical across the five betas **and**
|
||||
`original_protean` within each seed — seed 0 `cd821beb…`, seed 1 `59d2d2bc…`, seed 2
|
||||
`47befeff…`. `train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv` identical
|
||||
across each seed's runs. `manifest_reload_ok = true`, `smoke = false` everywhere.
|
||||
Pairing proof: round-0 snapshot byte-identical across the five betas **and** `original_protean`
|
||||
within each seed — seed 0 `8c477df6…`, seed 1 `41f0187c…`, seed 2 `be2ad5fb…`.
|
||||
`train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv` identical across each seed's
|
||||
runs. `manifest_reload_ok = true`, `smoke = false` everywhere.
|
||||
|
||||
## 5. Validation results (USED FOR SELECTION) and pooled selection
|
||||
|
||||
Primary metric: validation per-client-mean rare-class macro-F1. All 10 clients eligible in
|
||||
every seed. Full tables: `artifacts/analysis/xiiotid__alpha0.25__multiseed__validation_per_seed.csv`.
|
||||
every seed. Full tables:
|
||||
`artifacts/analysis/xiiotid__alpha0.75__multiseed__validation_per_seed.csv`.
|
||||
|
||||
Per-seed validation per-client-mean rare-class macro-F1:
|
||||
|
||||
| beta | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| 0.00 | 0.616380 | 0.492425 | 0.666307 |
|
||||
| 0.25 | 0.609913 | 0.514237 | 0.641922 |
|
||||
| 0.50 | 0.595247 | 0.537040 | 0.625267 |
|
||||
| 0.75 | 0.654033 | 0.535672 | 0.548685 |
|
||||
| 1.00 | 0.578385 | 0.530942 | 0.486970 |
|
||||
| 0.00 | 0.496803 | 0.408276 | 0.510038 |
|
||||
| 0.25 | 0.496892 | 0.421386 | 0.504827 |
|
||||
| 0.50 | 0.504388 | 0.422919 | 0.502942 |
|
||||
| 0.75 | 0.486868 | 0.421053 | 0.509702 |
|
||||
| 1.00 | 0.479146 | 0.419534 | 0.507136 |
|
||||
|
||||
Pooled across seeds (`...__validation_pooled.csv`):
|
||||
|
||||
| beta | pooled mean | std (sample) | min | max | selected |
|
||||
|---|---|---|---|---|---|
|
||||
| **0.00** | **0.591704** | 0.089529 | 0.492425 | 0.666307 | **yes** |
|
||||
| 0.25 | 0.588691 | 0.066435 | 0.514237 | 0.641922 | |
|
||||
| 0.50 | 0.585851 | 0.044858 | 0.537040 | 0.625267 | |
|
||||
| 0.75 | 0.579463 | 0.064906 | 0.535672 | 0.654033 | |
|
||||
| 1.00 | 0.532099 | 0.045718 | 0.486970 | 0.578385 | |
|
||||
| 0.00 | 0.471706 | 0.055329 | 0.408276 | 0.510038 | |
|
||||
| 0.25 | 0.474368 | 0.046055 | 0.421386 | 0.504827 | |
|
||||
| **0.50** | **0.476749** | 0.046624 | 0.422919 | 0.504388 | **yes** |
|
||||
| 0.75 | 0.472541 | 0.046028 | 0.421053 | 0.509702 | |
|
||||
| 1.00 | 0.468605 | 0.044742 | 0.419534 | 0.507136 | |
|
||||
|
||||
- **PRIMARY selected beta (`beta_star_alpha025`) = 0.00**, max pooled validation
|
||||
- **PRIMARY selected beta (`beta_star_alpha075`) = 0.50**, max pooled validation
|
||||
per-client-mean rare-class macro-F1; **no tie-break used**; selection is validation-only.
|
||||
- **SECONDARY per-seed diagnostics only:** seed 0 → 0.75, seed 1 → 0.50, seed 2 → 0.00.
|
||||
- **SECONDARY per-seed diagnostics only:** seed 0 → 0.50, seed 1 → 0.50, seed 2 → 0.00.
|
||||
|
||||
## 6. Final test evaluation of the pooled-selected beta
|
||||
|
||||
`beta_star = 0.00` was fixed from pooled validation before test was read.
|
||||
`beta_star = 0.50` was fixed from pooled validation before test was read.
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.641362 | 0.580555 | 0.674489 | 0.632136 ± 0.047642 |
|
||||
| rare-class recall | 0.830701 | 0.825699 | 0.807370 | 0.821257 ± 0.012283 |
|
||||
| macro-F1 | 0.674558 | 0.665039 | 0.704433 | 0.681343 ± 0.020555 |
|
||||
| accuracy | 0.874551 | 0.863490 | 0.884002 | 0.874015 ± 0.010267 |
|
||||
| weighted-F1 | 0.906156 | 0.899864 | 0.914627 | 0.906882 ± 0.007408 |
|
||||
| rare-class macro-F1 | 0.511889 | 0.624144 | 0.679704 | 0.605246 ± 0.085489 |
|
||||
| rare-class recall | 0.803887 | 0.876726 | 0.865681 | 0.848765 ± 0.039256 |
|
||||
| macro-F1 | 0.741977 | 0.761752 | 0.769179 | 0.757636 ± 0.014060 |
|
||||
| accuracy | 0.914365 | 0.932611 | 0.937352 | 0.928110 ± 0.012137 |
|
||||
| weighted-F1 | 0.935566 | 0.948344 | 0.951740 | 0.945216 ± 0.008528 |
|
||||
|
||||
`original_protean` for the same seeds/condition:
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.639810 | 0.582522 | 0.676039 | 0.632790 ± 0.047152 |
|
||||
| rare-class recall | 0.829601 | 0.826700 | 0.809595 | 0.821965 ± 0.010811 |
|
||||
| macro-F1 | 0.673103 | 0.666981 | 0.705969 | 0.682018 ± 0.020967 |
|
||||
| accuracy | 0.873722 | 0.867341 | 0.885231 | 0.875431 ± 0.009067 |
|
||||
| weighted-F1 | 0.905466 | 0.902075 | 0.915416 | 0.907652 ± 0.006934 |
|
||||
| rare-class macro-F1 | 0.521540 | 0.614061 | 0.676068 | 0.603890 ± 0.077764 |
|
||||
| rare-class recall | 0.816440 | 0.883238 | 0.869835 | 0.856504 ± 0.035338 |
|
||||
| macro-F1 | 0.742471 | 0.747395 | 0.765680 | 0.751849 ± 0.012228 |
|
||||
| accuracy | 0.913254 | 0.913898 | 0.934576 | 0.920576 ± 0.012129 |
|
||||
| weighted-F1 | 0.932563 | 0.935937 | 0.948908 | 0.939136 ± 0.008629 |
|
||||
|
||||
Descriptive difference for test rare-class macro-F1 (`beta_star` − `original_protean`) is
|
||||
+0.001552 / −0.001967 / −0.001550 (mean −0.000655) — i.e. essentially identical, as expected
|
||||
because `beta = 0` is mathematically the equal mean over `S_j`. Test is descriptive only; no
|
||||
superiority or generalisation is claimed.
|
||||
−0.009651 / +0.010083 / +0.003636 (mean +0.001356). Test is descriptive only; no superiority
|
||||
or generalisation claimed.
|
||||
|
||||
## 7. `beta = 0` equivalence result (all three seeds)
|
||||
|
||||
For seeds 0, 1, 2, `support_weighted beta=0` vs frozen `original_protean`:
|
||||
|
||||
- Same manifest/split/hashes: **yes**.
|
||||
- Round-0 snapshot byte-identical: **yes** (`cd821beb…`, `59d2d2bc…`, `47befeff…`).
|
||||
- Round-0 snapshot byte-identical: **yes** (`8c477df6…`, `41f0187c…`, `be2ad5fb…`).
|
||||
- `train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv`: **identical**.
|
||||
- Independent aggregation (`ΣC/n` vs `(1/n)ΣC`): ≤ 4.4e-16 (1 ULP ordering).
|
||||
- Test/history/per-client/per-class: **differ slightly** (history rare-macro-F1 max diff
|
||||
3.6e-3–6.6e-3; test metric max diff 1.6e-3–3.9e-3).
|
||||
- Independent aggregation (`ΣC/n` vs `(1/n)ΣC`): 4.4e-16 / 4.4e-16 / 2.2e-16.
|
||||
- History rare-macro-F1 max diff (rounds ≥ 1): 1.10e-2 / 8.12e-3 / 3.64e-3; test metric max
|
||||
diff 4.33e-3 / 1.14e-3 / 0.85e-3.
|
||||
- Classification: **B — mathematically equivalent + floating-point numerical divergence; no
|
||||
semantic discrepancy** in any seed. Machine-readable:
|
||||
`artifacts/analysis/xiiotid__alpha0.25__multiseed__beta0_equivalence.csv`.
|
||||
`artifacts/analysis/xiiotid__alpha0.75__multiseed__beta0_equivalence.csv`.
|
||||
|
||||
Conditional on the shared `S_j` reproduction assumption; not proof `beta=0` reproduces
|
||||
original PROTEAN.
|
||||
Conditional on the shared `S_j` reproduction assumption; not proof `beta=0` reproduces original
|
||||
PROTEAN.
|
||||
|
||||
## 8. Partition characterization (realized)
|
||||
|
||||
`|S_j| < M` class-condition cells and missing client-class cells (training split):
|
||||
`|S_j| < M` class-condition cells, missing client-class cells, and client-size ratios
|
||||
(training split):
|
||||
|
||||
| condition | `|S_j| < M` cells | missing client-class cells | client-size max/min (seeds) |
|
||||
|---|---|---|---|
|
||||
| alpha=0.25 | **19 / 27** | **38** | 15.6, 20.8, 7.8 |
|
||||
| alpha=0.50 | 7 / 27 | 9 | 5.1, 7.2, 9.9 |
|
||||
| condition | `|S_j| < M` cells | missing client-class cells | client-size ratio (seeds) | union unseen | mean class CV |
|
||||
|---|---|---|---|---|---|
|
||||
| alpha=0.25 | **19 / 27** | **38** | 15.6, 20.8, 7.8 | 6/6/5 | 1.442 |
|
||||
| alpha=0.50 | 7 / 27 | 9 | 5.1, 7.2, 9.9 | 3/3/1 | 1.183 |
|
||||
| alpha=0.75 | **3 / 27** | **4** | 4.15, 5.07, 4.78 | 2/1/0 | 0.881 |
|
||||
|
||||
Union unseen classes per seed: alpha=0.25 = 6/6/5 (max 4 on a client); alpha=0.50 = 3/3/1
|
||||
(max 2). Union rare classes per seed: alpha=0.25 = 8/7/8; alpha=0.50 = 6/7/7. All clients have
|
||||
exactly two rare classes at both alphas. Full per-class `N_ij`, CV and max/min in
|
||||
`docs/research/ALPHA025_BETA_VALIDATION.md` §B. The lower alpha does **not** uniformly increase
|
||||
every realized statistic (e.g. alpha=0.25 seed 2 client-size ratio 7.8 < alpha=0.50 seed 2's
|
||||
9.9), so interpretation uses the realized partitions, not alpha alone.
|
||||
Union rare classes per seed: alpha=0.75 = 4/5/6; all clients have exactly two rare classes.
|
||||
Full per-class `N_ij`, CV and max/min in `docs/research/ALPHA075_BETA_VALIDATION.md` §C and
|
||||
`artifacts/analysis/xiiotid__cross_alpha__partition_heterogeneity_*.csv`. Heterogeneity
|
||||
decreases with alpha overall but is **not uniformly monotone per seed** (alpha=0.75 seed 0
|
||||
ratio 4.15 < seed 2's 4.78; alpha=0.50 seed 2 ratio 9.9 > alpha=0.25 seed 2's 7.8), so
|
||||
interpretation uses realized partitions, not alpha alone.
|
||||
|
||||
## 9. Direct prototype-effect findings (fixed round-0 snapshots, no retraining)
|
||||
|
||||
`||Cbar_j(1) - Cbar_j(0)||`, cross-seed mean (std), alpha=0.25 vs alpha=0.50:
|
||||
`||Cbar_j(1) - Cbar_j(0)||`, cross-seed mean, alpha=0.25 vs 0.50 vs 0.75 (mean across classes
|
||||
3.422 / 2.099 / **1.305**):
|
||||
|
||||
| class | a25 mean (std) | a50 mean (std) |
|
||||
|---|---|---|
|
||||
| Exfiltration | 4.316 (0.898) | 2.020 (0.502) |
|
||||
| Tampering | 4.248 (0.241) | 2.354 (1.165) |
|
||||
| C&C | 4.020 (0.290) | 1.694 (0.505) |
|
||||
| Lateral_movement | 3.726 (1.383) | 2.724 (1.035) |
|
||||
| Exploitation | 3.464 (1.113) | 2.344 (0.412) |
|
||||
| Reconnaissance | 3.339 (1.107) | 1.002 (0.261) |
|
||||
| Weaponization | 3.124 (0.585) | 2.561 (1.228) |
|
||||
| RDOS | 2.566 (1.219) | 2.631 (1.165) |
|
||||
| Normal | 1.997 (0.533) | 1.557 (0.548) |
|
||||
| class | a25 | a50 | a75 |
|
||||
|---|---|---|---|
|
||||
| Tampering | 4.248 | 2.354 | 1.934 |
|
||||
| Exfiltration | 4.316 | 2.020 | 1.448 |
|
||||
| Exploitation | 3.464 | 2.344 | 1.539 |
|
||||
| Lateral_movement | 3.726 | 2.724 | 1.369 |
|
||||
| C&C | 4.020 | 1.694 | 1.284 |
|
||||
| Weaponization | 3.124 | 2.561 | 1.472 |
|
||||
| Reconnaissance | 3.339 | 1.002 | 0.983 |
|
||||
| Normal | 1.997 | 1.557 | 0.903 |
|
||||
| RDOS | 2.566 | 2.631 | 0.814 |
|
||||
|
||||
Mean displacement across classes is larger at alpha=0.25 (3.42) than alpha=0.50 (2.10), because
|
||||
support is more imbalanced and more classes are missing contributors. Displacement is a
|
||||
direct-aggregation measure only, not a detection result. Outputs:
|
||||
`artifacts/analysis/xiiotid__alpha0.25__multiseed__prototype_effect_pooled.csv` and per-seed
|
||||
Displacement tracks realized support imbalance and is a direct-aggregation measure only, not a
|
||||
detection result. Outputs:
|
||||
`artifacts/analysis/xiiotid__alpha0.75__multiseed__prototype_effect_pooled.csv` and per-seed
|
||||
`...__seed{0,1,2}__support_weighted__beta0__prototype_snapshot__prototype_effect.csv`.
|
||||
|
||||
## 10. Client-level findings (validation, vs beta = 0)
|
||||
|
||||
Detail: `...__multiseed__client_validation_detail.csv`; summary:
|
||||
`...__client_validation_summary.csv`.
|
||||
Detail: `...__client_validation_detail.csv`; summary: `...__client_validation_summary.csv`; and
|
||||
`artifacts/analysis/xiiotid__cross_alpha__cross_alpha_client_summary.csv`.
|
||||
|
||||
- At alpha=0.25 the higher-beta response is **seed-dependent and often negative**:
|
||||
improve/degrade/unchanged at beta=0.75 = 5/4/1 (seed 0, mean Δ +0.038), 5/4/1 (seed 1,
|
||||
+0.043), 3/5/2 (seed 2, −0.118). At beta=1.00 seed 2 degrades 6/10 clients with a worst
|
||||
client Δ = −0.678, while seed 1 still gains on average.
|
||||
- This **contrasts with alpha=0.50**, where beta=0.75 improved the majority of clients in every
|
||||
seed. Client ids are per-seed realisations and are not one population; the analysis is
|
||||
descriptive and non-causal, and `N_ij` vs ΔF1 remains non-monotonic.
|
||||
- At alpha=0.75, the response is smaller and seed-dependent. At the selected `beta=0.50`:
|
||||
6/4/0 (seed 0, mean Δ +0.008), 9/0/1 (seed 1, +0.015), 6/4/0 (seed 2, −0.007; worst client
|
||||
−0.152).
|
||||
- At a common `beta=0.75`: alpha=0.25 = 5/4/1, 5/4/1, 3/5/2 (often negative); alpha=0.50 =
|
||||
7/3/0, 8/1/1, 8/1/1 (majority improve); alpha=0.75 = 3/7/0, 9/0/1, 5/5/0.
|
||||
- `N_ij` vs ΔF1 remains non-monotonic at every alpha. Client ids are per-seed realisations and
|
||||
are not pooled.
|
||||
|
||||
## 11. Comparison with alpha=0.50 (RQ3)
|
||||
## 11. Three-alpha comparison (RQ3)
|
||||
|
||||
| | alpha=0.50 | alpha=0.25 |
|
||||
|---|---|---|
|
||||
| pooled-selected beta | 0.75 | **0.00** |
|
||||
| pooled validation curve | rises to 0.75 | monotone decreasing |
|
||||
| pooled mean range | 0.544–0.579 | 0.532–0.592 |
|
||||
| pooled std range | 0.075–0.100 | 0.045–0.090 |
|
||||
| `|S_j| < M` cells | 7/27 | 19/27 |
|
||||
| mean `||Cbar(1)−Cbar(0)||` | 2.10 | 3.42 |
|
||||
| | alpha=0.25 | alpha=0.50 | alpha=0.75 |
|
||||
|---|---|---|---|
|
||||
| pooled-selected beta | **0.00** | **0.75** | **0.50** |
|
||||
| pooled validation curve | monotone decreasing | rises to peak at 0.75 | peaks at 0.50 |
|
||||
| pooled mean range (spread) | 0.532–0.592 (0.060) | 0.544–0.579 (0.035) | 0.469–0.477 (0.008) |
|
||||
| pooled std range | 0.045–0.090 | 0.075–0.100 | 0.045–0.055 |
|
||||
| `|S_j| < M` cells | 19/27 | 7/27 | 3/27 |
|
||||
| mean `||Cbar(1)−Cbar(0)||` | 3.422 | 2.099 | 1.305 |
|
||||
|
||||
The selection **changes** with non-IID severity. This is documented as a descriptive
|
||||
association accompanied by the realized partition characteristics; **no causal claim is made
|
||||
that alpha by itself caused the difference**. The effect is small relative to between-seed
|
||||
variance at both settings.
|
||||
The pooled-selected beta **changes** with the non-IID condition (0.00 → 0.75 → 0.50) but is
|
||||
**not monotone** in alpha. This is documented as a descriptive association accompanied by the
|
||||
realized partition characteristics; **no causal claim** is made. With only three alpha levels,
|
||||
three project-defined seeds, one dataset and no inferential trend test, monotonicity is not
|
||||
claimed.
|
||||
|
||||
> **Audited label (do not confuse).** Condition-level **pooled** selections are alpha=0.50 →
|
||||
> **0.75** and alpha=0.25 → **0.00**. The value **0.50** is only the Session-5 **seed-0
|
||||
> single-condition** choice and the Session-6 alpha=0.50 **per-seed diagnostic** for seed 0;
|
||||
> it is not the alpha=0.50 pooled selection. Any "0.50 → 0.00" phrasing for condition-level
|
||||
> pooled selection is wrong. Verified in `docs/research/SESSION7_AUDIT.md` §M.
|
||||
> **Audited label (do not confuse).** Condition-level **pooled** selections are alpha=0.25 →
|
||||
> **0.00**, alpha=0.50 → **0.75**, alpha=0.75 → **0.50**. The value **0.50** is *not* the
|
||||
> alpha=0.50 pooled selection: it is only the Session-5 **seed-0 single-condition** choice, the
|
||||
> Session-6 alpha=0.50 **per-seed diagnostic** for seed 0, and now the alpha=0.75 pooled
|
||||
> selection. See `docs/research/SESSION7_AUDIT.md` §M and `SESSION8_AUDIT.md` §M.
|
||||
|
||||
## 12. Runtime / resources
|
||||
|
||||
CPU only, no GPU, no swap, exit 0 on all 18 runs. Per-run wall-clock 39:49–42:12; CPU
|
||||
741–746 %; peak RSS 8.38 GiB (seed 0), 10.80 GiB (seed 1), 7.51 GiB (seed 2). New compute:
|
||||
18 runs, ≈ 43,799 s ≈ **12.17 h** wall-clock, sequential (8 cores). Machine-readable:
|
||||
`artifacts/analysis/xiiotid__alpha0.25__multiseed__runtime.csv`.
|
||||
CPU only, no GPU, no swap, exit 0 on all 18 runs. Per-run wall-clock 40:00–49:02 (two runs
|
||||
slower under transient host contention); CPU 714–746 %; peak RSS 6.49 GiB (seed 0), 6.91 GiB
|
||||
(seed 1), 7.20 GiB (seed 2). New compute: 18 runs, 44,371 s ≈ **12.33 h** wall-clock,
|
||||
sequential (8 cores). Machine-readable:
|
||||
`artifacts/analysis/xiiotid__alpha0.75__multiseed__runtime.csv`.
|
||||
|
||||
## 13. Methodological decisions / assumptions
|
||||
|
||||
1. **`S_j` participant/denominator rule** (`S_j = {i | N_ij > 0 and valid C_ij}`, equal
|
||||
weights in the baseline) is a **REPRODUCTION ASSUMPTION**; material in 19/27
|
||||
class-condition cells at alpha=0.25 (7/27 at alpha=0.50). Carry this caveat in all reporting.
|
||||
1. **`S_j` participant/denominator rule** (`S_j = {i | N_ij > 0 and valid C_ij}`, equal weights
|
||||
in the baseline) is a **REPRODUCTION ASSUMPTION**; material in 19/27 class-condition cells at
|
||||
alpha=0.25, 7/27 at alpha=0.50, 3/27 at alpha=0.75. Carry this caveat in all reporting.
|
||||
2. `N_ij` is class-specific sample support, not prototype quality.
|
||||
3. Beta is one value for all classes/clients; candidates frozen at `{0,.25,.5,.75,1}`.
|
||||
4. **PRIMARY selection = pooled validation across clients and seeds**; per-seed selection is
|
||||
secondary stability diagnostics; test used only after the pooled beta is fixed.
|
||||
5. The `scripts/analyze_multiseed_beta.py` already supported `--alpha` (default 0.5) and
|
||||
already includes alpha in output names; **no code change was needed**. alpha=0.50 outputs
|
||||
were re-run and are byte-identical to the Session-6 artifacts (reproducibility verified).
|
||||
5. Three new read-only analysis scripts were added this session (no training/aggregation code
|
||||
changed): `scripts/analyze_partition_heterogeneity.py`,
|
||||
`scripts/verify_beta0_equivalence.py`, `scripts/analyze_cross_alpha.py`.
|
||||
6. `lambda=1.0`, Adam, lr 1e-3, batch 256 remain PROJECT DECISIONS. No `equal_available` mode.
|
||||
|
||||
## 14. Unresolved questions / blockers
|
||||
|
||||
1. `S_j` reproduction ambiguity — unresolved; no official PROTEAN code exists.
|
||||
2. **Pooled beta selection remains statistically weak at both alphas (descriptive only):**
|
||||
the across-seed sample std (0.045–0.090) is comparable to, or larger than, the total
|
||||
between-beta pooled-mean spread (0.035 at alpha=0.50, 0.060 at alpha=0.25). No inferential
|
||||
test was performed, so this is a descriptive comparison, not evidence of insignificance.
|
||||
3. `beta = 0` is not bitwise identical to `original_protean`; documentation decision on how to
|
||||
report this remains open.
|
||||
2. **Pooled beta selection remains statistically weak at all three alphas (descriptive only):**
|
||||
the across-seed sample std is comparable to or larger than the between-beta spread. No
|
||||
inferential test was performed.
|
||||
3. `beta = 0` is not bitwise identical to `original_protean`; the documentation decision on how
|
||||
to report this remains open.
|
||||
4. Session-2 manifests still store `preprocessing_hash = null` / `config_hash = null`.
|
||||
5. Only alpha=0.25 and 0.50, X-IIoTID, three seeds measured. No 5G-NIDD beta result.
|
||||
6. The alpha=0.25 result (pooled-selected beta = 0) is **one additional condition, not yet a
|
||||
trend across alpha**. Whether any relationship between non-IID severity and the selected beta
|
||||
is monotone would require alpha=0.75 (and is still not implied by two conditions).
|
||||
5. Only alpha=0.25/0.50/0.75, X-IIoTID, three seeds measured. No 5G-NIDD beta result.
|
||||
6. The three-alpha selected-beta sequence (0.00 / 0.75 / 0.50) is **not monotone**; no trend
|
||||
claim is made and no inference has been run.
|
||||
|
||||
## 15. Git status
|
||||
|
||||
No commit or push was made. `git status --short` shows only untracked docs/analysis/run
|
||||
artifacts (`results/` is gitignored; `artifacts/analysis/` is intentionally tracked):
|
||||
artifacts and the three new scripts (`results/` is gitignored; `artifacts/analysis/` is
|
||||
intentionally tracked):
|
||||
|
||||
```
|
||||
M docs/handoff/SESSION_HANDOFF.md
|
||||
?? docs/research/ALPHA025_BETA_VALIDATION.md
|
||||
?? docs/research/SESSION7_AUDIT.md
|
||||
?? artifacts/analysis/xiiotid__alpha0.25__* (runs' snapshots, effect CSVs, multiseed tables)
|
||||
?? (pre-existing Session-5/6 untracked alpha=0.5 views, unchanged)
|
||||
?? docs/research/ALPHA075_BETA_VALIDATION.md
|
||||
?? docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md
|
||||
?? docs/research/SESSION8_AUDIT.md
|
||||
?? scripts/analyze_cross_alpha.py
|
||||
?? scripts/analyze_partition_heterogeneity.py
|
||||
?? scripts/verify_beta0_equivalence.py
|
||||
?? artifacts/analysis/xiiotid__alpha0.75__* (runs' snapshots, effect CSVs, multiseed tables)
|
||||
?? artifacts/analysis/xiiotid__cross_alpha__* (heterogeneity, cross-alpha tables)
|
||||
?? (pre-existing Session-5/6/7 untracked alpha=0.25/0.5 views, unchanged)
|
||||
```
|
||||
|
||||
`planningawal/` and both raw datasets are unchanged (audit re-verified SHA-256:
|
||||
X-IIoTID `7b929005…`, 5G-NIDD `fa36f808…`). No model/training/aggregation source changed
|
||||
(working-tree blobs equal HEAD). `artifacts/partitions/` unmodified. No alpha=0.75 run and no
|
||||
5G-NIDD `support_weighted` run exists. `equal_available` absent.
|
||||
`HEAD = 6b97a0d38b839c737c49f41006817503d0672871`. No commit or push was made.
|
||||
No tracked source/test/config file differs from HEAD. `planningawal/` and both raw datasets are
|
||||
unchanged (audit re-verified SHA-256: X-IIoTID `7b929005…`, 5G-NIDD `fa36f808…`).
|
||||
`artifacts/partitions/` unmodified. No 5G-NIDD `support_weighted` run exists. `equal_available`
|
||||
absent. `HEAD = ff68d380e23859d21c07c8a3a7b56b1b7f06c91b`. No commit or push was made.
|
||||
|
||||
## 15b. Session 7 audit result (independent)
|
||||
## 16. Session 8 audit result
|
||||
|
||||
- Verdict: **PASS WITH MINOR CORRECTIONS** (`docs/research/SESSION7_AUDIT.md`).
|
||||
- Verdict: **PASS** (`docs/research/SESSION8_AUDIT.md`).
|
||||
- Run inventory 18/18, pairing hash-identical, manifests reproduced from raw data.
|
||||
- Validation recomputed exactly; **pooled `beta_star(alpha=0.25) = 0.00`** (validation only,
|
||||
monotone decreasing in beta). `beta_star` test and `original_protean` test means/stds exact.
|
||||
- Leakage audit: selection reads validation only; `test_used_for_selection=false` upheld.
|
||||
- Validation recomputed exactly; **pooled `beta_star(alpha=0.75) = 0.50`** (validation only).
|
||||
`beta_star` test and `original_protean` test means/stds exact.
|
||||
- Leakage audit PASS; `test_used_for_selection=false` upheld.
|
||||
- `beta=0` equivalence: class **B** for all three seeds (no semantic discrepancy).
|
||||
- Source/data integrity: PASS. pytest: **75 passed, 0 failed, 0 skipped, 0 warnings**.
|
||||
- Corrections: cross-alpha label disambiguation (documentation-only) and one wording nuance.
|
||||
No arithmetic, metric-label, std-convention, or provenance defect was found.
|
||||
- Direct-effect, client-level, rare/unseen, cross-alpha labels: PASS.
|
||||
- Source/data integrity PASS. pytest: **75 passed, 0 failed, 0 skipped, 0 warnings**.
|
||||
|
||||
## 16. Files created / modified this session
|
||||
## 17. Files created / modified this session
|
||||
|
||||
Created:
|
||||
|
||||
- `docs/research/ALPHA025_BETA_VALIDATION.md`
|
||||
- `docs/research/SESSION7_AUDIT.md` (independent post-experiment audit)
|
||||
- `artifacts/analysis/xiiotid__alpha0.25__seed{0,1,2}__*__prototype_snapshot.json`
|
||||
- `artifacts/analysis/xiiotid__alpha0.25__seed{0,1,2}__support_weighted__beta0__prototype_snapshot__{prototype_effect,prototype_weights}.csv`
|
||||
- `artifacts/analysis/xiiotid__alpha0.25__multiseed__{validation_per_seed,validation_pooled,client_validation_deltas,client_validation_detail,client_validation_summary,prototype_effect_pooled,test_descriptive,runtime,beta0_equivalence}.csv` and `...__selection_summary.json`
|
||||
- `results/xiiotid__alpha0.25__seed{0,1,2}__*` (18 run dirs, gitignored)
|
||||
- `docs/research/ALPHA075_BETA_VALIDATION.md`
|
||||
- `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md`
|
||||
- `docs/research/SESSION8_AUDIT.md`
|
||||
- `scripts/analyze_partition_heterogeneity.py`
|
||||
- `scripts/verify_beta0_equivalence.py`
|
||||
- `scripts/analyze_cross_alpha.py`
|
||||
- `artifacts/analysis/xiiotid__alpha0.75__*` (run snapshots, effect/weights CSVs, multiseed
|
||||
tables, `beta0_equivalence.csv`, `runtime.csv`)
|
||||
- `artifacts/analysis/xiiotid__cross_alpha__*` (partition heterogeneity per-seed/per-class/summary,
|
||||
cross-alpha beta table/displacement/client summary/summary)
|
||||
- `results/xiiotid__alpha0.75__seed{0,1,2}__*` (18 run dirs, gitignored)
|
||||
|
||||
Modified:
|
||||
|
||||
- `docs/handoff/SESSION_HANDOFF.md` (this file)
|
||||
|
||||
No source or test code was changed. `scripts/analyze_multiseed_beta.py` was re-run for
|
||||
alpha=0.50 and produced byte-identical outputs.
|
||||
No source or test code was changed.
|
||||
|
||||
## 17. Exact recommended next phase
|
||||
## 18. Exact recommended next phase
|
||||
|
||||
1. **Session 7 is audited and safe to checkpoint** (verdict PASS WITH MINOR CORRECTIONS;
|
||||
see §15b and `docs/research/SESSION7_AUDIT.md`). alpha=0.25 behaved cleanly: pairing valid,
|
||||
beta=0 class B, pooled selection clean, artifacts complete, no test leakage, runtime
|
||||
manageable (≈12 h per alpha). Before checkpointing, stage only the intended docs/analysis
|
||||
artifacts; do not commit `results/` or the raw datasets.
|
||||
2. **Next experiment:** run X-IIoTID `alpha=0.75` under the same protocol (seeds `{0,1,2}`,
|
||||
five betas + `original_protean`, sequential, reusing frozen manifests, pooled
|
||||
validation-only selection), then test evaluation, direct-effect, client-level, and
|
||||
cross-alpha comparison. This would complete the three planned non-IID severities. Do **not**
|
||||
infer a monotone alpha→beta relation from two conditions.
|
||||
3. Do **not** treat `beta = 0.00` (alpha=0.25) or `0.75` (alpha=0.50) as optima. Report each as
|
||||
"best among tested candidates under pooled validation for that condition".
|
||||
4. **Do not launch 5G-NIDD or the full grid** until alpha=0.75 is inspected.
|
||||
5. Use `.venv/bin/python`. Run one alpha at a time; measure runtime as before.
|
||||
6. Update `docs/research/ALPHA025_BETA_VALIDATION.md` (or add `ALPHA075_BETA_VALIDATION.md`),
|
||||
`EXPERIMENT_PROTOCOL.md` only if a real ambiguity appears, and this handoff. Do not commit or
|
||||
push unless asked.
|
||||
1. **Session 8 is audited and safe to checkpoint** (verdict PASS; see §16 and
|
||||
`docs/research/SESSION8_AUDIT.md`). X-IIoTID alpha=0.75 behaved cleanly. Before
|
||||
checkpointing, stage only the intended docs/scripts/analysis artifacts; do not commit
|
||||
`results/` or the raw datasets.
|
||||
2. **X-IIoTID three-alpha study is complete** (0.25, 0.50, 0.75 × seeds `{0,1,2}`), with
|
||||
audited condition-level pooled selections 0.00 / 0.75 / 0.50 and a cross-alpha report.
|
||||
3. **Next major phase: 5G-NIDD.** Follow the same protocol (the planned alphas `{0.75, 0.50,
|
||||
0.25}`, seeds `{0,1,2}`, five betas + `original_protean`, frozen manifests, pooled
|
||||
validation-only selection). Do **not** launch the full grid blindly: verify 5G-NIDD manifests
|
||||
exist and reproduce, run the same one-condition-at-a-time pacing, and measure runtime first.
|
||||
A single smoke/one-condition run before scaling is recommended.
|
||||
4. Do **not** treat any selected beta as an optimum. Report each as "best among tested
|
||||
candidates under pooled validation for that condition".
|
||||
5. Do **not** infer a monotone alpha→beta relation from three X-IIoTID conditions.
|
||||
6. Update `EXPERIMENT_PROTOCOL.md` only if a real ambiguity appears; update this handoff.
|
||||
7. Do not commit or push unless asked.
|
||||
|
||||
## 18. Important commands
|
||||
## 19. Important commands
|
||||
|
||||
```bash
|
||||
# Tests (75 expected)
|
||||
.venv/bin/python -m pytest
|
||||
|
||||
# One full paired beta run (CPU, ~40 min, ~7.5-11.3 GiB)
|
||||
.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.25 --seed 1 \
|
||||
# One full paired beta run (CPU, ~40 min, ~6.5-7.2 GiB)
|
||||
.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.75 --seed 1 \
|
||||
--clients 10 --rounds 10 --local-epochs 3 \
|
||||
--aggregation-mode support_weighted --beta 0.5
|
||||
|
||||
# Pooled multi-seed validation analysis (read-only, no retraining; --alpha already supported)
|
||||
.venv/bin/python scripts/analyze_multiseed_beta.py --dataset xiiotid --alpha 0.25 --seeds 0 1 2
|
||||
# Pooled multi-seed validation analysis (read-only, no retraining)
|
||||
.venv/bin/python scripts/analyze_multiseed_beta.py --dataset xiiotid --alpha 0.75 --seeds 0 1 2
|
||||
|
||||
# Direct prototype-effect analysis (no retraining)
|
||||
.venv/bin/python scripts/analyze_prototype_effect.py \
|
||||
--snapshot artifacts/analysis/xiiotid__alpha0.25__seed0__support_weighted__beta0__prototype_snapshot.json \
|
||||
--snapshot artifacts/analysis/xiiotid__alpha0.75__seed0__support_weighted__beta0__prototype_snapshot.json \
|
||||
--class-names "C&C,Exfiltration,Exploitation,Lateral_movement,Normal,RDOS,Reconnaissance,Tampering,Weaponization" \
|
||||
--output-dir artifacts/analysis
|
||||
|
||||
# Realized partition characterization (all alphas)
|
||||
.venv/bin/python scripts/analyze_partition_heterogeneity.py --dataset xiiotid --alphas 0.75 0.5 0.25
|
||||
|
||||
# beta=0 equivalence audit
|
||||
.venv/bin/python scripts/verify_beta0_equivalence.py --dataset xiiotid --alpha 0.75 --seeds 0 1 2
|
||||
|
||||
# Cross-alpha comparison
|
||||
.venv/bin/python scripts/analyze_cross_alpha.py --dataset xiiotid --alphas 0.25 0.5 0.75
|
||||
```
|
||||
|
||||
## 19. Recommended next-session prompt
|
||||
## 20. Recommended next-session prompt
|
||||
|
||||
```
|
||||
You are continuing the perdalam-protean research repository (Session 8).
|
||||
You are continuing the perdalam-protean research repository (Session 9).
|
||||
|
||||
FIRST, read completely:
|
||||
1. docs/handoff/SESSION_HANDOFF.md
|
||||
2. docs/research/ALPHA025_BETA_VALIDATION.md
|
||||
3. docs/research/MULTISEED_BETA_VALIDATION.md
|
||||
4. docs/research/EXPERIMENT_PROTOCOL.md
|
||||
5. docs/baseline/BASELINE_VALIDATION.md
|
||||
2. docs/research/ALPHA075_BETA_VALIDATION.md
|
||||
3. docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md
|
||||
4. docs/research/SESSION8_AUDIT.md
|
||||
5. docs/research/EXPERIMENT_PROTOCOL.md
|
||||
6. docs/baseline/BASELINE_VALIDATION.md
|
||||
|
||||
Critical rules:
|
||||
- planningawal/ is immutable.
|
||||
@@ -383,19 +405,18 @@ Critical rules:
|
||||
- The S_j participant rule is a REPRODUCTION ASSUMPTION (material in 19/27 cells at alpha=0.25).
|
||||
- Do NOT add an executable equal_available mode.
|
||||
- Beta selection uses validation only; PRIMARY is pooled across clients and seeds.
|
||||
- Do NOT launch 5G-NIDD or the full grid.
|
||||
- Do NOT launch the full grid blindly; verify manifests and pace one condition at a time.
|
||||
|
||||
Context: Session 7 completed X-IIoTID alpha=0.25 for seeds {0,1,2} (18 runs, ~12.17 h) and was
|
||||
independently audited (PASS WITH MINOR CORRECTIONS; docs/research/SESSION7_AUDIT.md). Pooled
|
||||
validation selected beta=0.00 (pooled mean 0.591704; monotone decreasing in beta), unlike
|
||||
alpha=0.50 which selected 0.75. Per-seed diagnostics 0.75/0.50/0.00. beta=0 equivalence class B
|
||||
(no semantic discrepancy). Realized partitions more heterogeneous (19/27 |S_j|<M vs 7/27).
|
||||
Condition-level pooled labels: alpha=0.50 -> 0.75, alpha=0.25 -> 0.00 (the value 0.50 is only the
|
||||
Session-5 seed-0 choice, not a condition-level selection).
|
||||
Context: Session 8 completed X-IIoTID alpha=0.75 for seeds {0,1,2} (18 runs, ~12.33 h) and was
|
||||
independently audited (PASS; docs/research/SESSION8_AUDIT.md). Pooled validation selected
|
||||
beta=0.50. Three-alpha X-IIoTID pooled selections: alpha=0.25 -> 0.00, alpha=0.50 -> 0.75,
|
||||
alpha=0.75 -> 0.50 (NOT monotone). beta=0 equivalence class B. Realized heterogeneity: 19/27,
|
||||
7/27, 3/27 cells |S_j|<M; mean displacement 3.42 / 2.10 / 1.31.
|
||||
|
||||
Task: run X-IIoTID alpha=0.75 for seeds {0,1,2} across the five frozen betas plus
|
||||
original_protean, sequential, reusing frozen manifests, pooled validation-only selection, then
|
||||
test evaluation, direct-effect, client-level, and cross-alpha comparison. Produce
|
||||
docs/research/ALPHA075_BETA_VALIDATION.md and update the handoff. Use .venv/bin/python. Do not
|
||||
commit or push unless asked.
|
||||
Task: X-IIoTID three-alpha study is complete. Recommended next phase is 5G-NIDD under the same
|
||||
frozen protocol (alphas {0.75, 0.50, 0.25}, seeds {0,1,2}, five betas + original_protean,
|
||||
frozen manifests, pooled validation-only selection). Before scaling, verify 5G-NIDD manifests
|
||||
exist and reproduce, run one paired condition, inspect it, then proceed. Produce a 5G-NIDD beta
|
||||
validation report and a cross-dataset comparison, update the handoff. Use .venv/bin/python. Do
|
||||
not commit or push unless asked.
|
||||
```
|
||||
@@ -0,0 +1,422 @@
|
||||
# Alpha-0.75 Beta Validation (Session 8)
|
||||
|
||||
Status: **three-seed paired protocol completed for the mildest non-IID condition**
|
||||
(X-IIoTID `alpha = 0.75`, seeds `{0,1,2}`) with pooled-across-seed validation-only beta
|
||||
selection, followed by a single final test evaluation of the pooled-selected beta. This is a
|
||||
**paired characterisation of the third and final non-IID severity** planned for X-IIoTID. It
|
||||
is not a sweep, not a hyper-parameter search over seeds, and not evidence of a global optimum.
|
||||
|
||||
Companion documents:
|
||||
|
||||
- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen protocol and selection hierarchy.
|
||||
- `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md` — three-condition (0.25/0.50/0.75) comparison.
|
||||
- `docs/research/MULTISEED_BETA_VALIDATION.md` — Session-6 alpha=0.50 condition.
|
||||
- `docs/research/ALPHA025_BETA_VALIDATION.md` — Session-7 alpha=0.25 condition.
|
||||
- `docs/research/BETA_AGGREGATION_IMPLEMENTATION.md` — beta mode implementation.
|
||||
- `docs/baseline/BASELINE_VALIDATION.md` — frozen baseline evidence.
|
||||
- `docs/handoff/SESSION_HANDOFF.md` — continuation point.
|
||||
|
||||
---
|
||||
|
||||
## A. Scope
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| dataset | `xiiotid` (X-IIoTID) |
|
||||
| alpha | 0.75 |
|
||||
| seeds | 0, 1, 2 |
|
||||
| aggregation | `original_protean` (per seed) + `support_weighted` for `beta ∈ {0, 0.25, 0.5, 0.75, 1}` |
|
||||
| runs | 18: 15 `support_weighted` + 3 `original_protean`, all new this session |
|
||||
| compute | sequential on CPU, one run at a time, ≈ 12.33 h wall-clock total |
|
||||
|
||||
No other alpha, dataset, seed, mode or beta was run. `equal_available` was not added.
|
||||
5G-NIDD was deliberately **not** run.
|
||||
|
||||
---
|
||||
|
||||
## B. Frozen protocol
|
||||
|
||||
Identical to `EXPERIMENT_PROTOCOL.md` §14 and to Sessions 6–7: 10 clients, 10 rounds, 3 local
|
||||
epochs, batch 256, Adam `lr=1e-3`, `mu=0.1`, `lambda=1.0`; unchanged architecture,
|
||||
preprocessing, train/validation/test split recipe, partition recipe, loss terms and inference.
|
||||
All 18 runs were performed this session with `scripts/run_baseline.py` at HEAD `ff68d380`
|
||||
(no training/aggregation source changed).
|
||||
|
||||
### B.1 Manifest identity (reused, never regenerated)
|
||||
|
||||
Train/validation/test sizes are 569,909 / 81,414 / 162,831 for every seed (10 clients, 9
|
||||
classes, `Normal` benign).
|
||||
|
||||
| seed | manifest | content hash | partition hash | split hash | rng seed |
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | `artifacts/partitions/xiiotid__alpha0.75__seed0.json` | `759f7e50…` | `473b1c4a…` | `d0acc0c1…` | 957653162 |
|
||||
| 1 | `artifacts/partitions/xiiotid__alpha0.75__seed1.json` | `4a50ba21…` | `a0ec9e9c…` | `880a9f82…` | 762050403 |
|
||||
| 2 | `artifacts/partitions/xiiotid__alpha0.75__seed2.json` | `922b7bb1…` | `00345255…` | `8e0db0be…` | 843543396 |
|
||||
|
||||
The split hash is identical to the corresponding seeds at alpha=0.25/0.50 (splits are
|
||||
per-dataset-seed, not per-alpha); the partition hash differs because the Dirichlet partition
|
||||
differs.
|
||||
|
||||
### B.2 Build-level pairing
|
||||
|
||||
Within each seed all six runs share one manifest content hash and a byte-identical round-0
|
||||
snapshot:
|
||||
|
||||
| seed | manifest content hash | round-0 snapshot SHA-256 (all 6 runs) |
|
||||
|---|---|---|
|
||||
| 0 | `759f7e504a7d442101238726334d121a746ea4a8ff366428c1c174e47f358723` | `8c477df6b3af645bf2db1c8005d86ea6258b283370ff3ef3e6e88d6f40095564` |
|
||||
| 1 | `4a50ba219de20af14f6ad73976111659fba2d5b25cd5eeb1a2f21b76de0d9303` | `41f0187c4f0a8e3db0a8ab7eff0aa50a8f3e4a64ded2448228bd133ea017869d` |
|
||||
| 2 | `922b7bb15169302be7dd4d4eb1644d077537872e1e91fe59b7525b7e3343cc1c` | `be2ad5fb5fcf87c49b2826936c8d5dae6cdc4ab25fe2677081acec32f8116648` |
|
||||
|
||||
`train_support_Nij.csv`, `characterization.csv` and `rare_unseen.csv` are byte-identical across
|
||||
the six runs of each seed; `manifest_reload_ok = true` and `smoke = false` everywhere. Pairing
|
||||
is established from content hashes, not filenames.
|
||||
|
||||
### B.3 Selection rule (unchanged)
|
||||
|
||||
- **PRIMARY:** pooled-across-seed validation per-client-mean rare-class macro-F1 (average each
|
||||
seed's per-client mean over seeds `{0,1,2}`), selected with validation only.
|
||||
- **SECONDARY (diagnostic):** per-seed validation selection, stability only.
|
||||
- **TEST:** read only after the pooled beta is fixed.
|
||||
|
||||
---
|
||||
|
||||
## C. Partition characterization (realized, before interpretation)
|
||||
|
||||
`M = 10` clients; `|S_j| < M` means fewer than 10 clients contribute a class prototype. All
|
||||
statistics are from the training split `N_ij`.
|
||||
|
||||
### Per-seed summary
|
||||
|
||||
| seed | `|S_j| < M` cells | missing client-class cells | client-size min/max (ratio) | union rare | union unseen | max unseen on a client |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0 | 2 / 9 | 3 | 24,236 / 100,573 (4.15) | 4 | 2 | 1 |
|
||||
| 1 | 1 / 9 | 1 | 21,623 / 109,690 (5.07) | 5 | 1 | 1 |
|
||||
| 2 | 0 / 9 | 0 | 24,224 / 115,896 (4.78) | 6 | 0 | 0 |
|
||||
| **total / mean** | **3 / 27** | **4** | ratio mean **4.67** | 4/5/6 | 2/1/0 | 1 |
|
||||
|
||||
Every client has exactly two rare classes in every seed. Union rare classes per seed:
|
||||
seed 0 = `C&C, Exfiltration, Exploitation, Tampering`; seed 1 adds `RDOS`;
|
||||
seed 2 adds `Lateral_movement` and `Weaponization`. Unseen unions: seed 0 =
|
||||
`{C&C, Exploitation}`, seed 1 = `{C&C}`, seed 2 = `{}`.
|
||||
|
||||
### Per-class `N_ij` inequality (training split)
|
||||
|
||||
`CV` is the population coefficient of variation of the positive supports within a class;
|
||||
`max/min` is over contributing clients only.
|
||||
|
||||
**Seed 0**
|
||||
|
||||
| class | total | #clients | min+ | max | mean+ | CV | max/min |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| C&C | 2,004 | 8 | 5 | 760 | 250.5 | 0.888 | 152.0 |
|
||||
| Exfiltration | 15,494 | 10 | 27 | 4,392 | 1,549.4 | 0.846 | 162.7 |
|
||||
| Exploitation | 793 | 9 | 1 | 240 | 88.1 | 0.875 | 240.0 |
|
||||
| Lateral_movement | 22,117 | 10 | 190 | 5,435 | 2,211.7 | 0.888 | 28.6 |
|
||||
| Normal | 292,777 | 10 | 1,151 | 92,460 | 29,277.7 | 0.912 | 80.3 |
|
||||
| RDOS | 98,883 | 10 | 347 | 22,821 | 9,888.3 | 0.770 | 65.8 |
|
||||
| Reconnaissance | 89,161 | 10 | 458 | 23,824 | 8,916.1 | 0.982 | 52.0 |
|
||||
| Tampering | 3,586 | 10 | 31 | 1,227 | 358.6 | 1.176 | 39.6 |
|
||||
| Weaponization | 45,094 | 10 | 285 | 12,915 | 4,509.4 | 0.906 | 45.3 |
|
||||
|
||||
**Seed 1**
|
||||
|
||||
| class | total | #clients | min+ | max | mean+ | CV | max/min |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| C&C | 2,004 | 9 | 87 | 538 | 222.7 | 0.612 | 6.2 |
|
||||
| Exfiltration | 15,494 | 10 | 239 | 3,749 | 1,549.4 | 0.879 | 15.7 |
|
||||
| Exploitation | 793 | 10 | 4 | 246 | 79.3 | 0.997 | 61.5 |
|
||||
| Lateral_movement | 22,117 | 10 | 556 | 4,082 | 2,211.7 | 0.596 | 7.3 |
|
||||
| Normal | 292,777 | 10 | 4,311 | 73,956 | 29,277.7 | 0.818 | 17.2 |
|
||||
| RDOS | 98,883 | 10 | 268 | 34,051 | 9,888.3 | 0.936 | 127.1 |
|
||||
| Reconnaissance | 89,161 | 10 | 520 | 22,127 | 8,916.1 | 0.781 | 42.6 |
|
||||
| Tampering | 3,586 | 10 | 25 | 1,004 | 358.6 | 0.857 | 40.2 |
|
||||
| Weaponization | 45,094 | 10 | 269 | 10,726 | 4,509.4 | 0.796 | 39.9 |
|
||||
|
||||
**Seed 2**
|
||||
|
||||
| class | total | #clients | min+ | max | mean+ | CV | max/min |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| C&C | 2,004 | 10 | 1 | 546 | 200.4 | 0.917 | 546.0 |
|
||||
| Exfiltration | 15,494 | 10 | 9 | 3,572 | 1,549.4 | 0.860 | 396.9 |
|
||||
| Exploitation | 793 | 10 | 11 | 382 | 79.3 | 1.322 | 34.7 |
|
||||
| Lateral_movement | 22,117 | 10 | 3 | 5,709 | 2,211.7 | 0.879 | 1,903.0 |
|
||||
| Normal | 292,777 | 10 | 1,930 | 64,353 | 29,277.7 | 0.791 | 33.3 |
|
||||
| RDOS | 98,883 | 10 | 636 | 18,655 | 9,888.3 | 0.564 | 29.3 |
|
||||
| Reconnaissance | 89,161 | 10 | 1,528 | 21,418 | 8,916.1 | 0.692 | 14.0 |
|
||||
| Tampering | 3,586 | 10 | 1 | 1,321 | 358.6 | 1.112 | 1,321.0 |
|
||||
| Weaponization | 45,094 | 10 | 169 | 17,328 | 4,509.4 | 1.126 | 102.5 |
|
||||
|
||||
Machine-readable: `artifacts/analysis/xiiotid__cross_alpha__partition_heterogeneity_per_class.csv`,
|
||||
`..._per_seed.csv`, `..._summary.csv`.
|
||||
|
||||
### Comparison with the other conditions (realized)
|
||||
|
||||
| statistic | alpha=0.25 | alpha=0.50 | alpha=0.75 |
|
||||
|---|---|---|---|
|
||||
| class-condition cells with `|S_j| < M` | **19 / 27** | 7 / 27 | **3 / 27** |
|
||||
| missing client-class cells | **38** | 9 | **4** |
|
||||
| client-size ratio mean (seeds) | 14.75 (15.6/20.8/7.8) | 7.41 (5.1/7.2/9.9) | **4.67 (4.15/5.07/4.78)** |
|
||||
| union rare per seed | 8/7/8 | 6/7/7 | 4/5/6 |
|
||||
| union unseen per seed | 6/6/5 | 3/3/1 | 2/1/0 |
|
||||
| mean class-support CV | 1.442 | 1.183 | **0.881** |
|
||||
|
||||
The realized heterogeneity decreases in the expected direction as alpha rises, but **not every
|
||||
per-seed statistic is monotone** (e.g. alpha=0.75 seed 0 has client ratio 4.15 < seed 2's 4.78,
|
||||
and seed 2's max/min for `Lateral_movement` is 1,903). Interpretation uses the realized
|
||||
partitions, not alpha alone.
|
||||
|
||||
---
|
||||
|
||||
## D. Per-seed validation results
|
||||
|
||||
`val rareMacroF1 (pc mean)` is the primary metric; all 10 clients are eligible in every seed.
|
||||
Validation only. Machine-readable:
|
||||
`artifacts/analysis/xiiotid__alpha0.75__multiseed__validation_per_seed.csv`.
|
||||
|
||||
### Seed 0
|
||||
|
||||
| beta | eligible | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | 10 | 0.496803 | 0.532184 | 0.843412 | 0.747865 | 0.913983 |
|
||||
| 0.25 | 10 | 0.496892 | 0.525361 | 0.832611 | 0.747575 | 0.915125 |
|
||||
| **0.50** | 10 | **0.504388** | 0.519202 | 0.829968 | 0.745780 | 0.914327 |
|
||||
| 0.75 | 10 | 0.486868 | 0.505979 | 0.821332 | 0.738752 | 0.912276 |
|
||||
| 1.00 | 10 | 0.479146 | 0.509581 | 0.839570 | 0.742219 | 0.913479 |
|
||||
|
||||
### Seed 1
|
||||
|
||||
| beta | eligible | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | 10 | 0.408276 | 0.616818 | 0.885404 | 0.747924 | 0.913234 |
|
||||
| 0.25 | 10 | 0.421386 | 0.628514 | 0.883839 | 0.762087 | 0.929816 |
|
||||
| **0.50** | 10 | **0.422919** | 0.628751 | 0.880953 | 0.763561 | **0.931314** |
|
||||
| 0.75 | 10 | 0.421053 | 0.627215 | 0.877243 | 0.762790 | 0.930295 |
|
||||
| 1.00 | 10 | 0.419534 | 0.624894 | 0.870670 | 0.760871 | 0.928378 |
|
||||
|
||||
### Seed 2
|
||||
|
||||
| beta | eligible | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **0.00** | 10 | **0.510038** | 0.674455 | 0.882275 | 0.764538 | 0.933378 |
|
||||
| 0.25 | 10 | 0.504827 | 0.676745 | 0.881395 | 0.766571 | 0.935453 |
|
||||
| 0.50 | 10 | 0.502942 | **0.675491** | 0.876840 | 0.765947 | 0.935232 |
|
||||
| 0.75 | 10 | 0.509702 | 0.675571 | 0.872743 | **0.766351** | **0.935662** |
|
||||
| 1.00 | 10 | 0.507136 | 0.671867 | 0.863907 | 0.765122 | 0.935122 |
|
||||
|
||||
---
|
||||
|
||||
## E. Per-seed diagnostic beta choices (SECONDARY, stability only)
|
||||
|
||||
| seed | selected beta | val rareMacroF1 (pc mean) | tie-break used | test_used_for_selection |
|
||||
|---|---|---|---|---|
|
||||
| 0 | 0.50 | 0.504388 | no | false |
|
||||
| 1 | 0.50 | 0.422919 | no | false |
|
||||
| 2 | 0.00 | 0.510038 | no | false |
|
||||
|
||||
The per-seed preferred beta is more stable than at the other two alphas (two seeds pick 0.50,
|
||||
one picks 0.00). These per-seed choices are **diagnostic only** and are not the condition-level
|
||||
selection rule. `val_selection.json` records `"selection_split": "validation"` and
|
||||
`"test_used_for_selection": false` for all 15 `support_weighted` runs.
|
||||
|
||||
---
|
||||
|
||||
## F. PRIMARY pooled-across-seed validation selection
|
||||
|
||||
| beta | pooled mean | std (sample, ddof=1) | std (pop) | min | max | pooled val rareRecall (pc mean) | selected |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 0.00 | 0.471706 | 0.055329 | 0.045176 | 0.408276 | 0.510038 | 0.787289 | |
|
||||
| 0.25 | 0.474368 | 0.046055 | 0.037604 | 0.421386 | 0.504827 | 0.784275 | |
|
||||
| **0.50** | **0.476749** | 0.046624 | 0.038069 | 0.422919 | 0.504388 | 0.786531 | **yes** |
|
||||
| 0.75 | 0.472541 | 0.046028 | 0.037582 | 0.421053 | 0.509702 | 0.780729 | |
|
||||
| 1.00 | 0.468605 | 0.044742 | 0.036532 | 0.419534 | 0.507136 | 0.781309 | |
|
||||
|
||||
- **PRIMARY selected beta (`beta_star_alpha075`) = 0.50.**
|
||||
- **Criterion:** maximum pooled validation per-client-mean rare-class macro-F1.
|
||||
- **Tie-break used:** no (the top two pooled means differ by ≈ 2.4e-3).
|
||||
- **`test_used_for_selection = false`.** Test metrics were not inspected until the pooled beta
|
||||
was fixed.
|
||||
- **Interpretation:** `beta = 0.50` is the *best among the tested candidates under pooled
|
||||
validation across the three project-defined seeds for X-IIoTID alpha=0.75.* It is **not** a
|
||||
global optimum. The pooled validation curve peaks at 0.50 and declines on both sides, but the
|
||||
total between-beta spread (≈ 8.1e-3) is small relative to the between-seed std (≈ 0.045).
|
||||
|
||||
Machine-readable: `artifacts/analysis/xiiotid__alpha0.75__multiseed__validation_pooled.csv`
|
||||
and `...__selection_summary.json`.
|
||||
|
||||
---
|
||||
|
||||
## G. Final selected-beta test metrics (descriptive only)
|
||||
|
||||
`beta_star = 0.50` was fixed from pooled validation; only then were test metrics read.
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.511889 | 0.624144 | 0.679704 | 0.605246 ± 0.085489 |
|
||||
| rare-class recall | 0.803887 | 0.876726 | 0.865681 | 0.848765 ± 0.039256 |
|
||||
| macro-F1 | 0.741977 | 0.761752 | 0.769179 | 0.757636 ± 0.014060 |
|
||||
| accuracy | 0.914365 | 0.932611 | 0.937352 | 0.928110 ± 0.012137 |
|
||||
| weighted-F1 | 0.935566 | 0.948344 | 0.951740 | 0.945216 ± 0.008528 |
|
||||
|
||||
---
|
||||
|
||||
## H. `original_protean` comparison
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.521540 | 0.614061 | 0.676068 | 0.603890 ± 0.077764 |
|
||||
| rare-class recall | 0.816440 | 0.883238 | 0.869835 | 0.856504 ± 0.035338 |
|
||||
| macro-F1 | 0.742471 | 0.747395 | 0.765680 | 0.751849 ± 0.012228 |
|
||||
| accuracy | 0.913254 | 0.913898 | 0.934576 | 0.920576 ± 0.012129 |
|
||||
| weighted-F1 | 0.932563 | 0.935937 | 0.948908 | 0.939136 ± 0.008629 |
|
||||
|
||||
Descriptive difference (`beta_star` − `original_protean`) for test rare-class macro-F1 is
|
||||
−0.009651 (seed 0), +0.010083 (seed 1), +0.003636 (seed 2); mean +0.001356. Seed 0 moves the
|
||||
opposite way to seeds 1–2, and the magnitude is far smaller than the between-seed std
|
||||
(≈ 0.078–0.085). No superiority or generalisation is claimed; this is one dataset, one alpha,
|
||||
three project-defined seeds.
|
||||
|
||||
Test columns are descriptive only and are recorded in
|
||||
`artifacts/analysis/xiiotid__alpha0.75__multiseed__test_descriptive.csv`.
|
||||
|
||||
---
|
||||
|
||||
## I. `beta = 0` equivalence vs `original_protean`
|
||||
|
||||
| property | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| same manifest / split / hashes | yes | yes | yes |
|
||||
| round-0 snapshot byte-identical | yes (`8c477df6…`) | yes (`41f0187c…`) | yes (`be2ad5fb…`) |
|
||||
| support/characterization/rare_unseen identical | yes | yes | yes |
|
||||
| independent aggregation diff (`ΣC/n` vs `(1/n)ΣC`) | 4.4e-16 | 4.4e-16 | 2.2e-16 |
|
||||
| history rare-macro-F1 round-0 diff | 0.0 | 0.0 | 0.0 |
|
||||
| history rare-macro-F1 max diff (rounds ≥ 1) | 1.10e-2 | 8.12e-3 | 3.64e-3 |
|
||||
| test metric max diff | 4.33e-3 | 1.14e-3 | 0.85e-3 |
|
||||
| **classification** | **B** | **B** | **B** |
|
||||
|
||||
Class **B**: semantic equivalence + floating-point numerical divergence; **no semantic
|
||||
discrepancy** in any seed. Round 0 is byte-identical; the ~1 ULP ordering difference enters the
|
||||
round-1 prototype-alignment training and the trajectories diverge slowly. Equivalence remains
|
||||
conditional on the shared `S_j` reproduction assumption and is **not** proof that `beta = 0`
|
||||
reproduces original PROTEAN. Machine-readable:
|
||||
`artifacts/analysis/xiiotid__alpha0.75__multiseed__beta0_equivalence.csv`.
|
||||
|
||||
---
|
||||
|
||||
## J. Direct prototype-effect findings (fixed round-0 snapshots, no retraining)
|
||||
|
||||
Computed with the production aggregation from each seed's fixed `{C_ij, N_ij}` round-0
|
||||
snapshot. `||Cbar_j(beta) - Cbar_j(0)||` at `beta = 1.0`, cross-seed mean (sample std):
|
||||
|
||||
| class | mean disp | std | mean rel disp |
|
||||
|---|---|---|---|
|
||||
| Tampering | 1.934 | 0.427 | 0.397 |
|
||||
| Weaponization | 1.472 | 0.170 | 0.235 |
|
||||
| Exfiltration | 1.448 | 0.086 | 0.207 |
|
||||
| Exploitation | 1.539 | 0.252 | 0.284 |
|
||||
| Lateral_movement | 1.369 | 0.553 | 0.262 |
|
||||
| C&C | 1.284 | 0.543 | 0.262 |
|
||||
| Reconnaissance | 0.983 | 0.412 | 0.171 |
|
||||
| Normal | 0.903 | 0.298 | 0.162 |
|
||||
| RDOS | 0.814 | 0.200 | 0.096 |
|
||||
|
||||
- Mean displacement across the nine classes is **1.305**, far smaller than at alpha=0.50 (2.099)
|
||||
and alpha=0.25 (3.422), consistent with the lower support imbalance at alpha=0.75.
|
||||
- The most support-imbalanced classes (`Tampering` CV ≈ 1.0–1.2, `Weaponization`,
|
||||
`Exfiltration`) move most; `RDOS` and `Normal` move least in relative terms.
|
||||
- Displacement is a **direct-aggregation measure only** and is not interpreted as improved or
|
||||
degraded detection. The end-to-end effect (§F/§G) is reported separately.
|
||||
|
||||
Outputs: `artifacts/analysis/xiiotid__alpha0.75__seed{0,1,2}__support_weighted__beta0__prototype_snapshot__prototype_effect.csv`
|
||||
and `...__prototype_weights.csv`; cross-seed aggregate
|
||||
`artifacts/analysis/xiiotid__alpha0.75__multiseed__prototype_effect_pooled.csv`.
|
||||
|
||||
---
|
||||
|
||||
## K. Client-level effect (validation, relative to beta = 0)
|
||||
|
||||
Full per-client table:
|
||||
`artifacts/analysis/xiiotid__alpha0.75__multiseed__client_validation_detail.csv`; summary:
|
||||
`...__client_validation_summary.csv`.
|
||||
|
||||
| seed | beta | improve | degrade | unchanged | mean Δ | min Δ | max Δ |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 0 | 0.25 | 5 | 5 | 0 | +0.0001 | −0.0192 | +0.0269 |
|
||||
| 0 | 0.50 | 6 | 4 | 0 | +0.0076 | −0.0361 | +0.0480 |
|
||||
| 0 | 0.75 | 3 | 7 | 0 | −0.0099 | −0.0433 | +0.0253 |
|
||||
| 0 | 1.00 | 1 | 9 | 0 | −0.0177 | −0.0650 | +0.0222 |
|
||||
| 1 | 0.25 | 9 | 0 | 1 | +0.0131 | 0.0000 | +0.0318 |
|
||||
| 1 | 0.50 | 9 | 0 | 1 | +0.0146 | 0.0000 | +0.0402 |
|
||||
| 1 | 0.75 | 9 | 0 | 1 | +0.0128 | 0.0000 | +0.0375 |
|
||||
| 1 | 1.00 | 7 | 2 | 1 | +0.0113 | −0.0116 | +0.0578 |
|
||||
| 2 | 0.25 | 7 | 3 | 0 | −0.0052 | −0.1212 | +0.0382 |
|
||||
| 2 | 0.50 | 6 | 4 | 0 | −0.0071 | −0.1524 | +0.0478 |
|
||||
| 2 | 0.75 | 5 | 5 | 0 | −0.0003 | −0.0860 | +0.0531 |
|
||||
| 2 | 1.00 | 6 | 4 | 0 | −0.0029 | −0.1212 | +0.0531 |
|
||||
|
||||
Descriptive observations:
|
||||
|
||||
- The response is seed-dependent. At the selected `beta = 0.50`, seed 1 improves on 9/10
|
||||
clients (mean +0.0146), seed 0 on 6/10 (mean +0.0076), and seed 2 on 6/10 but with a negative
|
||||
mean (−0.0071) driven by one client at −0.152.
|
||||
- The `beta = 0.50` deltas are much smaller in magnitude than at alpha=0.25 or alpha=0.50,
|
||||
reflecting the smaller direct displacement (§J).
|
||||
- Client ids are per-seed realisations and are not treated as one population. This analysis is
|
||||
descriptive and non-causal; `N_ij` vs ΔF1 is non-monotonic.
|
||||
|
||||
---
|
||||
|
||||
## L. Runtime and resources
|
||||
|
||||
CPU only, no GPU, no swap, exit status 0 for every run (`/usr/bin/time -v`).
|
||||
|
||||
| seed | tag | wall | CPU % | peak RSS |
|
||||
|---|---|---|---|---|
|
||||
| 0 | `original_protean` | 40:40.74 | 745 | 6.49 GiB |
|
||||
| 0 | beta 0.00 | 40:04.93 | 746 | 6.49 GiB |
|
||||
| 0 | beta 0.25 | 40:50.34 | 745 | 6.49 GiB |
|
||||
| 0 | beta 0.50 | 40:48.88 | 746 | 6.49 GiB |
|
||||
| 0 | beta 0.75 | 40:48.60 | 746 | 6.49 GiB |
|
||||
| 0 | beta 1.00 | 40:09.82 | 745 | 6.49 GiB |
|
||||
| 1 | `original_protean` | 40:25.70 | 745 | 6.91 GiB |
|
||||
| 1 | beta 0.00 | 40:00.25 | 745 | 6.91 GiB |
|
||||
| 1 | beta 0.25 | 40:10.91 | 745 | 6.91 GiB |
|
||||
| 1 | beta 0.50 | 40:16.31 | 745 | 6.91 GiB |
|
||||
| 1 | beta 0.75 | 40:18.58 | 744 | 6.91 GiB |
|
||||
| 1 | beta 1.00 | 49:01.62 | 714 | 6.91 GiB |
|
||||
| 2 | `original_protean` | 40:18.15 | 745 | 7.20 GiB |
|
||||
| 2 | beta 0.00 | 40:31.08 | 745 | 7.20 GiB |
|
||||
| 2 | beta 0.25 | 40:18.53 | 745 | 7.20 GiB |
|
||||
| 2 | beta 0.50 | 43:17.28 | 732 | 7.20 GiB |
|
||||
| 2 | beta 0.75 | 41:01.74 | 744 | 7.20 GiB |
|
||||
| 2 | beta 1.00 | 40:27.57 | 745 | 7.20 GiB |
|
||||
|
||||
- New compute this session: **18 runs, 44,371 s ≈ 12.33 h wall-clock**, sequential on an
|
||||
8-core host.
|
||||
- Peak RSS is partition-dependent, not beta-dependent: 6.49 GiB (seed 0), 6.91 GiB (seed 1),
|
||||
7.20 GiB (seed 2); all well within the 29 GiB host. Two runs were slightly slower
|
||||
(seed 1 beta 1.00, seed 2 beta 0.50) due to transient host contention; both exited 0.
|
||||
- No warnings, no client failures, 9/9 global prototypes in every round, no retries.
|
||||
- Machine-readable: `artifacts/analysis/xiiotid__alpha0.75__multiseed__runtime.csv`.
|
||||
|
||||
---
|
||||
|
||||
## M. Limitations
|
||||
|
||||
- **One dataset** (X-IIoTID). No cross-dataset conclusion.
|
||||
- **Three project-defined seeds** (`{0,1,2}`); the pooled mean's standard error is of the same
|
||||
order as the between-beta spread, so the selected beta is statistically weak.
|
||||
- The `S_j` participant/denominator rule
|
||||
(`S_j = {i | N_ij > 0 and valid C_ij}`, equal weights in the baseline) remains a
|
||||
**REPRODUCTION ASSUMPTION**. At alpha=0.75 it is material in only **3/27** class-condition
|
||||
cells, but the `beta = 0` comparison is still conditional on both modes sharing this rule and
|
||||
does **not** prove `beta = 0` reproduces original PROTEAN.
|
||||
- Train/validation/test adaptation (70/10/20 stratified) differs from the paper's literal
|
||||
80/20; the validation eval partition is a project decision.
|
||||
- **No 5G-NIDD beta result.**
|
||||
- **No global beta optimum claim.** `beta = 0.50` is only the best among the five tested
|
||||
candidates under pooled validation for X-IIoTID alpha=0.75 across seeds `{0,1,2}`.
|
||||
- The union of rare classes (and hence global rare support) differs by seed and by alpha;
|
||||
only the per-client-mean metric is directly comparable within a condition.
|
||||
- Test metrics of the selected beta are single-condition and must not be read as
|
||||
generalisation; no test metric influenced selection.
|
||||
@@ -0,0 +1,268 @@
|
||||
# Session 8 Post-Experiment Audit (X-IIoTID alpha=0.75)
|
||||
|
||||
Status: **PASS.** Independent recomputation from machine-readable artifacts confirms every
|
||||
Session 8 claim. No discovered issue changes `beta_star` or the scientific interpretation. No
|
||||
corrections were required. **No training was run, no aggregation/model source changed, no
|
||||
test-based selection occurred.**
|
||||
|
||||
Audit performed: 2026-09-26. Audited HEAD: `ff68d380e23859d21c07c8a3a7b56b1b7f06c91b`.
|
||||
|
||||
Companion documents: `docs/research/ALPHA075_BETA_VALIDATION.md`,
|
||||
`docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md`, `docs/research/EXPERIMENT_PROTOCOL.md`,
|
||||
`docs/handoff/SESSION_HANDOFF.md`.
|
||||
|
||||
---
|
||||
|
||||
## A. Scope
|
||||
|
||||
Independently verify the Session 8 claim for:
|
||||
|
||||
- dataset `xiiotid`, alpha `0.75`, seeds `{0,1,2}`, beta `{0, 0.25, 0.5, 0.75, 1.0}` plus
|
||||
`original_protean` (18 runs);
|
||||
- PRIMARY pooled-validation selection `beta_star = 0.50`, using validation per-client
|
||||
rare-class macro-F1 averaged within seed then across seeds;
|
||||
- cross-alpha labels (alpha=0.25 → 0.00, alpha=0.50 → 0.75, alpha=0.75 → 0.50).
|
||||
|
||||
The audit recomputed from run artifacts and re-derived selection; it repeated no training and
|
||||
ran no new condition.
|
||||
|
||||
---
|
||||
|
||||
## B. Run inventory
|
||||
|
||||
18 full-data runs enumerated from `results/*/metadata.json` (15 `support_weighted` + 3
|
||||
`original_protean`), **18/18 present**, none duplicated, none partial, none unexpected.
|
||||
|
||||
- Metadata verified per run: dataset=xiiotid, alpha=0.75, seed, aggregation mode, beta,
|
||||
clients=10, rounds=10, local_epochs=3, batch=256, lr=1e-3, mu=0.1, lambda=1.0,
|
||||
`manifest_reload_ok=true`, `smoke=false`.
|
||||
- All 14 expected artifacts present for every run; the round-0 snapshot exists for every run.
|
||||
- No run exists for any alpha ≠ 0.75 in this session; **no 5G-NIDD `support_weighted` run**
|
||||
exists; `equal_available` absent from `src/`, `scripts/`, `tests/`.
|
||||
|
||||
---
|
||||
|
||||
## C. Pairing / manifest integrity
|
||||
|
||||
Within each seed, all six runs share one manifest content hash and a byte-identical round-0
|
||||
snapshot (SHA-256):
|
||||
|
||||
| seed | manifest content hash | round-0 snapshot SHA-256 | support/char/rare_unseen |
|
||||
|---|---|---|---|
|
||||
| 0 | `759f7e504a7d442101238726334d121a746ea4a8ff366428c1c174e47f358723` | `8c477df6b3af645bf2db1c8005d86ea6258b283370ff3ef3e6e88d6f40095564` | identical |
|
||||
| 1 | `4a50ba219de20af14f6ad73976111659fba2d5b25cd5eeb1a2f21b76de0d9303` | `41f0187c4f0a8e3db0a8ab7eff0aa50a8f3e4a64ded2448228bd133ea017869d` | identical |
|
||||
| 2 | `922b7bb15169302be7dd4d4eb1644d077537872e1e91fe59b7525b7e3343cc1c` | `be2ad5fb5fcf87c49b2826936c8d5dae6cdc4ab25fe2677081acec32f8116648` | identical |
|
||||
|
||||
Each manifest was independently reproduced from the raw dataset (`prepare_dataset` →
|
||||
`reproduce_train_assignments`): all three seeds return exact `N_ij` match, partition-hash match,
|
||||
split sizes 569,909 / 81,414 / 162,831, `N_ij` sum 569,909, 10 clients. The split hash is
|
||||
identical across alpha for a given seed (splits are per dataset-seed); the partition hash
|
||||
differs, as it must. Pairing is proven by content hashes, not filenames.
|
||||
|
||||
---
|
||||
|
||||
## D. Partition characterization audit
|
||||
|
||||
Independently recomputed from the manifests. Matches the report exactly:
|
||||
|
||||
| statistic | alpha=0.75 seed 0 / 1 / 2 | reality (sum) |
|
||||
|---|---|---|
|
||||
| `|S_j| < M` cells | 2 / 1 / 0 | **3 / 27** |
|
||||
| missing client-class cells | 3 / 1 / 0 | **4** |
|
||||
| client-size ratio | 4.15 / 5.07 / 4.78 | mean **4.67** |
|
||||
| union unseen classes | 2 / 1 / 0 | max 1 on a client |
|
||||
| union rare classes | 4 / 5 / 6 | 2 rare per client (all) |
|
||||
|
||||
Cross-alpha comparison confirmed: alpha=0.25 = 19/27 cells / 38 missing; alpha=0.50 = 7/27 / 9.
|
||||
Interpretation uses realized statistics, not alpha alone.
|
||||
|
||||
---
|
||||
|
||||
## E. Validation metric recomputation
|
||||
|
||||
Recomputed from raw artifacts (no report numbers copied):
|
||||
|
||||
- seed-level primary score = mean over eligible clients of `val_per_client.csv`
|
||||
`rare_class_macro_f1`; equals `val_selection.json` per-client-mean to **< 1e-12** for all 15
|
||||
runs;
|
||||
- global validation accuracy / macro-F1 / weighted-F1 / rare macro-F1 / rare recall recomputed
|
||||
independently from `val_confusion_matrix.json` with the production formulas: **max diff 0.0**
|
||||
versus `val_global_metrics.json`.
|
||||
|
||||
All 10 clients are eligible in every seed/beta. Per-seed diagnostics reproduce **0.50 / 0.50 /
|
||||
0.00**.
|
||||
|
||||
---
|
||||
|
||||
## F. Rare / unseen integrity
|
||||
|
||||
Per-client rare classes recomputed from training `N_ij` (two smallest positive-support attack
|
||||
classes, tie by class id) and compared to `rare_unseen.csv` flags/N_ij across all 18 runs:
|
||||
|
||||
- **1,620 cells checked, 0 mismatches.**
|
||||
- Benign class (`Normal`) never flagged rare/unseen; rare and unseen never overlap.
|
||||
- Union training-derived rare classes equal the run's `evaluated_rare_classes`; validation/test
|
||||
labels did not influence rare identity.
|
||||
- Every client has exactly two rare classes.
|
||||
|
||||
---
|
||||
|
||||
## G. Pooled-selection recomputation
|
||||
|
||||
Computed from the independently recomputed seed-level scores (not from the report). Pooled =
|
||||
mean of the three seed means; std sample (`ddof=1`).
|
||||
|
||||
| beta | pooled mean | sample std | min | max |
|
||||
|---|---|---|---|---|
|
||||
| 0.00 | 0.471706 | 0.055329 | 0.408276 | 0.510038 |
|
||||
| 0.25 | 0.474368 | 0.046055 | 0.421386 | 0.504827 |
|
||||
| **0.50** | **0.476749** | 0.046624 | 0.422919 | 0.504388 |
|
||||
| 0.75 | 0.472541 | 0.046028 | 0.421053 | 0.509702 |
|
||||
| 1.00 | 0.468605 | 0.044742 | 0.419534 | 0.507136 |
|
||||
|
||||
- Recompute **matches the report** to formatting precision.
|
||||
- `argmax` = **0.50** (no tie; top-two gap ≈ 2.4e-3). `tie_break_used=false`.
|
||||
|
||||
---
|
||||
|
||||
## H. Validation / test leakage audit
|
||||
|
||||
- `run_baseline.py` derives `union_rare` from the training support matrix only; the
|
||||
`val_selection` payload is built only from validation artifacts, and `test_used_for_selection`
|
||||
/ `selection_split` are recorded for all 15 `support_weighted` runs.
|
||||
- `analyze_multiseed_beta.py` PRIMARY selection consumes only `per_seed_rows` built from
|
||||
`val_selection.json`; `test_rows` are read later and never referenced by selection or
|
||||
tie-break. No code path sorts/filters candidates by a test field.
|
||||
- The audit's own `beta_star = 0.50` was reproduced from validation artifacts alone.
|
||||
|
||||
Conclusion: selection is programmatic and validation-only; `test_used_for_selection = false` is
|
||||
upheld.
|
||||
|
||||
---
|
||||
|
||||
## I. Final test recomputation
|
||||
|
||||
`beta_star = 0.50` fixed from pooled validation; only then test metrics read. Recomputed
|
||||
independently from `test_confusion_matrix.json` (diff vs JSON = 0.0), then averaged across seeds
|
||||
with sample std:
|
||||
|
||||
| metric | selected beta 0.50 (mean ± std) | original_protean (mean ± std) |
|
||||
|---|---|---|
|
||||
| rare-class macro-F1 | 0.605246 ± 0.085489 | 0.603890 ± 0.077764 |
|
||||
| rare-class recall | 0.848765 ± 0.039256 | 0.856504 ± 0.035338 |
|
||||
| macro-F1 | 0.757636 ± 0.014060 | 0.751849 ± 0.012228 |
|
||||
| accuracy | 0.928110 ± 0.012137 | 0.920576 ± 0.012129 |
|
||||
| weighted-F1 | 0.945216 ± 0.008528 | 0.939136 ± 0.008629 |
|
||||
|
||||
Matches the report exactly. No superiority claim is made.
|
||||
|
||||
---
|
||||
|
||||
## J. beta = 0 equivalence audit
|
||||
|
||||
For seeds 0, 1, 2, `original_protean` vs `support_weighted beta=0`:
|
||||
|
||||
| property | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| same manifest / split / hashes | yes | yes | yes |
|
||||
| round-0 snapshot byte-identical | yes | yes | yes |
|
||||
| support/characterization/rare_unseen identical | yes | yes | yes |
|
||||
| independent aggregation diff (`ΣC/n` vs `(1/n)ΣC`) | 4.4e-16 | 4.4e-16 | 2.2e-16 |
|
||||
| history rare-macro-F1 round-0 diff | 0.0 | 0.0 | 0.0 |
|
||||
| history rare-macro-F1 max diff (rounds ≥ 1) | 1.10e-2 | 8.12e-3 | 3.64e-3 |
|
||||
| test metric max diff | 4.33e-3 | 1.14e-3 | 0.85e-3 |
|
||||
| classification | **B** | **B** | **B** |
|
||||
|
||||
Class **B** (semantically equivalent + floating-point divergence); **no semantic discrepancy**.
|
||||
Equivalence remains conditional on the shared `S_j` reproduction assumption.
|
||||
|
||||
---
|
||||
|
||||
## K. Direct-effect audit
|
||||
|
||||
Independently recomputed `||Cbar_j(beta) − Cbar_j(0)||` from the fixed round-0 snapshot with a
|
||||
hand-rolled NumPy aggregation: max absolute difference vs the per-seed
|
||||
`...__prototype_effect.csv` = **8.9e-16** (seed 0 check across all classes/betas). Cross-seed
|
||||
mean displacement at `beta=1`: **1.305** (alpha=0.75) vs 2.099 (alpha=0.50) and 3.422
|
||||
(alpha=0.25). Displacement is a direct-aggregation measure only; no detection causality is
|
||||
inferred.
|
||||
|
||||
---
|
||||
|
||||
## L. Client-level audit
|
||||
|
||||
Deltas recomputed from `val_per_client.csv` vs the `client_validation_detail.csv` deltas:
|
||||
**max absolute difference 0.0** across all seeds/betas/clients. At the selected
|
||||
`beta=0.50`: 6/4/0 (seed 0, mean +0.008), 9/0/1 (seed 1, +0.015), 6/4/0 (seed 2, −0.007).
|
||||
`N_ij` vs ΔF1 is non-monotonic. Client ids are per-seed realisations, not pooled.
|
||||
|
||||
---
|
||||
|
||||
## M. Cross-alpha audit
|
||||
|
||||
Condition-level **pooled** selections confirmed from `cross_alpha_summary.json`:
|
||||
|
||||
| condition | selected beta | tie-break | test_used_for_selection |
|
||||
|---|---|---|---|
|
||||
| alpha=0.25 | 0.00 | no | false |
|
||||
| alpha=0.50 | 0.75 | no | false |
|
||||
| alpha=0.75 | 0.50 | no | false |
|
||||
|
||||
The sequence 0.00 → 0.75 → 0.50 is **not monotone**. No document in the repository states a
|
||||
monotone alpha→beta trend. The report explicitly refuses a monotonic interpretation.
|
||||
|
||||
---
|
||||
|
||||
## N. Artifact provenance
|
||||
|
||||
| Reported quantity | Source artifact(s) | Derivation |
|
||||
|---|---|---|
|
||||
| run existence / config | `results/<run>/metadata.json` | direct read |
|
||||
| manifest hashes / reproduction | `artifacts/partitions/xiiotid__alpha0.75__seed{0,1,2}.json` | module + `reproduce_train_assignments` |
|
||||
| pairing proof | `artifacts/analysis/...__prototype_snapshot.json` | sha256 across 6 runs/seed |
|
||||
| seed-level validation score | `results/.../val_per_client.csv` + `val_selection.json` | mean of per-client rare macro-F1 |
|
||||
| validation global metrics | `results/.../val_confusion_matrix.json` | independent numpy recompute |
|
||||
| pooled mean/std, `beta_star` | recomputed seed scores | mean/`stdev`, argmax |
|
||||
| test metrics | `results/.../test_confusion_matrix.json` | independent recompute + mean/std |
|
||||
| beta=0 equivalence | original vs beta0 artifacts/snapshots | sha256 + independent aggregation + history diff |
|
||||
| direct-effect displacement | snapshots + `...__prototype_effect_pooled.csv` | independent numpy recompute |
|
||||
| client improve/degrade | `results/.../val_per_client.csv` | deltas vs beta=0 |
|
||||
| rare-class identity | `results/.../rare_unseen.csv` + manifest | recompute definition |
|
||||
| partition characterization | manifests + heterogeneity CSVs | independent numpy recompute |
|
||||
| runtime / RSS | `artifacts/analysis/xiiotid__alpha0.75__multiseed__runtime.csv` (source `/tmp/opencode/s8_alpha075/logs/*.time`, present) | parse `/usr/bin/time -v`; sum 44,371 s = 12.33 h |
|
||||
| cross-alpha table | `artifacts/analysis/xiiotid__cross_alpha__*` | read per-condition multiseed artifacts |
|
||||
|
||||
---
|
||||
|
||||
## O. Deviations / corrections found
|
||||
|
||||
**None.** No arithmetic, metric-label, std-convention, provenance or pairing defect was found in
|
||||
the Session 8 report or analysis artifacts. The three new read-only analysis scripts
|
||||
(`analyze_partition_heterogeneity.py`, `verify_beta0_equivalence.py`, `analyze_cross_alpha.py`)
|
||||
do not touch training or aggregation source.
|
||||
|
||||
Minor note (informational, not a defect): two runs were slower than 41 min (seed 1 beta 1.00 at
|
||||
49:02, seed 2 beta 0.50 at 43:17) due to transient host contention; all 18 exited 0.
|
||||
|
||||
---
|
||||
|
||||
## P. Final audit verdict
|
||||
|
||||
**PASS.**
|
||||
|
||||
- Run inventory: **PASS** (18/18; no duplicates/partials; no alpha≠0.75 or 5G-NIDD run).
|
||||
- Pairing / manifest reproduction: **PASS** (hash-identical; raw-data reproduction exact).
|
||||
- Partition characterization: **PASS** (3/27, 4, 4.15/5.07/4.78, unseen 2/1/0).
|
||||
- Validation recomputation (seed-level and global): **PASS** (exact).
|
||||
- Rare/unseen integrity: **PASS** (0 mismatches over 1,620 cells).
|
||||
- Pooled selection: **PASS**; independent `beta_star = 0.50` using validation only.
|
||||
- Validation/test leakage: **PASS**.
|
||||
- Test summaries: **PASS** (exact).
|
||||
- beta=0 equivalence: **PASS**, class **B** for all seeds.
|
||||
- Direct-effect and client-level analyses: **PASS** (independent recompute exact).
|
||||
- Cross-alpha labels: **PASS** (0.00 / 0.75 / 0.50; not monotone; no monotone claim).
|
||||
- Source/data integrity: **PASS**; pytest 75 passed.
|
||||
|
||||
Session 8 is **safe to checkpoint** with the documented limitations (single dataset, three
|
||||
alphas, three project-defined seeds, weak effect relative to seed variance, `S_j` reproduction
|
||||
assumption, no global-optimum claim).
|
||||
@@ -0,0 +1,223 @@
|
||||
# X-IIoTID Cross-Alpha Analysis (alpha = 0.25 / 0.50 / 0.75)
|
||||
|
||||
Status: **descriptive three-condition comparison** of the frozen beta protocol on X-IIoTID,
|
||||
using the audited condition-level pooled results of Sessions 6 (`alpha=0.50`), 7
|
||||
(`alpha=0.25`) and 8 (`alpha=0.75`). This document does **not** run anything and does not
|
||||
make a causal or monotonicity claim. Machine-readable counterpart:
|
||||
`artifacts/analysis/xiiotid__cross_alpha__cross_alpha_beta_table.csv`, `..._summary.json`,
|
||||
`..._displacement.csv`, `..._client_summary.csv`, and
|
||||
`artifacts/analysis/xiiotid__cross_alpha__partition_heterogeneity_*.csv`.
|
||||
|
||||
Companion documents: `docs/research/ALPHA075_BETA_VALIDATION.md`,
|
||||
`docs/research/ALPHA025_BETA_VALIDATION.md`, `docs/research/MULTISEED_BETA_VALIDATION.md`,
|
||||
`docs/research/EXPERIMENT_PROTOCOL.md`, `docs/handoff/SESSION_HANDOFF.md`.
|
||||
|
||||
---
|
||||
|
||||
## A. Experimental scope
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| dataset | X-IIoTID (`xiiotid`) |
|
||||
| alpha | 0.25, 0.50, 0.75 |
|
||||
| seeds | 0, 1, 2 |
|
||||
| betas | 0.00, 0.25, 0.50, 0.75, 1.00 |
|
||||
| aggregation | `original_protean` + `support_weighted` |
|
||||
| selection | pooled validation-only per condition, seeds `{0,1,2}` |
|
||||
| test | read only after each condition's pooled beta is fixed |
|
||||
|
||||
All three conditions share the frozen training protocol: 10 clients, 10 rounds, 3 local epochs,
|
||||
batch 256, Adam `lr=1e-3`, `mu=0.1`, `lambda=1.0`, unchanged architecture/preprocessing/split
|
||||
recipe/loss/inference. For each `(alpha, seed)` all six runs share one manifest and a
|
||||
byte-identical round-0 snapshot.
|
||||
|
||||
---
|
||||
|
||||
## B. Realized heterogeneity
|
||||
|
||||
`|S_j| < M` counts class-condition cells with fewer than 10 contributing clients, summed over
|
||||
the three seeds (out of 27). Missing cells count `(client, class)` pairs with `N_ij = 0`.
|
||||
|
||||
| statistic | alpha=0.25 | alpha=0.50 | alpha=0.75 |
|
||||
|---|---|---|---|
|
||||
| `|S_j| < M` cells (sum over 3 seeds) | **19 / 27** | 7 / 27 | **3 / 27** |
|
||||
| missing client-class cells | **38** | 9 | **4** |
|
||||
| client-size ratio (seed 0/1/2) | 15.6 / 20.8 / 7.8 | 5.1 / 7.2 / 9.9 | 4.15 / 5.07 / 4.78 |
|
||||
| client-size ratio mean | 14.75 | 7.41 | **4.67** |
|
||||
| union rare classes per seed | 8 / 7 / 8 | 6 / 7 / 7 | 4 / 5 / 6 |
|
||||
| union unseen classes per seed | 6 / 6 / 5 | 3 / 3 / 1 | 2 / 1 / 0 |
|
||||
| max unseen classes on a client | 4 | 2 | 1 |
|
||||
| mean class-support CV | 1.442 | 1.183 | **0.881** |
|
||||
| max per-class max/min ratio (over seeds) | 46,541 | 6,204 | 1,903 |
|
||||
|
||||
Realized heterogeneity decreases as alpha rises, but **not uniformly per seed** (e.g. at
|
||||
alpha=0.75 seed 0 the client ratio 4.15 is lower than seed 2's 4.78; alpha=0.50 seed 2's ratio
|
||||
9.9 exceeds alpha=0.25 seed 2's 7.8). Interpretation uses these realized partitions, not alpha
|
||||
alone.
|
||||
|
||||
---
|
||||
|
||||
## C. Pooled validation curves (alpha × beta)
|
||||
|
||||
Pooled validation per-client-mean rare-class macro-F1 (selection metric; validation only):
|
||||
|
||||
| beta | alpha=0.25 mean (std) | alpha=0.50 mean (std) | alpha=0.75 mean (std) |
|
||||
|---|---|---|---|
|
||||
| 0.00 | **0.591704** (0.089529) | 0.544361 (0.091447) | 0.471706 (0.055329) |
|
||||
| 0.25 | 0.588691 (0.066435) | 0.555434 (0.090068) | 0.474368 (0.046055) |
|
||||
| 0.50 | 0.585851 (0.044858) | 0.567322 (0.088539) | **0.476749** (0.046624) |
|
||||
| 0.75 | 0.579463 (0.064906) | **0.579367** (0.075121) | 0.472541 (0.046028) |
|
||||
| 1.00 | 0.532099 (0.045718) | 0.573479 (0.100450) | 0.468605 (0.044742) |
|
||||
| **curve shape** | monotone decreasing | rises to peak at 0.75 | peaks at 0.50, declines both sides |
|
||||
| **total between-beta spread** | 0.0596 | 0.0350 | **0.0081** |
|
||||
|
||||
The across-seed sample std (0.045–0.100) is comparable to or larger than the between-beta
|
||||
spread at every alpha, so these selections are statistically weak descriptions, not strong
|
||||
preferences.
|
||||
|
||||
---
|
||||
|
||||
## D. Selected beta by condition
|
||||
|
||||
Condition-level **pooled** selections only (per-seed diagnostics are excluded):
|
||||
|
||||
| condition | pooled-selected beta | pooled validation mean | tie-break used |
|
||||
|---|---|---|---|
|
||||
| alpha=0.25 | **0.00** | 0.591704 | no |
|
||||
| alpha=0.50 | **0.75** | 0.579367 | no |
|
||||
| alpha=0.75 | **0.50** | 0.476749 | no |
|
||||
|
||||
The selected beta **changes with the non-IID condition but is not monotone** in alpha
|
||||
(0.00 → 0.75 → 0.50). The value 0.50 at alpha=0.50 is only the Session-5 seed-0 choice and the
|
||||
Session-6 per-seed diagnostic for seed 0; it is **not** the alpha=0.50 pooled selection.
|
||||
|
||||
---
|
||||
|
||||
## E. Final test summaries (descriptive)
|
||||
|
||||
Selected beta vs `original_protean`, mean ± sample std across seeds `{0,1,2}`.
|
||||
|
||||
| condition | selected beta | test rare macro-F1 (selected) | test rare macro-F1 (original) | test macro-F1 (selected) | test macro-F1 (original) | test acc (selected) | test acc (original) |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| alpha=0.25 | 0.00 | 0.632136 ± 0.047642 | 0.632790 ± 0.047152 | 0.681343 ± 0.020555 | 0.682018 ± 0.020967 | 0.874015 ± 0.010267 | 0.875431 ± 0.009067 |
|
||||
| alpha=0.50 | 0.75 | 0.665503 ± 0.026685 | 0.642520 ± 0.028683 | 0.731169 ± 0.006234 | 0.708965 ± 0.001172 | 0.906392 ± 0.018450 | 0.899454 ± 0.008398 |
|
||||
| alpha=0.75 | 0.50 | 0.605246 ± 0.085489 | 0.603890 ± 0.077764 | 0.757636 ± 0.014060 | 0.751849 ± 0.012228 | 0.928110 ± 0.012137 | 0.920576 ± 0.012129 |
|
||||
|
||||
**Do not compare the absolute test rare-class macro-F1 across conditions**: the training-derived
|
||||
union of rare classes (and hence the global rare evaluation support) differs by alpha and seed
|
||||
(e.g. alpha=0.75 test rare support 6,251 / 34,503 / 25,454). Only the selected-vs-original
|
||||
difference *within* a condition is meaningful, and even that is dominated by between-seed
|
||||
variance. No superiority or generalisation is claimed.
|
||||
|
||||
---
|
||||
|
||||
## F. Direct prototype displacement (beta = 1 vs beta = 0)
|
||||
|
||||
Cross-seed mean `||Cbar_j(1) − Cbar_j(0)||` on the fixed round-0 snapshots (no retraining):
|
||||
|
||||
| class | alpha=0.25 | alpha=0.50 | alpha=0.75 |
|
||||
|---|---|---|---|
|
||||
| Tampering | 4.248 | 2.354 | 1.934 |
|
||||
| Exfiltration | 4.316 | 2.020 | 1.448 |
|
||||
| Exploitation | 3.464 | 2.344 | 1.539 |
|
||||
| Lateral_movement | 3.726 | 2.724 | 1.369 |
|
||||
| C&C | 4.020 | 1.694 | 1.284 |
|
||||
| Weaponization | 3.124 | 2.561 | 1.472 |
|
||||
| Reconnaissance | 3.339 | 1.002 | 0.983 |
|
||||
| Normal | 1.997 | 1.557 | 0.903 |
|
||||
| RDOS | 2.566 | 2.631 | 0.814 |
|
||||
| **mean across classes** | **3.422** | **2.099** | **1.305** |
|
||||
|
||||
Mean displacement tracks realized support imbalance (alpha=0.25 > 0.50 > 0.75). The per-seed
|
||||
rank order of classes is seed-sensitive at every alpha. Displacement is a direct-aggregation
|
||||
measure only; it is **not** interpreted as improved or degraded detection.
|
||||
|
||||
---
|
||||
|
||||
## G. Client-level behaviour
|
||||
|
||||
Improve/degrade/unchanged counts (validation, relative to `beta = 0`).
|
||||
|
||||
At each condition's pooled-selected beta:
|
||||
|
||||
| condition | selected beta | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|---|
|
||||
| alpha=0.25 | 0.00 | 0/0/10 (trivial) | 0/0/10 (trivial) | 0/0/10 (trivial) |
|
||||
| alpha=0.50 | 0.75 | 7/3/0 (+0.021) | 8/1/1 (+0.027) | 8/1/1 (+0.058) |
|
||||
| alpha=0.75 | 0.50 | 6/4/0 (+0.008) | 9/0/1 (+0.015) | 6/4/0 (−0.007) |
|
||||
|
||||
(mean Δ in parentheses.) At a common `beta = 0.75`:
|
||||
|
||||
| condition | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| alpha=0.25 | 5/4/1 | 5/4/1 | 3/5/2 |
|
||||
| alpha=0.50 | 7/3/0 | 8/1/1 | 8/1/1 |
|
||||
| alpha=0.75 | 3/7/0 | 9/0/1 | 5/5/0 |
|
||||
|
||||
The client-level response is seed-dependent at every alpha. `N_ij` vs ΔF1 is non-monotonic
|
||||
everywhere. Client ids are per-seed realisations and are never pooled across seeds.
|
||||
|
||||
---
|
||||
|
||||
## H. Relation to RQ1 (support imbalance ↔ displacement ↔ rare-class performance)
|
||||
|
||||
Observed, descriptive:
|
||||
|
||||
- Realized support imbalance decreases with alpha (mean class CV 1.442 / 1.183 / 0.881;
|
||||
`|S_j| < M` cells 19 / 7 / 3), and mean prototype displacement decreases with it
|
||||
(3.422 / 2.099 / 1.305). Directionally consistent across the three conditions.
|
||||
- Rare-class validation *levels* do **not** follow the same ordering (pooled means ≈ 0.59 /
|
||||
0.56–0.58 / 0.47). Absolute cross-condition levels are not controlled because the
|
||||
training-derived rare-class sets and per-client eval partitions differ.
|
||||
|
||||
**No causal conclusion** is drawn: the association between support imbalance and displacement is
|
||||
mechanical under the beta formula, while the relationship to rare-class performance is
|
||||
confounded by which classes are rare and how much support they have in each partition.
|
||||
|
||||
---
|
||||
|
||||
## I. Relation to RQ2 (how beta changes prototypes and rare-class performance)
|
||||
|
||||
- **Prototype position:** increasing beta moves every class prototype monotonically away from
|
||||
its beta=0 equal mean (by construction, because supports are unequal); the magnitude grows
|
||||
with support imbalance (mean class displacement at beta=1: 1.31 / 2.10 / 3.42 as imbalance
|
||||
grows). `|S_j| = 1` would leave the aggregate unchanged, but that case does not occur here.
|
||||
- **Rare-class performance:** the validation response is **non-monotone in beta** and
|
||||
condition-dependent: monotone decreasing at alpha=0.25, peaking at 0.75 at alpha=0.50,
|
||||
peaking at 0.50 at alpha=0.75. The effect magnitude is small relative to between-seed
|
||||
variance at all three conditions.
|
||||
|
||||
---
|
||||
|
||||
## J. Relation to RQ3 (does selected beta differ across non-IID conditions?)
|
||||
|
||||
Yes, descriptively: the pooled-selected beta is **0.00 (alpha=0.25), 0.75 (alpha=0.50),
|
||||
0.50 (alpha=0.75)**. It is **not monotone** in alpha.
|
||||
|
||||
Caveats that forbid a trend claim:
|
||||
|
||||
- only **three** alpha levels and **three** project-defined seeds, **one** dataset;
|
||||
- one selection per condition, each statistically weak (between-seed std ≈ between-beta
|
||||
spread or larger) and with **no tie-break** exercised;
|
||||
- the realized partitions differ, so the non-IID "condition" is only partially summarized by
|
||||
alpha;
|
||||
- **no inferential monotonic-trend test** has been performed.
|
||||
|
||||
RQ3 asks whether beta changes with conditions; it does not require a monotonic answer.
|
||||
|
||||
---
|
||||
|
||||
## K. Limitations
|
||||
|
||||
- One dataset (X-IIoTID), three alpha values, three seeds; no cross-dataset claim.
|
||||
- The `S_j` participant/denominator rule remains a **REPRODUCTION ASSUMPTION** (material in
|
||||
19/27 cells at alpha=0.25, 7/27 at alpha=0.50, 3/27 at alpha=0.75); all comparisons are
|
||||
conditional on both modes sharing it.
|
||||
- Train/validation/test adaptation (70/10/20 stratified) differs from the paper's literal
|
||||
80/20; the validation eval partition is a project decision.
|
||||
- **No 5G-NIDD beta result yet.**
|
||||
- **No inferential significance testing**; no global beta optimum. Each selected beta is
|
||||
"best among the tested candidates under pooled validation for that condition".
|
||||
- Cross-alpha absolute metrics are not directly comparable because the rare-class sets, global
|
||||
rare support, and per-client eval partitions differ by condition/seed; only within-condition
|
||||
contrasts and relative patterns are interpreted.
|
||||
Reference in new issue
Block a user