experiment: validate beta selection at alpha 0.25
This commit is contained in:
willtanoe committed
1 parent
6b97a0d38b
commit
ff68d380e2
22 files changed
+33471
-209
No files matched your search
+253
-209
@@ -3,25 +3,27 @@
|
||||
This is the canonical continuation point for the next agent session. It is written to be
|
||||
readable without prior chat history, terminal scrollback, or git history.
|
||||
|
||||
Last updated: 2026-09-25 (Session 6 + post-experiment audit). Session 6 type:
|
||||
**protocol-scaling / stability experiment**. The single Session-5 condition was extended
|
||||
from seed 0 to seeds `{0,1,2}`; beta was selected by the **pooled-across-seed validation**
|
||||
rule and the selected beta evaluated on test. **No full grid was launched; no
|
||||
alpha/dataset expansion was run.** A strict independent audit
|
||||
(`docs/research/SESSION6_AUDIT.md`) returned **PASS WITH MINOR CORRECTIONS**; `beta_star =
|
||||
0.75` was independently reproduced from validation artifacts.
|
||||
Last updated: 2026-09-26 (Session 7, post-experiment audit complete). Session 7 type:
|
||||
**protocol-scaling / non-IID-severity experiment**. Session 6 completed X-IIoTID `alpha=0.50`
|
||||
for seeds `{0,1,2}`. Session 7 completed X-IIoTID **`alpha=0.25`** for seeds `{0,1,2}` across
|
||||
the five frozen beta candidates plus `original_protean`, using the **same pooled-across-seed
|
||||
validation-only** selection rule. **alpha=0.75 and 5G-NIDD were not run.** No commit or push
|
||||
was made.
|
||||
|
||||
> **Audit summary (Session 6).** Verdict: **PASS WITH MINOR CORRECTIONS**. Independently
|
||||
> recomputed `beta_star = 0.75` from `val_per_client.csv`/`val_selection.json`; all 15
|
||||
> seed×beta seed-level scores and the pooled mean/sample-std match the report exactly.
|
||||
> Validation-only selection confirmed in code (`analyze_multiseed_beta.py` reads only
|
||||
> `val_selection.json`; test read in a later block); `test_used_for_selection=false`
|
||||
> upheld. Rare-class flags match the training-derived definition for all 30 cells;
|
||||
> manifests reproduce from raw data with disjoint splits. beta=0 vs `original_protean` =
|
||||
> semantically equivalent + floating-point divergence (class B). Minor corrections:
|
||||
> persisted `artifacts/analysis/xiiotid__alpha0.5__multiseed__runtime.csv` (runtime
|
||||
> provenance) and clarified the client improve/degrade counts (unchanged/tie clients).
|
||||
> **Session 6 is safe to checkpoint.**
|
||||
> **Audit status (Session 7).** Independent audit verdict: **PASS WITH MINOR CORRECTIONS**
|
||||
> (`docs/research/SESSION7_AUDIT.md`). `beta_star(alpha=0.25) = 0.00` confirmed from validation
|
||||
> only; leakage audit PASS; `beta=0` class **B**; source/data integrity PASS; pytest 75 passed,
|
||||
> 0 failed, 0 skipped, 0 warnings. Corrections were documentation-only (cross-alpha label
|
||||
> disambiguation + one wording nuance); no `beta_star` or scientific interpretation changed.
|
||||
> **Session 7 is safe to checkpoint.**
|
||||
|
||||
> **Headline (Session 7).** PRIMARY pooled-validation selected beta for X-IIoTID
|
||||
> `alpha=0.25` = **0.00** (validation only). The pooled validation curve is monotonically
|
||||
> **decreasing** in beta at alpha=0.25, whereas at alpha=0.50 it peaked at 0.75. Per-seed
|
||||
> diagnostics were 0.75 / 0.50 / 0.00. beta=0 is class **B** (semantically equivalent +
|
||||
> floating-point divergence) for all three seeds; no semantic discrepancy. The realized
|
||||
> alpha=0.25 partitions are materially more heterogeneous (19/27 class-condition cells with
|
||||
> `|S_j| < M` vs 7/27 at alpha=0.50). Effect remains small relative to between-seed variance.
|
||||
|
||||
---
|
||||
|
||||
@@ -29,8 +31,8 @@ alpha/dataset expansion was run.** A strict independent audit
|
||||
|
||||
Build the code and experiment infrastructure for a thesis study that modifies **only the
|
||||
server-side prototype aggregation** of PROTEAN (Chennoufi et al., 2025), a federated
|
||||
intrusion-detection framework. The planned mechanism weights each contributing client's
|
||||
class prototype by its class sample support:
|
||||
intrusion-detection framework. The mechanism weights each contributing client's class
|
||||
prototype by its class sample support:
|
||||
|
||||
```
|
||||
Cbar_j(beta) = ( sum_{i in S_j} N_ij^beta * C_ij ) / ( sum_{i in S_j} N_ij^beta )
|
||||
@@ -55,303 +57,345 @@ bitwise identical to the baseline (see §7).
|
||||
- Frozen baseline (`original_protean`) source unchanged; no `equal_available` mode added;
|
||||
beta candidate set unchanged (`{0, .25, .5, .75, 1}`).
|
||||
|
||||
## 3. Session 6 objective and outcome
|
||||
## 3. Session 7 objective and outcome
|
||||
|
||||
Objective: extend the validated paired protocol from one seed to seeds `{0,1,2}` for
|
||||
X-IIoTID alpha=0.50, reuse the seed-0 Session-5 results, run the two new seeds across the
|
||||
five beta candidates plus `original_protean`, select beta with **pooled validation only**,
|
||||
evaluate the selected beta on test, and produce cross-seed direct-effect / client-level /
|
||||
stability analyses + docs. **All acceptance items were met.** No commit or push was made.
|
||||
Objective: evaluate one additional non-IID severity (X-IIoTID `alpha=0.25`, seeds `{0,1,2}`)
|
||||
across the five beta candidates using the exact Session-6 three-seed pooled-validation
|
||||
protocol, and test whether the pooled-selected beta changes under a more heterogeneous
|
||||
condition. All 18 runs (15 `support_weighted` + 3 `original_protean`) completed with rc=0.
|
||||
Report: `docs/research/ALPHA025_BETA_VALIDATION.md`.
|
||||
|
||||
Outcome: **PRIMARY pooled-validation selected beta = 0.75** (validation only). Per-seed
|
||||
diagnostic selections varied (0.50 / 1.00 / 0.75), and between-seed variance is comparable
|
||||
to the between-beta spread, so the selection is arithmetically clear but statistically
|
||||
weak. The protocol ran cleanly end to end.
|
||||
Outcome: **PRIMARY pooled-validation selected beta = 0.00** (validation only);
|
||||
`test_used_for_selection = false` for all 15 support_weighted runs. Pooled means: 0.00 →
|
||||
0.591704, 0.25 → 0.588691, 0.50 → 0.585851, 0.75 → 0.579463, 1.00 → 0.532099 (monotone
|
||||
decreasing). Between-seed std 0.045–0.090; the effect is small relative to seed variance.
|
||||
|
||||
## 4. Condition and pairing
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| dataset / alpha / seeds | `xiiotid` / 0.50 / `{0,1,2}` |
|
||||
| runs | 15 `support_weighted` + 3 `original_protean` (seed 0 reused; 12 run this session) |
|
||||
| dataset / alpha / seeds | `xiiotid` / 0.25 / `{0,1,2}` |
|
||||
| runs | 15 `support_weighted` + 3 `original_protean`, all new this session |
|
||||
| training | 10 clients, 10 rounds, 3 local epochs, batch 256, lr 1e-3, mu 0.1, lambda 1.0, Adam |
|
||||
| code revision | unchanged from HEAD `50f2da72166e4f609077d65b4916f745da5e8e3b`; only read-only analysis code added |
|
||||
| code revision | unchanged from HEAD `6b97a0d38b839c737c49f41006817503d0672871`; only docs/analysis artifacts added |
|
||||
|
||||
Manifests (reused, never regenerated; one content hash per seed across all six runs):
|
||||
Manifests (reused, never regenerated; one content hash per seed across all six runs; each was
|
||||
reproduced from the raw dataset with split/`N_ij`/partition hashes verified):
|
||||
|
||||
| seed | manifest | content hash | partition hash | split hash | rng seed |
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | `artifacts/partitions/xiiotid__alpha0.5__seed0.json` | `e81a6538…` | `2465485539…` | `d0acc0c1…` | 2009062324 |
|
||||
| 1 | `artifacts/partitions/xiiotid__alpha0.5__seed1.json` | `54ea1df1…` | `4061c5a1…` | `880a9f82…` | 621679668 |
|
||||
| 2 | `artifacts/partitions/xiiotid__alpha0.5__seed2.json` | `be881ac6…` | `1118e7fa…` | `8e0db0be…` | 1438483982 |
|
||||
| 0 | `artifacts/partitions/xiiotid__alpha0.25__seed0.json` | `839ba18e…` | `60faf299…` | `d0acc0c1…` | 1617371214 |
|
||||
| 1 | `artifacts/partitions/xiiotid__alpha0.25__seed1.json` | `84c20b4b…` | `6ffb10ae…` | `880a9f82…` | 1670706564 |
|
||||
| 2 | `artifacts/partitions/xiiotid__alpha0.25__seed2.json` | `d6e68ee1…` | `c8e1ad31…` | `8e0db0be…` | 795318573 |
|
||||
|
||||
Pairing proof: round-0 snapshot byte-identical across the five betas **and**
|
||||
`original_protean` within each seed — seed 0 `e27f41b3a62eb83c…`, seed 1
|
||||
`b7f61c995e97474c…`, seed 2 `5b749f4185e11fc7…`. `train_support_Nij.csv`,
|
||||
`characterization.csv`, `rare_unseen.csv` identical across each seed's runs.
|
||||
`manifest_reload_ok = true` everywhere.
|
||||
`original_protean` within each seed — seed 0 `cd821beb…`, seed 1 `59d2d2bc…`, seed 2
|
||||
`47befeff…`. `train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv` identical
|
||||
across each seed's runs. `manifest_reload_ok = true`, `smoke = false` everywhere.
|
||||
|
||||
## 5. Validation results (USED FOR SELECTION) and pooled selection
|
||||
|
||||
Primary metric: validation per-client-mean rare-class macro-F1. All 10 clients eligible
|
||||
in every seed. Full per-seed table:
|
||||
`artifacts/analysis/xiiotid__alpha0.5__multiseed__validation_per_seed.csv`.
|
||||
Primary metric: validation per-client-mean rare-class macro-F1. All 10 clients eligible in
|
||||
every seed. Full tables: `artifacts/analysis/xiiotid__alpha0.25__multiseed__validation_per_seed.csv`.
|
||||
|
||||
Per-seed validation per-client-mean rare-class macro-F1:
|
||||
|
||||
| beta | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| 0.00 | 0.557340 | 0.628624 | 0.447118 |
|
||||
| 0.25 | 0.571787 | 0.636205 | 0.458310 |
|
||||
| 0.50 | 0.585762 | 0.645189 | 0.471015 |
|
||||
| 0.75 | 0.578003 | 0.655160 | 0.504937 |
|
||||
| 1.00 | 0.551443 | 0.683117 | 0.485877 |
|
||||
| 0.00 | 0.616380 | 0.492425 | 0.666307 |
|
||||
| 0.25 | 0.609913 | 0.514237 | 0.641922 |
|
||||
| 0.50 | 0.595247 | 0.537040 | 0.625267 |
|
||||
| 0.75 | 0.654033 | 0.535672 | 0.548685 |
|
||||
| 1.00 | 0.578385 | 0.530942 | 0.486970 |
|
||||
|
||||
Pooled across seeds (`...__validation_pooled.csv`):
|
||||
|
||||
| beta | pooled mean | std (sample) | min | max | selected |
|
||||
|---|---|---|---|---|---|
|
||||
| 0.00 | 0.544361 | 0.091447 | 0.447118 | 0.628624 | |
|
||||
| 0.25 | 0.555434 | 0.090068 | 0.458310 | 0.636205 | |
|
||||
| 0.50 | 0.567322 | 0.088539 | 0.471015 | 0.645189 | |
|
||||
| **0.75** | **0.579367** | **0.075121** | 0.504937 | 0.655160 | **yes** |
|
||||
| 1.00 | 0.573479 | 0.100450 | 0.485877 | 0.683117 | |
|
||||
| **0.00** | **0.591704** | 0.089529 | 0.492425 | 0.666307 | **yes** |
|
||||
| 0.25 | 0.588691 | 0.066435 | 0.514237 | 0.641922 | |
|
||||
| 0.50 | 0.585851 | 0.044858 | 0.537040 | 0.625267 | |
|
||||
| 0.75 | 0.579463 | 0.064906 | 0.535672 | 0.654033 | |
|
||||
| 1.00 | 0.532099 | 0.045718 | 0.486970 | 0.578385 | |
|
||||
|
||||
- **PRIMARY selected beta (`beta_star`) = 0.75**, max pooled validation per-client-mean
|
||||
rare-class macro-F1; **no tie-break used**; `test_used_for_selection = false` for all 15
|
||||
runs. The secondary pooled recall is highest at beta=0.25 (recall is only a tie-break).
|
||||
- **SECONDARY per-seed diagnostics only:** seed 0 → 0.50, seed 1 → 1.00, seed 2 → 0.75.
|
||||
These are stability diagnostics, not the condition's selection rule.
|
||||
- **PRIMARY selected beta (`beta_star_alpha025`) = 0.00**, max pooled validation
|
||||
per-client-mean rare-class macro-F1; **no tie-break used**; selection is validation-only.
|
||||
- **SECONDARY per-seed diagnostics only:** seed 0 → 0.75, seed 1 → 0.50, seed 2 → 0.00.
|
||||
|
||||
## 6. Final test evaluation of the pooled-selected beta
|
||||
|
||||
`beta_star = 0.75` was fixed from pooled validation before test was read.
|
||||
`beta_star = 0.00` was fixed from pooled validation before test was read.
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std |
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.636189 | 0.671937 | 0.688381 | 0.665503 ± 0.026685 |
|
||||
| rare-class recall | 0.857543 | 0.851883 | 0.878321 | 0.862583 ± 0.013921 |
|
||||
| macro-F1 | 0.726298 | 0.729015 | 0.738194 | 0.731169 ± 0.006234 |
|
||||
| accuracy | 0.888670 | 0.925493 | 0.905012 | 0.906392 ± 0.018450 |
|
||||
| weighted-F1 | 0.922278 | 0.944734 | 0.933884 | 0.933632 ± 0.011230 |
|
||||
| rare-class macro-F1 | 0.641362 | 0.580555 | 0.674489 | 0.632136 ± 0.047642 |
|
||||
| rare-class recall | 0.830701 | 0.825699 | 0.807370 | 0.821257 ± 0.012283 |
|
||||
| macro-F1 | 0.674558 | 0.665039 | 0.704433 | 0.681343 ± 0.020555 |
|
||||
| accuracy | 0.874551 | 0.863490 | 0.884002 | 0.874015 ± 0.010267 |
|
||||
| weighted-F1 | 0.906156 | 0.899864 | 0.914627 | 0.906882 ± 0.007408 |
|
||||
|
||||
`original_protean` for the same seeds/condition:
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std |
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.609620 | 0.655665 | 0.662275 | 0.642520 ± 0.028683 |
|
||||
| rare-class recall | 0.871487 | 0.889365 | 0.870018 | 0.876957 ± 0.010771 |
|
||||
| macro-F1 | 0.707701 | 0.709176 | 0.710017 | 0.708965 ± 0.001172 |
|
||||
| accuracy | 0.890432 | 0.900885 | 0.907045 | 0.899454 ± 0.008398 |
|
||||
| weighted-F1 | 0.917585 | 0.929132 | 0.927481 | 0.924733 ± 0.006245 |
|
||||
| rare-class macro-F1 | 0.639810 | 0.582522 | 0.676039 | 0.632790 ± 0.047152 |
|
||||
| rare-class recall | 0.829601 | 0.826700 | 0.809595 | 0.821965 ± 0.010811 |
|
||||
| macro-F1 | 0.673103 | 0.666981 | 0.705969 | 0.682018 ± 0.020967 |
|
||||
| accuracy | 0.873722 | 0.867341 | 0.885231 | 0.875431 ± 0.009067 |
|
||||
| weighted-F1 | 0.905466 | 0.902075 | 0.915416 | 0.907652 ± 0.006934 |
|
||||
|
||||
Test columns are **descriptive only** and played no part in selection. No superiority or
|
||||
generalisation is claimed: single dataset, single alpha, three project-defined seeds, and
|
||||
between-seed spread ~ between-method difference.
|
||||
Descriptive difference for test rare-class macro-F1 (`beta_star` − `original_protean`) is
|
||||
+0.001552 / −0.001967 / −0.001550 (mean −0.000655) — i.e. essentially identical, as expected
|
||||
because `beta = 0` is mathematically the equal mean over `S_j`. Test is descriptive only; no
|
||||
superiority or generalisation is claimed.
|
||||
|
||||
## 7. `beta = 0` equivalence result (seeds 1 and 2)
|
||||
## 7. `beta = 0` equivalence result (all three seeds)
|
||||
|
||||
For seeds 1 and 2, `support_weighted beta=0` vs frozen `original_protean`:
|
||||
For seeds 0, 1, 2, `support_weighted beta=0` vs frozen `original_protean`:
|
||||
|
||||
- Same manifest/split/hashes: **yes**.
|
||||
- Round-0 snapshot byte-identical: **yes** (seed 1 `b7f61c99…`, seed 2 `5b749f41…`).
|
||||
- Round-0 snapshot byte-identical: **yes** (`cd821beb…`, `59d2d2bc…`, `47befeff…`).
|
||||
- `train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv`: **identical**.
|
||||
- Aggregation: **mathematically equivalent, not bitwise** (`Σ(C/count)` vs
|
||||
`Σ((1/count)·C)`, ~1 ULP ordering difference that feeds back through round≥1).
|
||||
- Test/history/per-client/per-class: **differ slightly**.
|
||||
- Classification: **mathematically equivalent + floating-point numerical divergence;
|
||||
no semantic discrepancy** in either seed. Same result as Session 5 seed 0.
|
||||
- Independent aggregation (`ΣC/n` vs `(1/n)ΣC`): ≤ 4.4e-16 (1 ULP ordering).
|
||||
- Test/history/per-client/per-class: **differ slightly** (history rare-macro-F1 max diff
|
||||
3.6e-3–6.6e-3; test metric max diff 1.6e-3–3.9e-3).
|
||||
- Classification: **B — mathematically equivalent + floating-point numerical divergence; no
|
||||
semantic discrepancy** in any seed. Machine-readable:
|
||||
`artifacts/analysis/xiiotid__alpha0.25__multiseed__beta0_equivalence.csv`.
|
||||
|
||||
Conditional on the shared `S_j` reproduction assumption; not proof `beta=0` reproduces
|
||||
original PROTEAN.
|
||||
|
||||
## 8. Direct prototype-effect findings (fixed round-0 snapshots, no retraining)
|
||||
## 8. Partition characterization (realized)
|
||||
|
||||
`||Cbar_j(1) - Cbar_j(0)||`, cross-seed mean (std):
|
||||
`|S_j| < M` class-condition cells and missing client-class cells (training split):
|
||||
|
||||
- Largest means: `Lateral_movement` 2.724 (1.035), `RDOS` 2.631 (1.165),
|
||||
`Weaponization` 2.561 (1.228), `Tampering` 2.354 (1.165), `Exploitation` 2.344 (0.412).
|
||||
- Smallest: `Reconnaissance` 1.002 (0.261), `Normal` 1.557 (0.548), `C&C` 1.694 (0.505).
|
||||
- Every class moves for beta>0 (all classes have unequal support); the **rank order is
|
||||
seed-sensitive** (seed 0 top = Weaponization, seed 1 top = RDOS, seed 2 top =
|
||||
Lateral_movement). Displacement is a direct-aggregation measure only, not a detection
|
||||
result. Outputs: `...__prototype_effect_pooled.csv` and the per-seed
|
||||
`...__prototype_snapshot__prototype_effect.csv` / `..._weights.csv`.
|
||||
| condition | `|S_j| < M` cells | missing client-class cells | client-size max/min (seeds) |
|
||||
|---|---|---|---|
|
||||
| alpha=0.25 | **19 / 27** | **38** | 15.6, 20.8, 7.8 |
|
||||
| alpha=0.50 | 7 / 27 | 9 | 5.1, 7.2, 9.9 |
|
||||
|
||||
## 9. Client-level findings (validation, vs beta = 0)
|
||||
Union unseen classes per seed: alpha=0.25 = 6/6/5 (max 4 on a client); alpha=0.50 = 3/3/1
|
||||
(max 2). Union rare classes per seed: alpha=0.25 = 8/7/8; alpha=0.50 = 6/7/7. All clients have
|
||||
exactly two rare classes at both alphas. Full per-class `N_ij`, CV and max/min in
|
||||
`docs/research/ALPHA025_BETA_VALIDATION.md` §B. The lower alpha does **not** uniformly increase
|
||||
every realized statistic (e.g. alpha=0.25 seed 2 client-size ratio 7.8 < alpha=0.50 seed 2's
|
||||
9.9), so interpretation uses the realized partitions, not alpha alone.
|
||||
|
||||
Full detail: `...__multiseed__client_validation_detail.csv`; summary:
|
||||
## 9. Direct prototype-effect findings (fixed round-0 snapshots, no retraining)
|
||||
|
||||
`||Cbar_j(1) - Cbar_j(0)||`, cross-seed mean (std), alpha=0.25 vs alpha=0.50:
|
||||
|
||||
| class | a25 mean (std) | a50 mean (std) |
|
||||
|---|---|---|
|
||||
| Exfiltration | 4.316 (0.898) | 2.020 (0.502) |
|
||||
| Tampering | 4.248 (0.241) | 2.354 (1.165) |
|
||||
| C&C | 4.020 (0.290) | 1.694 (0.505) |
|
||||
| Lateral_movement | 3.726 (1.383) | 2.724 (1.035) |
|
||||
| Exploitation | 3.464 (1.113) | 2.344 (0.412) |
|
||||
| Reconnaissance | 3.339 (1.107) | 1.002 (0.261) |
|
||||
| Weaponization | 3.124 (0.585) | 2.561 (1.228) |
|
||||
| RDOS | 2.566 (1.219) | 2.631 (1.165) |
|
||||
| Normal | 1.997 (0.533) | 1.557 (0.548) |
|
||||
|
||||
Mean displacement across classes is larger at alpha=0.25 (3.42) than alpha=0.50 (2.10), because
|
||||
support is more imbalanced and more classes are missing contributors. Displacement is a
|
||||
direct-aggregation measure only, not a detection result. Outputs:
|
||||
`artifacts/analysis/xiiotid__alpha0.25__multiseed__prototype_effect_pooled.csv` and per-seed
|
||||
`...__seed{0,1,2}__support_weighted__beta0__prototype_snapshot__prototype_effect.csv`.
|
||||
|
||||
## 10. Client-level findings (validation, vs beta = 0)
|
||||
|
||||
Detail: `...__multiseed__client_validation_detail.csv`; summary:
|
||||
`...__client_validation_summary.csv`.
|
||||
|
||||
- At `beta = 0.75`, improve/degrade/**unchanged** counts are 7/3/0 (seed 0), 8/1/1
|
||||
(seed 1, tie = client 7), 8/1/1 (seed 2, tie = client 4); mean Δ is positive in all
|
||||
three (+0.021 / +0.027 / +0.058).
|
||||
- At `beta = 1.00`, seed 0 flips to 4 improve / 6 degrade / 0 unchanged, while seed 1
|
||||
improves 9/10 with 1 unchanged (client 7) — the high-beta response is seed-dependent.
|
||||
- `min positive attack N_ij` vs ΔF1 is **non-monotonic**: N=1 clients range −0.021…+0.041
|
||||
at beta=0.75; the largest gain (+0.177) is an N=10 client; low support neither
|
||||
systematically benefits nor degrades. Client ids are different realisations across
|
||||
seeds and are not treated as one population. Descriptive only.
|
||||
- At alpha=0.25 the higher-beta response is **seed-dependent and often negative**:
|
||||
improve/degrade/unchanged at beta=0.75 = 5/4/1 (seed 0, mean Δ +0.038), 5/4/1 (seed 1,
|
||||
+0.043), 3/5/2 (seed 2, −0.118). At beta=1.00 seed 2 degrades 6/10 clients with a worst
|
||||
client Δ = −0.678, while seed 1 still gains on average.
|
||||
- This **contrasts with alpha=0.50**, where beta=0.75 improved the majority of clients in every
|
||||
seed. Client ids are per-seed realisations and are not one population; the analysis is
|
||||
descriptive and non-causal, and `N_ij` vs ΔF1 remains non-monotonic.
|
||||
|
||||
## 10. Runtime / resources
|
||||
## 11. Comparison with alpha=0.50 (RQ3)
|
||||
|
||||
CPU only, no GPU, no swap, exit 0 on all 18 runs (15 new-session + reused). Per-run
|
||||
wall-clock 40:23–42:21; CPU 733–745 %; peak RSS 6.21 GiB (seed 0), 7.48 GiB (seed 1),
|
||||
6.45 GiB (seed 2). New compute this session: 12 runs, ≈ 8 h 11 m wall-clock, run
|
||||
sequentially (8 cores; concurrent runs would not reduce total CPU work).
|
||||
Seed-0 numbers were reused and not re-run. Machine-readable per-run values:
|
||||
`artifacts/analysis/xiiotid__alpha0.5__multiseed__runtime.csv` (added by the audit;
|
||||
seed 0 sourced from Session-5 docs, seeds 1–2 from `/usr/bin/time -v` logs).
|
||||
| | alpha=0.50 | alpha=0.25 |
|
||||
|---|---|---|
|
||||
| pooled-selected beta | 0.75 | **0.00** |
|
||||
| pooled validation curve | rises to 0.75 | monotone decreasing |
|
||||
| pooled mean range | 0.544–0.579 | 0.532–0.592 |
|
||||
| pooled std range | 0.075–0.100 | 0.045–0.090 |
|
||||
| `|S_j| < M` cells | 7/27 | 19/27 |
|
||||
| mean `||Cbar(1)−Cbar(0)||` | 2.10 | 3.42 |
|
||||
|
||||
## 11. Methodological decisions / assumptions
|
||||
The selection **changes** with non-IID severity. This is documented as a descriptive
|
||||
association accompanied by the realized partition characteristics; **no causal claim is made
|
||||
that alpha by itself caused the difference**. The effect is small relative to between-seed
|
||||
variance at both settings.
|
||||
|
||||
> **Audited label (do not confuse).** Condition-level **pooled** selections are alpha=0.50 →
|
||||
> **0.75** and alpha=0.25 → **0.00**. The value **0.50** is only the Session-5 **seed-0
|
||||
> single-condition** choice and the Session-6 alpha=0.50 **per-seed diagnostic** for seed 0;
|
||||
> it is not the alpha=0.50 pooled selection. Any "0.50 → 0.00" phrasing for condition-level
|
||||
> pooled selection is wrong. Verified in `docs/research/SESSION7_AUDIT.md` §M.
|
||||
|
||||
## 12. Runtime / resources
|
||||
|
||||
CPU only, no GPU, no swap, exit 0 on all 18 runs. Per-run wall-clock 39:49–42:12; CPU
|
||||
741–746 %; peak RSS 8.38 GiB (seed 0), 10.80 GiB (seed 1), 7.51 GiB (seed 2). New compute:
|
||||
18 runs, ≈ 43,799 s ≈ **12.17 h** wall-clock, sequential (8 cores). Machine-readable:
|
||||
`artifacts/analysis/xiiotid__alpha0.25__multiseed__runtime.csv`.
|
||||
|
||||
## 13. Methodological decisions / assumptions
|
||||
|
||||
1. **`S_j` participant/denominator rule** (`S_j = {i | N_ij > 0 and valid C_ij}`, equal
|
||||
weights in the baseline) is a **REPRODUCTION ASSUMPTION**; material in 53/162
|
||||
class-condition cells (32.7 %), 15/18 conditions (`BASELINE_VALIDATION.md` §D).
|
||||
Carry this caveat in all reporting.
|
||||
weights in the baseline) is a **REPRODUCTION ASSUMPTION**; material in 19/27
|
||||
class-condition cells at alpha=0.25 (7/27 at alpha=0.50). Carry this caveat in all reporting.
|
||||
2. `N_ij` is class-specific sample support, not prototype quality.
|
||||
3. Beta is one value for all classes/clients; candidates frozen at `{0,.25,.5,.75,1}`.
|
||||
4. **PRIMARY selection = pooled validation across clients and seeds**; per-seed selection
|
||||
is secondary stability diagnostics; test used only after the pooled beta is fixed.
|
||||
Documented in `EXPERIMENT_PROTOCOL.md` §8 (updated this session).
|
||||
5. The union of rare classes (and global rare support) differs by seed because partitions
|
||||
differ; only the per-client-mean metric is directly comparable across seeds.
|
||||
6. `lambda=1.0`, Adam, lr 1e-3, batch 256 remain PROJECT DECISIONS. No `equal_available`
|
||||
mode was added.
|
||||
4. **PRIMARY selection = pooled validation across clients and seeds**; per-seed selection is
|
||||
secondary stability diagnostics; test used only after the pooled beta is fixed.
|
||||
5. The `scripts/analyze_multiseed_beta.py` already supported `--alpha` (default 0.5) and
|
||||
already includes alpha in output names; **no code change was needed**. alpha=0.50 outputs
|
||||
were re-run and are byte-identical to the Session-6 artifacts (reproducibility verified).
|
||||
6. `lambda=1.0`, Adam, lr 1e-3, batch 256 remain PROJECT DECISIONS. No `equal_available` mode.
|
||||
|
||||
## 12. Unresolved questions / blockers
|
||||
## 14. Unresolved questions / blockers
|
||||
|
||||
1. `S_j` reproduction ambiguity — unresolved; no official PROTEAN code exists.
|
||||
2. **Pooled beta selection is statistically weak here:** pooled means span only
|
||||
0.544→0.579 while within-beta across-seed std is 0.075–0.100; three seeds give a
|
||||
standard error of the same order as the between-beta spread. The beta effect is small
|
||||
relative to seed variance.
|
||||
3. `beta = 0` is not bitwise identical to `original_protean`; documentation decision on how
|
||||
to report this remains open.
|
||||
2. **Pooled beta selection remains statistically weak at both alphas (descriptive only):**
|
||||
the across-seed sample std (0.045–0.090) is comparable to, or larger than, the total
|
||||
between-beta pooled-mean spread (0.035 at alpha=0.50, 0.060 at alpha=0.25). No inferential
|
||||
test was performed, so this is a descriptive comparison, not evidence of insignificance.
|
||||
3. `beta = 0` is not bitwise identical to `original_protean`; documentation decision on how to
|
||||
report this remains open.
|
||||
4. Session-2 manifests still store `preprocessing_hash = null` / `config_hash = null`.
|
||||
5. Only alpha=0.50, X-IIoTID, three seeds measured.
|
||||
5. Only alpha=0.25 and 0.50, X-IIoTID, three seeds measured. No 5G-NIDD beta result.
|
||||
6. The alpha=0.25 result (pooled-selected beta = 0) is **one additional condition, not yet a
|
||||
trend across alpha**. Whether any relationship between non-IID severity and the selected beta
|
||||
is monotone would require alpha=0.75 (and is still not implied by two conditions).
|
||||
|
||||
**Audit status (see `docs/research/SESSION6_AUDIT.md`).** No unresolved issue that could
|
||||
change `beta_star` or the scientific interpretation. One inherent audit limitation: the
|
||||
trained model is not persisted, so per-client prediction metrics cannot be recomputed
|
||||
without retraining (forbidden); they were verified via defining source code, exact
|
||||
agreement of their mean with `val_selection.json`, and exact agreement of the global
|
||||
union-rare metric recomputed from `val_confusion_matrix.json`. Pooled std uses the
|
||||
**sample** convention (ddof=1), labelled "(sample)" in the report.
|
||||
## 15. Git status
|
||||
|
||||
## 13. Git status
|
||||
|
||||
No commit or push was made. `git status --short` (results/ is ignored):
|
||||
No commit or push was made. `git status --short` shows only untracked docs/analysis/run
|
||||
artifacts (`results/` is gitignored; `artifacts/analysis/` is intentionally tracked):
|
||||
|
||||
```
|
||||
M docs/handoff/SESSION_HANDOFF.md
|
||||
M docs/research/EXPERIMENT_PROTOCOL.md
|
||||
?? docs/research/MULTISEED_BETA_VALIDATION.md
|
||||
?? docs/research/SESSION6_AUDIT.md
|
||||
?? scripts/analyze_multiseed_beta.py
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__multiseed__*.csv / __selection_summary.json
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__seed{1,2}__*__prototype_snapshot.json
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__seed{1,2}__support_weighted__beta0__prototype_snapshot__{prototype_effect,prototype_weights}.csv
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__seed0__support_weighted__beta{0.25,0.5,0.75,1}__prototype_snapshot.json (pre-existing from Session 5)
|
||||
?? docs/research/ALPHA025_BETA_VALIDATION.md
|
||||
?? docs/research/SESSION7_AUDIT.md
|
||||
?? artifacts/analysis/xiiotid__alpha0.25__* (runs' snapshots, effect CSVs, multiseed tables)
|
||||
?? (pre-existing Session-5/6 untracked alpha=0.5 views, unchanged)
|
||||
```
|
||||
|
||||
`planningawal/` and both raw datasets are unchanged; `results/` is gitignored.
|
||||
`HEAD = 50f2da72166e4f609077d65b4916f745da5e8e3b`. No commit or push was made.
|
||||
`planningawal/` and both raw datasets are unchanged (audit re-verified SHA-256:
|
||||
X-IIoTID `7b929005…`, 5G-NIDD `fa36f808…`). No model/training/aggregation source changed
|
||||
(working-tree blobs equal HEAD). `artifacts/partitions/` unmodified. No alpha=0.75 run and no
|
||||
5G-NIDD `support_weighted` run exists. `equal_available` absent.
|
||||
`HEAD = 6b97a0d38b839c737c49f41006817503d0672871`. No commit or push was made.
|
||||
|
||||
## 14. Files created / modified this session
|
||||
## 15b. Session 7 audit result (independent)
|
||||
|
||||
- Verdict: **PASS WITH MINOR CORRECTIONS** (`docs/research/SESSION7_AUDIT.md`).
|
||||
- Run inventory 18/18, pairing hash-identical, manifests reproduced from raw data.
|
||||
- Validation recomputed exactly; **pooled `beta_star(alpha=0.25) = 0.00`** (validation only,
|
||||
monotone decreasing in beta). `beta_star` test and `original_protean` test means/stds exact.
|
||||
- Leakage audit: selection reads validation only; `test_used_for_selection=false` upheld.
|
||||
- `beta=0` equivalence: class **B** for all three seeds (no semantic discrepancy).
|
||||
- Source/data integrity: PASS. pytest: **75 passed, 0 failed, 0 skipped, 0 warnings**.
|
||||
- Corrections: cross-alpha label disambiguation (documentation-only) and one wording nuance.
|
||||
No arithmetic, metric-label, std-convention, or provenance defect was found.
|
||||
|
||||
## 16. Files created / modified this session
|
||||
|
||||
Created:
|
||||
|
||||
- `docs/research/MULTISEED_BETA_VALIDATION.md`
|
||||
- `docs/research/SESSION6_AUDIT.md` (post-experiment audit)
|
||||
- `scripts/analyze_multiseed_beta.py`
|
||||
- `artifacts/analysis/xiiotid__alpha0.5__multiseed__*` (validation per-seed/pooled,
|
||||
client summary/detail/deltas, prototype-effect pooled, test descriptive, selection
|
||||
summary, and `...__runtime.csv` added by the audit)
|
||||
- seed-1/seed-2 `results/...` run dirs (gitignored) and their snapshots / effect CSVs
|
||||
- `docs/research/ALPHA025_BETA_VALIDATION.md`
|
||||
- `docs/research/SESSION7_AUDIT.md` (independent post-experiment audit)
|
||||
- `artifacts/analysis/xiiotid__alpha0.25__seed{0,1,2}__*__prototype_snapshot.json`
|
||||
- `artifacts/analysis/xiiotid__alpha0.25__seed{0,1,2}__support_weighted__beta0__prototype_snapshot__{prototype_effect,prototype_weights}.csv`
|
||||
- `artifacts/analysis/xiiotid__alpha0.25__multiseed__{validation_per_seed,validation_pooled,client_validation_deltas,client_validation_detail,client_validation_summary,prototype_effect_pooled,test_descriptive,runtime,beta0_equivalence}.csv` and `...__selection_summary.json`
|
||||
- `results/xiiotid__alpha0.25__seed{0,1,2}__*` (18 run dirs, gitignored)
|
||||
|
||||
Modified:
|
||||
|
||||
- `docs/research/EXPERIMENT_PROTOCOL.md` (§8 selection hierarchy clarified)
|
||||
- `docs/handoff/SESSION_HANDOFF.md` (this file)
|
||||
|
||||
Reused (not regenerated): all seed-0 Session-5 runs and artifacts.
|
||||
No source or test code was changed. `scripts/analyze_multiseed_beta.py` was re-run for
|
||||
alpha=0.50 and produced byte-identical outputs.
|
||||
|
||||
## 15. Exact recommended next phase
|
||||
## 17. Exact recommended next phase
|
||||
|
||||
**Session 6 is safe to checkpoint** (audit PASS WITH MINOR CORRECTIONS; no issue affects
|
||||
`beta_star`). The protocol is **mechanically** ready to scale, but Session 6 shows the beta
|
||||
effect at alpha=0.50 is small relative to seed-to-seed variance. Recommended, in order:
|
||||
1. **Session 7 is audited and safe to checkpoint** (verdict PASS WITH MINOR CORRECTIONS;
|
||||
see §15b and `docs/research/SESSION7_AUDIT.md`). alpha=0.25 behaved cleanly: pairing valid,
|
||||
beta=0 class B, pooled selection clean, artifacts complete, no test leakage, runtime
|
||||
manageable (≈12 h per alpha). Before checkpointing, stage only the intended docs/analysis
|
||||
artifacts; do not commit `results/` or the raw datasets.
|
||||
2. **Next experiment:** run X-IIoTID `alpha=0.75` under the same protocol (seeds `{0,1,2}`,
|
||||
five betas + `original_protean`, sequential, reusing frozen manifests, pooled
|
||||
validation-only selection), then test evaluation, direct-effect, client-level, and
|
||||
cross-alpha comparison. This would complete the three planned non-IID severities. Do **not**
|
||||
infer a monotone alpha→beta relation from two conditions.
|
||||
3. Do **not** treat `beta = 0.00` (alpha=0.25) or `0.75` (alpha=0.50) as optima. Report each as
|
||||
"best among tested candidates under pooled validation for that condition".
|
||||
4. **Do not launch 5G-NIDD or the full grid** until alpha=0.75 is inspected.
|
||||
5. Use `.venv/bin/python`. Run one alpha at a time; measure runtime as before.
|
||||
6. Update `docs/research/ALPHA025_BETA_VALIDATION.md` (or add `ALPHA075_BETA_VALIDATION.md`),
|
||||
`EXPERIMENT_PROTOCOL.md` only if a real ambiguity appears, and this handoff. Do not commit or
|
||||
push unless asked.
|
||||
|
||||
1. **Do not treat beta=0.75 as a settled optimum.** Report it as "best among tested
|
||||
candidates under pooled validation for X-IIoTID alpha=0.50".
|
||||
2. Before expanding alpha, decide the interpretation question: is the goal to characterise
|
||||
small beta effects against seed noise (requires more seeds), or to run alpha=0.25/0.75
|
||||
as a paired characterisation (accepting the same weak-effect caveat)?
|
||||
3. If expanding, run alpha=0.25 then alpha=0.75 for seeds `{0,1,2}`, the five betas each,
|
||||
reusing frozen manifests, sequentially (~8 h per alpha), with the same pooled
|
||||
validation-only selection. Extend `scripts/analyze_multiseed_beta.py` with `--alpha`.
|
||||
4. Do **not** launch 5G-NIDD or the full grid until the alpha results are inspected.
|
||||
|
||||
## 16. Important commands
|
||||
## 18. Important commands
|
||||
|
||||
```bash
|
||||
# Tests (75 expected)
|
||||
.venv/bin/python -m pytest
|
||||
|
||||
# One full paired beta run (CPU, ~41 min, ~6.5-7.5 GiB)
|
||||
.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.5 --seed 1 \
|
||||
# One full paired beta run (CPU, ~40 min, ~7.5-11.3 GiB)
|
||||
.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.25 --seed 1 \
|
||||
--clients 10 --rounds 10 --local-epochs 3 \
|
||||
--aggregation-mode support_weighted --beta 0.5
|
||||
|
||||
# Pooled multi-seed validation analysis (read-only, no retraining)
|
||||
.venv/bin/python scripts/analyze_multiseed_beta.py --dataset xiiotid --alpha 0.5 \
|
||||
--seeds 0 1 2
|
||||
# Pooled multi-seed validation analysis (read-only, no retraining; --alpha already supported)
|
||||
.venv/bin/python scripts/analyze_multiseed_beta.py --dataset xiiotid --alpha 0.25 --seeds 0 1 2
|
||||
|
||||
# Direct prototype-effect analysis (no retraining)
|
||||
.venv/bin/python scripts/analyze_prototype_effect.py \
|
||||
--snapshot artifacts/analysis/xiiotid__alpha0.5__seed1__support_weighted__beta0__prototype_snapshot.json \
|
||||
--snapshot artifacts/analysis/xiiotid__alpha0.25__seed0__support_weighted__beta0__prototype_snapshot.json \
|
||||
--class-names "C&C,Exfiltration,Exploitation,Lateral_movement,Normal,RDOS,Reconnaissance,Tampering,Weaponization" \
|
||||
--output-dir artifacts/analysis
|
||||
```
|
||||
|
||||
## 17. Recommended next-session prompt
|
||||
## 19. Recommended next-session prompt
|
||||
|
||||
```
|
||||
You are continuing the perdalam-protean research repository (Session 7).
|
||||
You are continuing the perdalam-protean research repository (Session 8).
|
||||
|
||||
FIRST, read completely:
|
||||
1. docs/handoff/SESSION_HANDOFF.md
|
||||
2. docs/research/MULTISEED_BETA_VALIDATION.md
|
||||
3. docs/research/SESSION6_AUDIT.md
|
||||
2. docs/research/ALPHA025_BETA_VALIDATION.md
|
||||
3. docs/research/MULTISEED_BETA_VALIDATION.md
|
||||
4. docs/research/EXPERIMENT_PROTOCOL.md
|
||||
5. docs/baseline/BASELINE_VALIDATION.md
|
||||
|
||||
Critical rules:
|
||||
- planningawal/ is immutable. Do not edit/create/move/rename/delete anything inside it.
|
||||
- ~/datasets/X-IIoTID/X-IIoTID-dataset.csv and ~/datasets/5G-NIDD/5G-NIDD.csv are
|
||||
read-only external source data. Do not modify or copy them into the repo.
|
||||
- The S_j participant rule is a REPRODUCTION ASSUMPTION (material in 32.7% of
|
||||
class-condition cells). Carry this caveat in all reporting.
|
||||
- planningawal/ is immutable.
|
||||
- ~/datasets/X-IIoTID/X-IIoTID-dataset.csv and ~/datasets/5G-NIDD/5G-NIDD.csv are read-only.
|
||||
- The S_j participant rule is a REPRODUCTION ASSUMPTION (material in 19/27 cells at alpha=0.25).
|
||||
- Do NOT add an executable equal_available mode.
|
||||
- Do NOT launch the full alpha x seed x dataset x beta grid.
|
||||
- Beta selection uses validation metrics only; test never influences selection.
|
||||
- PRIMARY selection is pooled across clients and seeds; per-seed selection is
|
||||
diagnostic only.
|
||||
- Beta selection uses validation only; PRIMARY is pooled across clients and seeds.
|
||||
- Do NOT launch 5G-NIDD or the full grid.
|
||||
|
||||
Context: Session 6 completed X-IIoTID alpha=0.50 for seeds {0,1,2} across beta in
|
||||
{0,0.25,0.5,0.75,1}. Seed 0 was reused from Session 5. Pooled validation selected
|
||||
beta=0.75 (val per-client-mean rare macro-F1 0.579367); per-seed diagnostics were
|
||||
0.50/1.00/0.75. Pooled-selected test rare macro-F1 0.665503 +/- 0.026685 vs
|
||||
original_protean 0.642520 +/- 0.028683. beta=0 is mathematically but not bitwise equal
|
||||
to original_protean (float ordering), no semantic discrepancy. Between-seed variance is
|
||||
comparable to the between-beta spread, so the selection is weak. A strict independent
|
||||
audit (docs/research/SESSION6_AUDIT.md) returned PASS WITH MINOR CORRECTIONS and
|
||||
independently reproduced beta_star=0.75 from validation artifacts alone.
|
||||
Context: Session 7 completed X-IIoTID alpha=0.25 for seeds {0,1,2} (18 runs, ~12.17 h) and was
|
||||
independently audited (PASS WITH MINOR CORRECTIONS; docs/research/SESSION7_AUDIT.md). Pooled
|
||||
validation selected beta=0.00 (pooled mean 0.591704; monotone decreasing in beta), unlike
|
||||
alpha=0.50 which selected 0.75. Per-seed diagnostics 0.75/0.50/0.00. beta=0 equivalence class B
|
||||
(no semantic discrepancy). Realized partitions more heterogeneous (19/27 |S_j|<M vs 7/27).
|
||||
Condition-level pooled labels: alpha=0.50 -> 0.75, alpha=0.25 -> 0.00 (the value 0.50 is only the
|
||||
Session-5 seed-0 choice, not a condition-level selection).
|
||||
|
||||
Task: decide and document the alpha-expansion plan (recommended: alpha=0.25 then 0.75,
|
||||
seeds {0,1,2}, five betas each, sequential, reusing frozen manifests, pooled
|
||||
validation-only selection) OR argue for more seeds at alpha=0.50 first. Do not launch
|
||||
5G-NIDD or the full grid. Update docs and docs/handoff/SESSION_HANDOFF.md. Use
|
||||
.venv/bin/python. Do not commit or push unless asked.
|
||||
Task: run X-IIoTID alpha=0.75 for seeds {0,1,2} across the five frozen betas plus
|
||||
original_protean, sequential, reusing frozen manifests, pooled validation-only selection, then
|
||||
test evaluation, direct-effect, client-level, and cross-alpha comparison. Produce
|
||||
docs/research/ALPHA075_BETA_VALIDATION.md and update the handoff. Use .venv/bin/python. Do not
|
||||
commit or push unless asked.
|
||||
```
|
||||
@@ -0,0 +1,467 @@
|
||||
# Alpha-0.25 Beta Validation (Session 7)
|
||||
|
||||
Status: **three-seed paired protocol completed for a second non-IID condition**
|
||||
(X-IIoTID `alpha = 0.25`, seeds `{0,1,2}`) with pooled-across-seed validation-only beta
|
||||
selection, followed by a single final test evaluation of the pooled-selected beta. This is a
|
||||
**paired characterisation of a more heterogeneous non-IID setting**, not a sweep, not a
|
||||
hyper-parameter search over seeds, and not evidence of a global optimum.
|
||||
|
||||
Companion documents:
|
||||
|
||||
- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen protocol and selection hierarchy.
|
||||
- `docs/research/MULTISEED_BETA_VALIDATION.md` — Session-6 alpha=0.50 three-seed condition.
|
||||
- `docs/research/PAIRED_BETA_VALIDATION.md` — Session-5 single-seed condition.
|
||||
- `docs/research/SESSION6_AUDIT.md` — independent audit of the alpha=0.50 protocol.
|
||||
- `docs/research/BETA_AGGREGATION_IMPLEMENTATION.md` — beta mode implementation.
|
||||
- `docs/baseline/BASELINE_VALIDATION.md` — frozen baseline evidence.
|
||||
- `docs/handoff/SESSION_HANDOFF.md` — continuation point.
|
||||
|
||||
---
|
||||
|
||||
## A. Scope
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| dataset | `xiiotid` (X-IIoTID) |
|
||||
| alpha | 0.25 |
|
||||
| seeds | 0, 1, 2 |
|
||||
| aggregation | `original_protean` (per seed) + `support_weighted` for `beta ∈ {0, 0.25, 0.5, 0.75, 1}` |
|
||||
| runs | 18: 15 `support_weighted` + 3 `original_protean` |
|
||||
| new this session | all 18 runs (no alpha=0.25 run pre-existed) |
|
||||
| compute | sequential on CPU, one run at a time, ≈ 12.17 h wall-clock total |
|
||||
|
||||
No other alpha, dataset, seed, mode or beta was run. `equal_available` was not added.
|
||||
alpha=0.75 and 5G-NIDD were deliberately **not** run.
|
||||
|
||||
---
|
||||
|
||||
## B. Partition characterization (realized, before interpretation)
|
||||
|
||||
Manifests are reused, never regenerated. Each manifest was reloaded and independently
|
||||
reproduced from the raw dataset (`reproduce_train_assignments`: split hash, `N_ij` and
|
||||
partition hash all verified). All three seeds share train/validation/test sizes
|
||||
569,909 / 81,414 / 162,831 (10 clients, 9 classes, `Normal` benign).
|
||||
|
||||
| seed | manifest | content hash | partition hash | split hash | rng seed |
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | `artifacts/partitions/xiiotid__alpha0.25__seed0.json` | `839ba18e…` | `60faf299…` | `d0acc0c1…` | 1617371214 |
|
||||
| 1 | `artifacts/partitions/xiiotid__alpha0.25__seed1.json` | `84c20b4b…` | `6ffb10ae…` | `880a9f82…` | 1670706564 |
|
||||
| 2 | `artifacts/partitions/xiiotid__alpha0.25__seed2.json` | `d6e68ee1…` | `c8e1ad31…` | `8e0db0be…` | 795318573 |
|
||||
|
||||
Client-sample totals and class-support characterization (`N_ij`, training split). `CV` is the
|
||||
population coefficient of variation of the positive supports within a class; `max/min` is over
|
||||
contributing clients only.
|
||||
|
||||
### Seed 0 (`|S_j| < 10` in 6/9 classes)
|
||||
|
||||
| class | total | #clients | min+ | max | mean+ | CV | max/min |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| C&C | 2,004 | 10 | 21 | 1,065 | 200.4 | 1.478 | 50.7 |
|
||||
| Exfiltration | 15,494 | 8 | 4 | 7,950 | 1,936.8 | 1.405 | 1987.5 |
|
||||
| Exploitation | 793 | 6 | 2 | 595 | 132.2 | 1.580 | 297.5 |
|
||||
| Lateral_movement | 22,117 | 9 | 2 | 13,857 | 2,457.4 | 1.810 | 6928.5 |
|
||||
| Normal | 292,777 | 10 | 1 | 122,800 | 29,277.7 | 1.346 | 122800.0 |
|
||||
| RDOS | 98,883 | 10 | 166 | 31,089 | 9,888.3 | 1.080 | 187.3 |
|
||||
| Reconnaissance | 89,161 | 8 | 46 | 65,307 | 11,145.1 | 1.855 | 1419.7 |
|
||||
| Tampering | 3,586 | 8 | 9 | 2,272 | 448.2 | 1.569 | 252.4 |
|
||||
| Weaponization | 45,094 | 8 | 43 | 23,381 | 5,636.8 | 1.528 | 543.7 |
|
||||
|
||||
client-sample totals min 9,075 / max 141,358 (ratio 15.6).
|
||||
|
||||
### Seed 1 (`|S_j| < 10` in 7/9 classes)
|
||||
|
||||
| class | total | #clients | min+ | max | mean+ | CV | max/min |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| C&C | 2,004 | 9 | 1 | 986 | 222.7 | 1.559 | 986.0 |
|
||||
| Exfiltration | 15,494 | 10 | 1 | 6,204 | 1,549.4 | 1.238 | 6204.0 |
|
||||
| Exploitation | 793 | 8 | 1 | 321 | 99.1 | 1.295 | 321.0 |
|
||||
| Lateral_movement | 22,117 | 10 | 1 | 10,094 | 2,211.7 | 1.434 | 10094.0 |
|
||||
| Normal | 292,777 | 8 | 198 | 129,678 | 36,597.1 | 1.163 | 654.9 |
|
||||
| RDOS | 98,883 | 9 | 168 | 53,482 | 10,987.0 | 1.468 | 318.4 |
|
||||
| Reconnaissance | 89,161 | 8 | 138 | 65,101 | 11,145.1 | 1.852 | 471.8 |
|
||||
| Tampering | 3,586 | 9 | 2 | 1,804 | 398.4 | 1.348 | 902.0 |
|
||||
| Weaponization | 45,094 | 9 | 1 | 12,024 | 5,010.4 | 0.833 | 12024.0 |
|
||||
|
||||
client-sample totals min 9,278 / max 193,326 (ratio 20.8).
|
||||
|
||||
### Seed 2 (`|S_j| < 10` in 6/9 classes)
|
||||
|
||||
| class | total | #clients | min+ | max | mean+ | CV | max/min |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| C&C | 2,004 | 6 | 2 | 1,265 | 334.0 | 1.315 | 632.5 |
|
||||
| Exfiltration | 15,494 | 8 | 4 | 8,269 | 1,936.8 | 1.337 | 2067.3 |
|
||||
| Exploitation | 793 | 6 | 1 | 485 | 132.2 | 1.328 | 485.0 |
|
||||
| Lateral_movement | 22,117 | 7 | 1 | 12,496 | 3,159.6 | 1.475 | 12496.0 |
|
||||
| Normal | 292,777 | 9 | 452 | 118,937 | 32,530.8 | 1.104 | 263.1 |
|
||||
| RDOS | 98,883 | 10 | 2 | 48,839 | 9,888.3 | 1.421 | 24419.5 |
|
||||
| Reconnaissance | 89,161 | 10 | 1 | 46,541 | 8,916.1 | 1.518 | 46541.0 |
|
||||
| Tampering | 3,586 | 10 | 1 | 2,584 | 358.6 | 2.134 | 2584.0 |
|
||||
| Weaponization | 45,094 | 9 | 4 | 22,853 | 5,010.4 | 1.471 | 5713.3 |
|
||||
|
||||
client-sample totals min 15,664 / max 122,526 (ratio 7.8).
|
||||
|
||||
### Rare / unseen (training-derived, per client)
|
||||
|
||||
Every client has exactly two rare classes in every seed. Unseen classes are much more
|
||||
frequent than at alpha=0.50:
|
||||
|
||||
| seed | rare per client | union rare | unseen per client | union unseen |
|
||||
|---|---|---|---|---|
|
||||
| 0 | 2, 2, 2, 2, 2, 2, 2, 2, 2, 2 | 8 classes | 0, 3, 1, 2, 2, 1, 2, 0, 0, 2 | 6 classes |
|
||||
| 1 | 2 × 10 | 7 classes | 0, 0, 4, 1, 1, 0, 1, 0, 1, 0 | 6 classes |
|
||||
| 2 | 2 × 10 | 8 classes | 3, 0, 0, 2, 2, 2, 2, 0, 1, 2 | 5 classes |
|
||||
|
||||
### Comparison of realized heterogeneity (alpha=0.25 vs alpha=0.50)
|
||||
|
||||
| statistic | alpha=0.25 | alpha=0.50 |
|
||||
|---|---|---|
|
||||
| class-condition cells with `|S_j| < M` | **19 / 27** | 7 / 27 |
|
||||
| missing client-class cells | **38** | 9 |
|
||||
| client-sample-total max/min (per seed) | 15.6, 20.8, 7.8 | 5.1, 7.2, 9.9 |
|
||||
| union rare classes per seed | 8, 7, 8 | 6, 7, 7 |
|
||||
| union unseen classes per seed | 6, 6, 5 | 3, 3, 1 |
|
||||
| max unseen classes on a client | 4 | 2 |
|
||||
|
||||
The lower Dirichlet parameter does propagate to more missing classes, more unseen classes, and
|
||||
a larger client-size spread — but **not uniformly**: seed 2 at alpha=0.25 has a smaller
|
||||
client-size ratio (7.8) than alpha=0.50 seed 2 (9.9), and its `|S_j| < M` count is 6/9 versus
|
||||
1/9 at alpha=0.50. Interpretation uses the realized partitions, not alpha alone.
|
||||
|
||||
---
|
||||
|
||||
## C. Frozen protocol
|
||||
|
||||
Identical to `EXPERIMENT_PROTOCOL.md` §14 and to Session 6: 10 clients, 10 rounds, 3 local
|
||||
epochs, batch 256, Adam `lr=1e-3`, `mu=0.1`, `lambda=1.0`; unchanged architecture,
|
||||
preprocessing, train/validation/test split recipe, partition recipe, loss terms and inference.
|
||||
All 18 runs were performed this session with `scripts/run_baseline.py` at HEAD `6b97a0d`
|
||||
(no training/aggregation source changed).
|
||||
|
||||
### C.1 Manifest and build-level pairing
|
||||
|
||||
Within each seed all six runs share one manifest content hash and byte-identical deterministic
|
||||
artifacts:
|
||||
|
||||
| seed | manifest content hash | round-0 snapshot SHA-256 (all 6 runs) | support / characterization / rare_unseen |
|
||||
|---|---|---|---|
|
||||
| 0 | `839ba18e…` | `cd821beb…` | identical |
|
||||
| 1 | `84c20b4b…` | `59d2d2bc…` | identical |
|
||||
| 2 | `d6e68ee1…` | `47befeff…` | identical |
|
||||
|
||||
`manifest_reload_ok = true` and `smoke = false` in all 18 runs; metadata confirms
|
||||
`dataset=xiiotid`, `alpha=0.25`, seed, mode, beta, clients=10, rounds=10, local_epochs=3,
|
||||
batch=256, lr=1e-3, mu=0.1, lambda=1.0. Pairing is proven by content hashes, not filenames.
|
||||
|
||||
### C.2 Selection rule (unchanged)
|
||||
|
||||
- **PRIMARY:** pooled-across-seed validation per-client-mean rare-class macro-F1 (average the
|
||||
seed-level per-client mean over seeds `{0,1,2}`), selected with validation only.
|
||||
- **SECONDARY (diagnostic):** per-seed validation selection, for stability only.
|
||||
- **TEST:** read only after the pooled beta is fixed.
|
||||
|
||||
---
|
||||
|
||||
## D. Per-seed validation results
|
||||
|
||||
`val rareMacroF1 (pc mean)` is the primary metric; all 10 clients are eligible in every seed.
|
||||
Validation only.
|
||||
|
||||
### Seed 0
|
||||
|
||||
| beta | eligible | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | 10 | 0.616380 | 0.642939 | 0.841886 | 0.676032 | 0.874776 |
|
||||
| 0.25 | 10 | 0.609913 | 0.646423 | 0.834580 | 0.679089 | 0.867038 |
|
||||
| 0.50 | 10 | 0.595247 | 0.633677 | 0.807172 | 0.667755 | 0.853256 |
|
||||
| **0.75** | 10 | **0.654033** | **0.666477** | 0.797400 | **0.697353** | 0.854313 |
|
||||
| 1.00 | 10 | 0.578385 | 0.585755 | 0.710305 | 0.626018 | 0.847188 |
|
||||
|
||||
### Seed 1
|
||||
|
||||
| beta | eligible | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | 10 | 0.492425 | 0.580976 | 0.824292 | 0.665021 | 0.861707 |
|
||||
| 0.25 | 10 | 0.514237 | 0.605558 | 0.833188 | 0.684847 | 0.870391 |
|
||||
| **0.50** | 10 | **0.537040** | 0.647568 | 0.829322 | 0.718155 | 0.880917 |
|
||||
| 0.75 | 10 | 0.535672 | **0.655554** | 0.817017 | **0.723896** | 0.879210 |
|
||||
| 1.00 | 10 | 0.530942 | 0.652257 | 0.803940 | 0.721164 | 0.874346 |
|
||||
|
||||
### Seed 2
|
||||
|
||||
| beta | eligible | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **0.00** | 10 | **0.666307** | **0.673945** | 0.819363 | **0.703911** | **0.882895** |
|
||||
| 0.25 | 10 | 0.641922 | 0.655402 | 0.825502 | 0.687134 | 0.872024 |
|
||||
| 0.50 | 10 | 0.625267 | 0.636281 | 0.806218 | 0.670651 | 0.868462 |
|
||||
| 0.75 | 10 | 0.548685 | 0.558044 | 0.712391 | 0.601211 | 0.851058 |
|
||||
| 1.00 | 10 | 0.486970 | 0.499711 | 0.637821 | 0.549558 | 0.838517 |
|
||||
|
||||
Machine-readable: `artifacts/analysis/xiiotid__alpha0.25__multiseed__validation_per_seed.csv`.
|
||||
|
||||
---
|
||||
|
||||
## E. Per-seed diagnostic beta choices (SECONDARY, stability only)
|
||||
|
||||
| seed | selected beta | val rareMacroF1 (pc mean) | tie-break used | test_used_for_selection |
|
||||
|---|---|---|---|---|
|
||||
| 0 | 0.75 | 0.654033 | no | false |
|
||||
| 1 | 0.50 | 0.537040 | no | false |
|
||||
| 2 | 0.00 | 0.666307 | no | false |
|
||||
|
||||
The preferred beta is seed-dependent (0.75 / 0.50 / 0.00). These per-seed choices are
|
||||
**diagnostic only** and are not the condition-level selection rule.
|
||||
`val_selection.json` records `"selection_split": "validation"` and
|
||||
`"test_used_for_selection": false` for all 15 `support_weighted` runs.
|
||||
|
||||
---
|
||||
|
||||
## F. PRIMARY pooled-across-seed validation selection
|
||||
|
||||
| beta | pooled mean | std (sample, ddof=1) | std (pop) | min | max | pooled val rareRecall (pc mean) | selected |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| **0.00** | **0.591704** | 0.089529 | 0.073100 | 0.492425 | 0.666307 | 0.759482 | **yes** |
|
||||
| 0.25 | 0.588691 | 0.066435 | 0.054244 | 0.514237 | 0.641922 | 0.780771 | |
|
||||
| 0.50 | 0.585851 | 0.044858 | 0.036626 | 0.537040 | 0.625267 | 0.772536 | |
|
||||
| 0.75 | 0.579463 | 0.064906 | 0.052996 | 0.535672 | 0.654033 | 0.760791 | |
|
||||
| 1.00 | 0.532099 | 0.045718 | 0.037329 | 0.486970 | 0.578385 | 0.704575 | |
|
||||
|
||||
- **PRIMARY selected beta (`beta_star_alpha025`) = 0.00.**
|
||||
- **Criterion:** maximum pooled validation per-client-mean rare-class macro-F1.
|
||||
- **Tie-break used:** no (no exact tie; the pooled means separate by > 1e-3).
|
||||
- **`test_used_for_selection = false`.** Test metrics were not inspected until the pooled beta
|
||||
was fixed.
|
||||
- **Interpretation:** `beta = 0.00` is the *best among the tested candidates under pooled
|
||||
validation across the three project-defined seeds for X-IIoTID alpha=0.25.* It is **not** a
|
||||
global optimum. The pooled validation ordering is **monotonically decreasing in beta**.
|
||||
|
||||
Machine-readable: `artifacts/analysis/xiiotid__alpha0.25__multiseed__validation_pooled.csv`
|
||||
and `...__selection_summary.json`.
|
||||
|
||||
---
|
||||
|
||||
## G. Final selected-beta test metrics (descriptive only)
|
||||
|
||||
`beta_star = 0.00` was fixed from pooled validation; only then were test metrics read.
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.641362 | 0.580555 | 0.674489 | 0.632136 ± 0.047642 |
|
||||
| rare-class recall | 0.830701 | 0.825699 | 0.807370 | 0.821257 ± 0.012283 |
|
||||
| macro-F1 | 0.674558 | 0.665039 | 0.704433 | 0.681343 ± 0.020555 |
|
||||
| accuracy | 0.874551 | 0.863490 | 0.884002 | 0.874015 ± 0.010267 |
|
||||
| weighted-F1 | 0.906156 | 0.899864 | 0.914627 | 0.906882 ± 0.007408 |
|
||||
|
||||
---
|
||||
|
||||
## H. `original_protean` comparison
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.639810 | 0.582522 | 0.676039 | 0.632790 ± 0.047152 |
|
||||
| rare-class recall | 0.829601 | 0.826700 | 0.809595 | 0.821965 ± 0.010811 |
|
||||
| macro-F1 | 0.673103 | 0.666981 | 0.705969 | 0.682018 ± 0.020967 |
|
||||
| accuracy | 0.873722 | 0.867341 | 0.885231 | 0.875431 ± 0.009067 |
|
||||
| weighted-F1 | 0.905466 | 0.902075 | 0.915416 | 0.907652 ± 0.006934 |
|
||||
|
||||
Descriptive difference (`beta_star` − `original_protean`) for test rare-class macro-F1 is
|
||||
+0.001552 (seed 0), −0.001967 (seed 1), −0.001550 (seed 2); mean −0.000655. This is expected:
|
||||
`beta = 0` is mathematically the equal mean over `S_j` (see §I), so the end-to-end difference is
|
||||
floating-point trajectory noise, not a method effect. No superiority or generalisation is
|
||||
claimed; this is one dataset, one alpha, three project-defined seeds.
|
||||
|
||||
Test columns are descriptive only and are recorded in
|
||||
`artifacts/analysis/xiiotid__alpha0.25__multiseed__test_descriptive.csv`.
|
||||
|
||||
---
|
||||
|
||||
## I. `beta = 0` equivalence vs `original_protean`
|
||||
|
||||
For all three seeds, `support_weighted beta=0` vs frozen `original_protean`:
|
||||
|
||||
| property | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| same manifest / split / hashes | yes | yes | yes |
|
||||
| round-0 snapshot byte-identical | yes (`cd821beb…`) | yes (`59d2d2bc…`) | yes (`47befeff…`) |
|
||||
| `train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv` identical | yes | yes | yes |
|
||||
| independent aggregation diff (`ΣC/n` vs `(1/n)ΣC`) | 4.4e-16 | 4.4e-16 | 2.2e-16 |
|
||||
| history rare-macro-F1 round-0 diff | 0.0 | 0.0 | 0.0 |
|
||||
| history rare-macro-F1 max diff (rounds ≥ 1) | 6.6e-3 | 3.6e-3 | 3.7e-3 |
|
||||
| test metric max diff | 1.6e-3 | 3.9e-3 | 2.2e-3 |
|
||||
| **classification** | **B** | **B** | **B** |
|
||||
|
||||
Class **B**: semantic equivalence + floating-point numerical divergence; **no semantic
|
||||
discrepancy** in any seed. Round 0 is byte-identical; the ~1 ULP ordering difference enters the
|
||||
round-1 prototype-alignment training and the trajectories diverge slowly. Equivalence remains
|
||||
conditional on the shared `S_j` reproduction assumption and is **not** proof that `beta = 0`
|
||||
reproduces original PROTEAN.
|
||||
|
||||
---
|
||||
|
||||
## J. Direct prototype-effect findings (fixed round-0 snapshots, no retraining)
|
||||
|
||||
Computed with the production aggregation from each seed's fixed `{C_ij, N_ij}` round-0
|
||||
snapshot. `||Cbar_j(beta) - Cbar_j(0)||` at `beta = 1.0`, cross-seed mean (sample std):
|
||||
|
||||
| class | mean disp | std | min | max | mean rel disp | typical `|S_j|` |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Exfiltration | 4.316 | 0.898 | 3.588 | 5.319 | 0.664 | 8 |
|
||||
| Tampering | 4.248 | 0.241 | 3.978 | 4.441 | 0.965 | 8 |
|
||||
| C&C | 4.020 | 0.290 | 3.691 | 4.242 | 0.847 | 10 |
|
||||
| Lateral_movement | 3.726 | 1.383 | 2.342 | 5.109 | 0.790 | 9 |
|
||||
| Exploitation | 3.464 | 1.113 | 2.180 | 4.152 | 0.702 | 6 |
|
||||
| Reconnaissance | 3.339 | 1.107 | 2.089 | 4.195 | 0.640 | 8 |
|
||||
| Weaponization | 3.124 | 0.585 | 2.486 | 3.636 | 0.563 | 8 |
|
||||
| RDOS | 2.566 | 1.219 | 1.848 | 3.974 | 0.328 | 10 |
|
||||
| Normal | 1.997 | 0.533 | 1.667 | 2.611 | 0.370 | 10 |
|
||||
|
||||
Findings (descriptive):
|
||||
|
||||
- Every class moves for `beta > 0` in every seed, because every class has unequal support across
|
||||
its contributors.
|
||||
- Classes with the fewest contributors / strongest support imbalance (`Tampering`, `C&C`,
|
||||
`Exfiltration`, `Exploitation`) have the largest relative displacement; `Tampering` at
|
||||
`rel = 0.965` moves almost one full prototype norm.
|
||||
- `RDOS` and `Normal` move least in relative terms despite large absolute class sizes, because
|
||||
their support is spread more evenly across all 10 clients (`RDOS` CV ≈ 1.08–1.47).
|
||||
- Displacement is a **direct-aggregation measure only** and is not interpreted as improved or
|
||||
degraded detection. The end-to-end effect (§F/§G) is reported separately.
|
||||
|
||||
Outputs: `artifacts/analysis/xiiotid__alpha0.25__seed{0,1,2}__support_weighted__beta0__prototype_snapshot__prototype_effect.csv`
|
||||
and `...__prototype_weights.csv`; cross-seed aggregate
|
||||
`artifacts/analysis/xiiotid__alpha0.25__multiseed__prototype_effect_pooled.csv`.
|
||||
|
||||
---
|
||||
|
||||
## K. Client-level effect (validation, relative to beta = 0)
|
||||
|
||||
Full per-client table:
|
||||
`artifacts/analysis/xiiotid__alpha0.25__multiseed__client_validation_detail.csv`; summary:
|
||||
`...__client_validation_summary.csv`.
|
||||
|
||||
| seed | beta | improve | degrade | unchanged | mean Δ | min Δ | max Δ |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 0 | 0.25 | 2 | 5 | 3 | −0.0065 | −0.0429 | +0.0188 |
|
||||
| 0 | 0.50 | 2 | 7 | 1 | −0.0211 | −0.0769 | +0.0574 |
|
||||
| 0 | 0.75 | 5 | 4 | 1 | +0.0377 | −0.1117 | +0.3759 |
|
||||
| 0 | 1.00 | 4 | 5 | 1 | −0.0380 | −0.2570 | +0.1410 |
|
||||
| 1 | 0.25 | 7 | 1 | 2 | +0.0218 | −0.0064 | +0.0718 |
|
||||
| 1 | 0.50 | 7 | 2 | 1 | +0.0446 | −0.0158 | +0.2356 |
|
||||
| 1 | 0.75 | 5 | 4 | 1 | +0.0432 | −0.0183 | +0.3167 |
|
||||
| 1 | 1.00 | 5 | 4 | 1 | +0.0385 | −0.0387 | +0.3774 |
|
||||
| 2 | 0.25 | 5 | 3 | 2 | −0.0244 | −0.3029 | +0.0819 |
|
||||
| 2 | 0.50 | 4 | 4 | 2 | −0.0410 | −0.3398 | +0.1185 |
|
||||
| 2 | 0.75 | 3 | 5 | 2 | −0.1176 | −0.6011 | +0.1241 |
|
||||
| 2 | 1.00 | 3 | 6 | 1 | −0.1793 | −0.6775 | +0.0754 |
|
||||
|
||||
Descriptive observations:
|
||||
|
||||
- Unlike alpha=0.50 (where `beta = 0.75` improved the majority of clients in every seed), at
|
||||
alpha=0.25 the higher-beta response is **seed-dependent and often negative**: seeds 0 and 2
|
||||
degrade on average for `beta ≥ 0.25`, while seed 1 improves for `0.25–1.00`.
|
||||
- The magnitude of losses grows with beta: seed 2 loses up to −0.677 on one client at
|
||||
`beta = 1.00`. The largest single gains also grow with beta (seed 1, +0.377), i.e. beta
|
||||
redistributes sharply rather than uniformly.
|
||||
- Client ids are per-seed realisations and are not treated as one population. This analysis is
|
||||
descriptive and non-causal; `N_ij` vs ΔF1 remains non-monotonic (some low-support clients
|
||||
gain, some lose).
|
||||
|
||||
---
|
||||
|
||||
## L. Comparison with alpha=0.50
|
||||
|
||||
Validation-side evidence (selection basis); test is not used to explain the selection.
|
||||
|
||||
| beta | alpha=0.50 pooled mean (std) | alpha=0.25 pooled mean (std) |
|
||||
|---|---|---|
|
||||
| 0.00 | 0.544361 (0.091447) | **0.591704 (0.089529)** |
|
||||
| 0.25 | 0.555434 (0.090068) | 0.588691 (0.066435) |
|
||||
| 0.50 | 0.567322 (0.088539) | 0.585851 (0.044858) |
|
||||
| 0.75 | **0.579367 (0.075121)** | 0.579463 (0.064906) |
|
||||
| 1.00 | 0.573479 (0.100450) | 0.532099 (0.045718) |
|
||||
| selected | **0.75** | **0.00** |
|
||||
| pooled mean range | 0.544–0.579 (0.035) | 0.532–0.592 (0.060) |
|
||||
|
||||
> **Disambiguation (audited, Session 7).** The condition-level pooled selections are
|
||||
> alpha=0.50 → **0.75** (Session 6) and alpha=0.25 → **0.00** (Session 7). The value **0.50**
|
||||
> is *not* the alpha=0.50 condition-level selection: it is only the Session-5 **seed-0
|
||||
> single-condition** choice and the Session-6 alpha=0.50 **per-seed diagnostic** for seed 0.
|
||||
> Any phrasing of the form "0.50 → 0.00" that refers to condition-level pooled selection is
|
||||
> incorrect. See `docs/research/SESSION7_AUDIT.md` §M.
|
||||
|
||||
Observations:
|
||||
|
||||
- **The pooled validation-selected beta changes with non-IID severity**: 0.75 at alpha=0.50 vs
|
||||
**0.00** at alpha=0.25. At alpha=0.25 the pooled validation curve is monotonically decreasing
|
||||
in beta; at alpha=0.50 it rises to a peak at 0.75.
|
||||
- **Across-seed std does not shrink at the more heterogeneous setting**: 0.045–0.090
|
||||
(alpha=0.25) vs 0.075–0.100 (alpha=0.50). The selected beta has a lower across-seed std at
|
||||
alpha=0.25 (0.0895) but this is not the selection criterion (no tie).
|
||||
- **Realized partition differences accompany the beta change** (§B): alpha=0.25 has 19/27
|
||||
`|S_j| < M` cell conditions vs 7/27, more unseen classes, and larger per-class support
|
||||
imbalance, so up-weighting high-support clients redistributes the global prototypes further
|
||||
(mean `||Cbar(1)−Cbar(0)||` 3.42 vs 2.10; §M/§J) and is more often harmful on validation.
|
||||
- **Client-level pattern flips**: at alpha=0.50 higher beta helped most clients in every seed;
|
||||
at alpha=0.25 it helps mainly in seed 1 and hurts on average in seeds 0 and 2.
|
||||
|
||||
These are descriptive associations. The realized partition characteristics (not alpha alone)
|
||||
are reported alongside, and **no causal claim is made that alpha by itself caused the
|
||||
difference**. The beta effect remains small relative to between-seed variance at both settings.
|
||||
|
||||
---
|
||||
|
||||
## M. Runtime and resources
|
||||
|
||||
CPU only, no GPU, no swap, exit status 0 for every run (`/usr/bin/time -v`). Per-run
|
||||
wall-clock 39:49–42:12.
|
||||
|
||||
| seed | tag | wall | CPU % | peak RSS |
|
||||
|---|---|---|---|---|
|
||||
| 0 | `original_protean` | 40:10.80 | 746 | 8.38 GiB |
|
||||
| 0 | beta 0.00 | 40:57.24 | 744 | 8.38 GiB |
|
||||
| 0 | beta 0.25 | 40:27.96 | 745 | 8.38 GiB |
|
||||
| 0 | beta 0.50 | 40:21.77 | 743 | 8.39 GiB |
|
||||
| 0 | beta 0.75 | 39:49.26 | 746 | 8.38 GiB |
|
||||
| 0 | beta 1.00 | 40:25.86 | 745 | 8.38 GiB |
|
||||
| 1 | `original_protean` | 40:29.86 | 745 | 10.80 GiB |
|
||||
| 1 | beta 0.00 | 39:52.79 | 745 | 10.80 GiB |
|
||||
| 1 | beta 0.25 | 40:45.79 | 745 | 10.80 GiB |
|
||||
| 1 | beta 0.50 | 40:18.93 | 745 | 10.80 GiB |
|
||||
| 1 | beta 0.75 | 39:58.73 | 744 | 10.80 GiB |
|
||||
| 1 | beta 1.00 | 40:47.41 | 745 | 10.80 GiB |
|
||||
| 2 | `original_protean` | 40:13.48 | 745 | 7.51 GiB |
|
||||
| 2 | beta 0.00 | 40:24.04 | 746 | 7.51 GiB |
|
||||
| 2 | beta 0.25 | 41:45.12 | 743 | 7.51 GiB |
|
||||
| 2 | beta 0.50 | 40:22.35 | 744 | 7.51 GiB |
|
||||
| 2 | beta 0.75 | 42:11.66 | 741 | 7.51 GiB |
|
||||
| 2 | beta 1.00 | 40:35.98 | 744 | 7.51 GiB |
|
||||
|
||||
- New compute this session: **18 runs, ≈ 43,799 s ≈ 12.17 h wall-clock**, sequential on an
|
||||
8-core host (seed chains ≈ 4.04 / 4.04 / 4.09 h).
|
||||
- Peak RSS is partition-dependent, not beta-dependent: 8.38 GiB (seed 0), 10.80 GiB (seed 1),
|
||||
7.51 GiB (seed 2); all well within the 29 GiB host.
|
||||
- No warnings, no client failures, 9/9 global prototypes in every round, no retries.
|
||||
- Machine-readable: `artifacts/analysis/xiiotid__alpha0.25__multiseed__runtime.csv`.
|
||||
|
||||
---
|
||||
|
||||
## N. Limitations
|
||||
|
||||
- **One dataset** (X-IIoTID). No cross-dataset conclusion.
|
||||
- **Only alpha 0.25 and 0.50 evaluated so far.** alpha=0.75 remains unmeasured.
|
||||
- **Three project-defined seeds** (`{0,1,2}`); the pooled mean’s standard error is of the same
|
||||
order as the between-beta spread, so the selected beta is statistically weak.
|
||||
- The `S_j` participant/denominator rule
|
||||
(`S_j = {i | N_ij > 0 and valid C_ij}`, equal weights in the baseline) remains a
|
||||
**REPRODUCTION ASSUMPTION**. At alpha=0.25 it is material in **19/27 class-condition cells**.
|
||||
The `beta = 0` comparison is conditional on both modes sharing this rule and does **not**
|
||||
prove `beta = 0` reproduces original PROTEAN.
|
||||
- Train/validation/test adaptation (70/10/20 stratified) differs from the paper’s literal
|
||||
80/20; the validation eval partition is a project decision.
|
||||
- **No 5G-NIDD beta result.**
|
||||
- **No global beta optimum claim.** `beta = 0.00` is only the best among the five tested
|
||||
candidates under pooled validation for X-IIoTID alpha=0.25 across seeds `{0,1,2}`.
|
||||
- The union of rare classes (and hence global rare support) differs by seed and by alpha;
|
||||
only the per-client-mean metric is directly comparable within a condition. Cross-alpha
|
||||
comparisons in §L are descriptive, not controlled.
|
||||
- Test metrics of the selected beta are single-condition and must not be read as
|
||||
generalisation; no test metric influenced selection.
|
||||
@@ -0,0 +1,378 @@
|
||||
# Session 7 Post-Experiment Audit (X-IIoTID alpha=0.25)
|
||||
|
||||
Status: **PASS WITH MINOR CORRECTIONS.** Independent recomputation from machine-readable
|
||||
artifacts confirms every Session 7 claim. No discovered issue changes `beta_star` or the
|
||||
scientific interpretation. Corrections are documentation-only (cross-alpha disambiguation and
|
||||
one wording clarification). **No training was run, no aggregation/model source changed, no
|
||||
test-based selection occurred.**
|
||||
|
||||
Audit performed: 2026-09-26. Audited HEAD: `6b97a0d38b839c737c49f41006817503d0672871`.
|
||||
|
||||
Companion documents: `docs/research/ALPHA025_BETA_VALIDATION.md`,
|
||||
`docs/research/MULTISEED_BETA_VALIDATION.md`, `docs/research/SESSION6_AUDIT.md`,
|
||||
`docs/research/EXPERIMENT_PROTOCOL.md`, `docs/research/PAIRED_BETA_VALIDATION.md`,
|
||||
`docs/handoff/SESSION_HANDOFF.md`.
|
||||
|
||||
---
|
||||
|
||||
## A. Scope
|
||||
|
||||
Independently verify the Session 7 claim for:
|
||||
|
||||
- dataset `xiiotid`, alpha `0.25`, seeds `{0,1,2}`, beta `{0, 0.25, 0.5, 0.75, 1.0}` plus
|
||||
`original_protean` (18 runs);
|
||||
- PRIMARY pooled-validation selection `beta_star = 0.00`, using validation per-client
|
||||
rare-class macro-F1 averaged within seed then across seeds.
|
||||
|
||||
The audit recomputed metrics from run artifacts and re-derived the selection; it did not repeat
|
||||
the report, and it ran no training. No alpha=0.75 or 5G-NIDD run was launched.
|
||||
|
||||
---
|
||||
|
||||
## B. Run inventory
|
||||
|
||||
18 full-data runs were enumerated from `results/*/metadata.json` (15 `support_weighted` + 3
|
||||
`original_protean`). Every expected run is present, none duplicated, none partial, none
|
||||
unexpected.
|
||||
|
||||
| seed | runs present | metadata fields | manifest reload | smoke | missing artifacts |
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | original + beta {0,.25,.5,.75,1} | all correct | true | false | none |
|
||||
| 1 | original + beta {0,.25,.5,.75,1} | all correct | true | false | none |
|
||||
| 2 | original + beta {0,.25,.5,.75,1} | all correct | true | false | none |
|
||||
|
||||
Verified per run: dataset=xiiotid, alpha=0.25, seed, aggregation mode, beta, clients=10,
|
||||
rounds=10, local_epochs=3, batch=256, lr=1e-3, mu=0.1, lambda=1.0, `manifest_reload_ok=true`,
|
||||
`smoke=false`; every run has all 14 expected artifacts. No run exists for any alpha ≠ 0.25
|
||||
(alpha=0.50 runs are the Session-6 condition) and no `5gnidd...support_weighted` run exists
|
||||
(only `5gnidd__alpha0.5__seed0__original_protean`).
|
||||
|
||||
---
|
||||
|
||||
## C. Pairing / manifest integrity
|
||||
|
||||
Within each seed, all six runs share one manifest content hash and the deterministic artifacts
|
||||
are byte-identical (SHA-256 computed by this audit):
|
||||
|
||||
| seed | manifest content hash | round-0 snapshot SHA-256 (all 6 runs) | support / characterization / rare_unseen |
|
||||
|---|---|---|---|
|
||||
| 0 | `839ba18e2d87245d198be8f5f89d4a24f13cdb6fc64c004d406c6d4f8246f3da` | `cd821beb8684ed762934a00b9f541bdfae5a4498f061ffe07132ad9d00ad300c` | identical |
|
||||
| 1 | `84c20b4bd6cd20010fa35d139a0416cb06e79c82fbd8e1d57441cafdcdfbf15d` | `59d2d2bc1a86a858d061d08219ee84669177ce8346e388fc8eada299e6fb5cf5` | identical |
|
||||
| 2 | `d6e68ee13cb11f98cfb4aea668f3e905d3ebc94884632352c0045fc5f86df28c` | `47befefff4cfa84250d1b6783fe00ce98dcea83b0649efe9745fa70bd4f3fa33` | identical |
|
||||
|
||||
The stored `train_support_Nij.csv` equals the manifest support matrix, and each manifest was
|
||||
independently reproduced from the raw dataset (`prepare_dataset` →
|
||||
`reproduce_train_assignments`): all three seeds return `reproduce_ok=True`, `support_match=True`,
|
||||
`split_hash_ok=True`, `partition_hash_ok=True`, `nij_sum = 569,909`, 10 clients, split sizes
|
||||
569,909 / 81,414 / 162,831 for both alpha=0.25 and alpha=0.50. `history.json` and
|
||||
`test_global_metrics.json` differ across runs of a seed, exactly as expected because beta
|
||||
changes training. Pairing is established by content hashes, not filenames.
|
||||
|
||||
---
|
||||
|
||||
## D. Partition characterization audit (realized)
|
||||
|
||||
Independently recomputed from `artifacts/partitions/xiiotid__alpha{0.25,0.5}__seed{0,1,2}.json`
|
||||
(`support` matrix, training split). `M = 10` clients; `|S_j| < M` means fewer than 10 clients
|
||||
contribute a class prototype; "missing cells" counts `(client, class)` pairs with `N_ij = 0`
|
||||
over attack+benign classes.
|
||||
|
||||
| statistic | alpha=0.25 seed0 / seed1 / seed2 | alpha=0.50 seed0 / seed1 / seed2 |
|
||||
|---|---|---|
|
||||
| class-condition cells with `|S_j| < M` | 6 / 7 / 6 (sum **19/27**) | 3 / 3 / 1 (sum **7/27**) |
|
||||
| missing client-class cells | 13 / 10 / 15 (sum **38**) | 4 / 4 / 1 (sum **9**) |
|
||||
| client-size max/min | 15.58 / 20.84 / 7.82 | 5.11 / 7.22 / 9.89 |
|
||||
| union unseen classes | 6 / 6 / 5 | 3 / 3 / 1 |
|
||||
| max unseen on one client | 3 / 4 / 3 | 2 / 2 / 1 |
|
||||
| union rare classes | 8 / 7 / 8 | 6 / 7 / 7 |
|
||||
| rare classes per client | 2 × 10 | 2 × 10 |
|
||||
|
||||
These match the report claims exactly: alpha=0.25 = **19/27 cells**, **38 missing cells**,
|
||||
client-size ratios **15.6 / 20.8 / 7.8**, unseen union/count **6/6/5** (max 4). The comparison
|
||||
against alpha=0.50 is confirmed (7/27, 9, 5.1/7.2/9.9, 3/3/1, max 2). Per-class `N_ij`, CV and
|
||||
max/min recomputed from the manifests equal `characterization.csv` for every seed and class.
|
||||
Interpretation uses these realized statistics, not alpha alone; note the non-uniform case
|
||||
(alpha=0.25 seed 2 ratio 7.82 < alpha=0.50 seed 2 ratio 9.89).
|
||||
|
||||
---
|
||||
|
||||
## E. Validation metric recomputation
|
||||
|
||||
Recomputed from raw artifacts (no report numbers copied):
|
||||
|
||||
- seed-level primary score = mean over eligible clients of `val_per_client.csv`
|
||||
`rare_class_macro_f1`; **equals `val_selection.json` per-client-mean to ≤ 1.1e-16**;
|
||||
- global validation accuracy / macro-F1 / weighted-F1 / rare macro-F1 / rare recall
|
||||
recomputed **independently from `val_confusion_matrix.json`** using the production formulas;
|
||||
**equal to `val_global_metrics.json` exactly (diff 0.0)**.
|
||||
|
||||
| seed | beta | eligible | recomputed seed mean | report claim |
|
||||
|---|---|---|---|---|
|
||||
| 0 | 0.00 / 0.25 / 0.50 / 0.75 / 1.00 | 10 each | 0.616380 / 0.609913 / 0.595247 / 0.654033 / 0.578385 | same |
|
||||
| 1 | 0.00 / 0.25 / 0.50 / 0.75 / 1.00 | 10 each | 0.492425 / 0.514237 / 0.537040 / 0.535672 / 0.530942 | same |
|
||||
| 2 | 0.00 / 0.25 / 0.50 / 0.75 / 1.00 | 10 each | 0.666307 / 0.641922 / 0.625267 / 0.548685 / 0.486970 | same |
|
||||
|
||||
All recomputed values agree with the report to six-decimal formatting precision; no mismatch
|
||||
beyond precision was found. Per-seed diagnostic argmax reproduces **0.75 / 0.50 / 0.00**.
|
||||
|
||||
Limitation (unavoidable, documented): the *per-client predictions* cannot be recomputed because
|
||||
the trained model is not persisted and retraining is forbidden by the audit. The per-client
|
||||
metric is verified via (a) its defining source code
|
||||
(`evaluation/metrics.py::rare_class_metrics` over each client's training-derived rare ids),
|
||||
(b) exact equality of its mean with `val_selection.json`, and (c) exact equality of the global
|
||||
union-rare metric recomputed from the confusion matrix. This does not affect `beta_star`, which
|
||||
depends only on the seed-level means.
|
||||
|
||||
---
|
||||
|
||||
## F. Rare / unseen integrity
|
||||
|
||||
Per-client rare classes recomputed from training `N_ij` (two smallest positive-support attack
|
||||
classes, tie by class id) and compared to `rare_unseen.csv` flags across **all 18 runs**:
|
||||
|
||||
- **0 mismatches** (flag mismatches counted over every `(seed, mode, beta, client, class)` cell).
|
||||
- Benign class (`Normal`) never flagged rare/unseen; rare and unseen never overlap.
|
||||
- Unseen = attack class with `N_ij = 0`; union of training-derived rare classes equals
|
||||
`val_global_metrics.evaluated_rare_classes` **and**
|
||||
`test_global_metrics.evaluated_rare_classes` for every run — i.e. validation/test labels did
|
||||
**not** influence rare-class identity.
|
||||
- Eligible clients = **10 in every seed and run**; every client has exactly 2 rare classes; no
|
||||
client has only 1 attack class.
|
||||
- Union rare classes: seed 0 = `[0,1,2,3,5,6,7,8]`, seed 1 = `[0,1,2,3,6,7,8]`,
|
||||
seed 2 = `[0,1,2,3,5,6,7,8]` (class ids; `Normal` is benign).
|
||||
|
||||
---
|
||||
|
||||
## G. Pooled-selection recomputation
|
||||
|
||||
Computed from the independently recomputed seed-level scores (not from the report). Pooled
|
||||
score = mean of the three seed-level per-client means; std is sample (`ddof=1`).
|
||||
|
||||
| beta | pooled mean | sample std | pop std | min | max |
|
||||
|---|---|---|---|---|---|
|
||||
| **0.00** | **0.591703815** | **0.089528837** | 0.073099989 | 0.492424736 | 0.666306547 |
|
||||
| 0.25 | 0.588690626 | 0.066435256 | 0.054244160 | 0.514237103 | 0.641922010 |
|
||||
| 0.50 | 0.585851480 | 0.044857850 | 0.036626281 | 0.537039860 | 0.625267154 |
|
||||
| 0.75 | 0.579463399 | 0.064906057 | 0.052995574 | 0.535672367 | 0.654032964 |
|
||||
| 1.00 | 0.532098701 | 0.045718339 | 0.037328867 | 0.486969813 | 0.578384529 |
|
||||
|
||||
- Recompute **matches the report** (mean and sample std) to ≤ 4e-7 (formatting precision).
|
||||
- `argmax` of the pooled validation score = **0.00** (no tie; `tie_break_used=false`).
|
||||
- The pooled validation curve is **numerically monotonically decreasing** in beta:
|
||||
0.591704 > 0.588691 > 0.585851 > 0.579463 > 0.532099.
|
||||
- The secondary pooled validation rare-class recall peaks at beta=0.25, but recall is only a
|
||||
tie-break and is not reached; the primary metric governs.
|
||||
- **Std convention:** pooled std is sample (`ddof=1`), explicitly labelled `std (sample)` in
|
||||
the report; `std_pop` is also emitted. Consistent with Session 6; no convention changed.
|
||||
|
||||
---
|
||||
|
||||
## H. Validation / test leakage audit
|
||||
|
||||
- `run_baseline.py`: `union_rare` is derived from the **training** support matrix only
|
||||
(lines 189–190). The `val_selection` payload (lines 297–313) is built **only** from
|
||||
`val_rare`, `val_metrics`, `val_per_client_rows`. Test variables appear in a separate block
|
||||
and feed only test artifacts. `test_used_for_selection: false` and
|
||||
`selection_split: "validation"` are recorded for all 15 `support_weighted` runs.
|
||||
- `analyze_multiseed_beta.py`: the PRIMARY selection block (lines 176–208) consumes only
|
||||
`per_seed_rows`, which are built from `val_selection.json`. Tie-break uses pooled validation
|
||||
recall then validation std then smaller beta. `test_rows` are read in a later block (line 348+)
|
||||
and are never referenced by the selection logic or tie-break.
|
||||
- No code path sorts/filters candidates by a test field.
|
||||
|
||||
Conclusion: selection is **programmatic and validation-only**; the boolean flag is corroborated
|
||||
by the independent recomputation of `beta_star` from validation artifacts alone.
|
||||
`test_used_for_selection = false` is upheld.
|
||||
|
||||
---
|
||||
|
||||
## I. Final test recomputation (selected beta and original)
|
||||
|
||||
`beta_star = 0.00` was fixed from pooled validation; only then were test metrics read.
|
||||
Test metrics recomputed independently from `test_confusion_matrix.json` (diff vs the JSON = 0.0),
|
||||
then averaged across seeds with sample std:
|
||||
|
||||
| metric | recomputed mean ± std | report claim |
|
||||
|---|---|---|
|
||||
| rare-class macro-F1 | 0.632136 ± 0.047642 | 0.632136 ± 0.047642 |
|
||||
| rare-class recall | 0.821257 ± 0.012283 | 0.821257 ± 0.012283 |
|
||||
| macro-F1 | 0.681343 ± 0.020555 | 0.681343 ± 0.020555 |
|
||||
| accuracy | 0.874015 ± 0.010267 | 0.874015 ± 0.010267 |
|
||||
| weighted-F1 | 0.906882 ± 0.007408 | 0.906882 ± 0.007408 |
|
||||
|
||||
`original_protean` recomputed identically: rare macro-F1 0.632790 ± 0.047152; rare recall
|
||||
0.821965 ± 0.010811; macro-F1 0.682018 ± 0.020967; accuracy 0.875431 ± 0.009067;
|
||||
weighted-F1 0.907652 ± 0.006934 — all match the report. No superiority claim is made.
|
||||
|
||||
---
|
||||
|
||||
## J. beta = 0 equivalence audit
|
||||
|
||||
For seeds 0, 1 and 2, `original_protean` vs `support_weighted beta=0`:
|
||||
|
||||
| property | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| same manifest / split / hashes | yes | yes | yes |
|
||||
| round-0 snapshot byte-identical | yes (`cd821beb…`) | yes (`59d2d2bc…`) | yes (`47befeff…`) |
|
||||
| support/characterization/rare_unseen identical | yes | yes | yes |
|
||||
| independent aggregation diff (`ΣC/n` vs `(1/n)ΣC`) | 4.4e-16 | 4.4e-16 | 2.2e-16 |
|
||||
| history rare-macro-F1 round-0 diff | 0.0 | 0.0 | 0.0 |
|
||||
| history rare-macro-F1 max diff (rounds ≥ 1) | 6.6e-3 | 3.6e-3 | 3.7e-3 |
|
||||
| test metric max diff | 1.6e-3 | 3.9e-3 | 2.2e-3 |
|
||||
| classification | **B** | **B** | **B** |
|
||||
|
||||
Class **B** (semantically equivalent + floating-point numerical divergence); **no semantic
|
||||
discrepancy** in any seed. Round 0 is byte-identical; the ~1 ULP ordering difference enters the
|
||||
round-1 prototype-alignment training and trajectories diverge slowly. Equivalence remains
|
||||
conditional on the shared `S_j` reproduction assumption and is **not** proof that `beta = 0`
|
||||
reproduces original PROTEAN. Machine-readable:
|
||||
`artifacts/analysis/xiiotid__alpha0.25__multiseed__beta0_equivalence.csv`.
|
||||
|
||||
---
|
||||
|
||||
## K. Direct-effect audit
|
||||
|
||||
Recomputed independently from the fixed round-0 snapshots using the production aggregation
|
||||
(`direct_prototype.prototype_effect_rows`). `||Cbar_j(1) − Cbar_j(0)||`, cross-seed mean:
|
||||
|
||||
| class | alpha=0.25 mean (std) | alpha=0.50 mean (std) |
|
||||
|---|---|---|
|
||||
| Exfiltration | 4.316 (0.898) | 2.020 (0.502) |
|
||||
| Tampering | 4.248 (0.241) | 2.354 (1.165) |
|
||||
| C&C | 4.020 (0.290) | 1.694 (0.505) |
|
||||
| Lateral_movement | 3.726 (1.383) | 2.724 (1.035) |
|
||||
| Exploitation | 3.464 (1.113) | 2.344 (0.412) |
|
||||
| Reconnaissance | 3.339 (1.107) | 1.002 (0.261) |
|
||||
| Weaponization | 3.124 (0.585) | 2.561 (1.228) |
|
||||
| RDOS | 2.566 (1.219) | 2.631 (1.165) |
|
||||
| Normal | 1.997 (0.533) | 1.557 (0.548) |
|
||||
|
||||
- Mean across classes: alpha=0.25 **3.422**, alpha=0.50 **2.099** (report: 3.42 / 2.10).
|
||||
- alpha=0.25 largest classes: Exfiltration 4.316, Tampering 4.248, C&C 4.020; smallest:
|
||||
Normal 1.997, RDOS 2.566 — all match the report.
|
||||
- Per-`(class, beta)` displacement and normalised weights recomputed from snapshots match the
|
||||
per-seed `...__prototype_effect.csv` and pooled `...__prototype_effect_pooled.csv` **exactly**.
|
||||
- Displacement is a **direct-aggregation measure only**; no detection causality is inferred.
|
||||
The end-to-end effect is reported separately and is not used to interpret displacement.
|
||||
|
||||
---
|
||||
|
||||
## L. Client-level audit
|
||||
|
||||
Recomputed from `val_per_client.csv` deltas vs `beta = 0` (independent of the summary CSV):
|
||||
|
||||
| seed | beta=0.75 improve/degrade/unchanged | mean Δ | min Δ | max Δ |
|
||||
|---|---|---|---|---|
|
||||
| 0 | **5 / 4 / 1** | +0.0377 | −0.1117 | +0.3759 |
|
||||
| 1 | **5 / 4 / 1** | +0.0432 | −0.0183 | +0.3167 |
|
||||
| 2 | **3 / 5 / 2** | −0.1176 | −0.6011 | +0.1241 |
|
||||
|
||||
- At beta=1.00 seed 2 degrades 6/10 clients with a worst client Δ = **−0.6775** (≈ −0.678),
|
||||
while seed 1 still gains on average — all confirmed.
|
||||
- `client_validation_detail.csv` deltas match recomputation with max abs diff **0.0**.
|
||||
- `min positive attack N_ij` vs ΔF1 is confirmed **non-monotonic** at beta=0.75 (e.g. seed 2:
|
||||
N=1 → −0.601, N=2 → 0.000, N=4 → +0.124; seed 0: N=2 → −0.029 and +0.002). Client ids are
|
||||
per-seed realisations and are not pooled across seeds.
|
||||
|
||||
---
|
||||
|
||||
## M. Cross-alpha audit (critical)
|
||||
|
||||
Use only audited condition-level pooled results.
|
||||
|
||||
| | alpha=0.50 | alpha=0.25 |
|
||||
|---|---|---|
|
||||
| **pooled-selected beta** | **0.75** | **0.00** |
|
||||
| pooled validation curve | rises to a peak at 0.75 | numerically monotone decreasing |
|
||||
| pooled mean range | 0.544361–0.579367 (0.035) | 0.532099–0.591704 (0.060) |
|
||||
| pooled sample std range | 0.075121–0.100450 | 0.044858–0.089529 |
|
||||
| `|S_j| < M` cells | 7/27 | 19/27 |
|
||||
| missing client-class cells | 9 | 38 |
|
||||
| union unseen per seed | 3/3/1 (max 2) | 6/6/5 (max 4) |
|
||||
| mean `||Cbar(1)−Cbar(0)||` | 2.099 | 3.422 |
|
||||
| beta=0.75 client pattern | majority improve every seed | seed-dependent, often negative |
|
||||
|
||||
Alpha=0.50 pooled selection was independently reproduced from Session-6 artifacts
|
||||
(`results/xiiotid__alpha0.5__seed{0,1,2}__support_weighted__beta{0,.25,.5,.75,1}`):
|
||||
pooled means 0.544361 / 0.555434 / 0.567322 / **0.579367** / 0.573479; `argmax = 0.75`.
|
||||
|
||||
**Reporting-confusion check.** The audit searched the entire repository (docs, research notes,
|
||||
handoff) for any statement of the form "selected beta changes (0.50 -> 0.00)" or equivalent
|
||||
conflating the two conditions. **No such statement exists in the repository.** Every written
|
||||
occurrence is correct: alpha=0.50 pooled `beta_star = 0.75`; alpha=0.25 pooled
|
||||
`beta_star = 0.00`. The value `0.50` appears only as (a) the Session-5 **seed-0 single-condition**
|
||||
selection for alpha=0.50 and (b) the Session-6 **per-seed diagnostic** for alpha=0.50 seed 0.
|
||||
It is **not** the condition-level pooled selection for alpha=0.50. This disambiguation has been
|
||||
added explicitly to the report (§L) and handoff.
|
||||
|
||||
These are descriptive associations; the realized partition characteristics (not alpha alone) are
|
||||
reported alongside, and **no causal claim is made that alpha by itself caused the difference**.
|
||||
The beta effect remains small relative to between-seed variance at both settings.
|
||||
|
||||
---
|
||||
|
||||
## N. Artifact provenance
|
||||
|
||||
Every reported quantity traces to a machine-readable artifact or a deterministic recomputation.
|
||||
|
||||
| Reported quantity | Source artifact(s) | Derivation | Verified |
|
||||
|---|---|---|---|
|
||||
| run existence / config | `results/<run>/metadata.json` | direct read | yes |
|
||||
| manifest content/partition/split hashes | `artifacts/partitions/xiiotid__alpha0.25__seed{0,1,2}.json` | module hash + `reproduce_train_assignments` | yes |
|
||||
| pairing proof | `artifacts/analysis/...__prototype_snapshot.json` | sha256 across 6 runs/seed | yes |
|
||||
| seed-level validation score | `results/.../val_per_client.csv` + `val_selection.json` | mean of per-client rare macro-F1 | yes |
|
||||
| validation global metrics | `results/.../val_confusion_matrix.json` | independent numpy recompute | yes |
|
||||
| pooled mean/std, `beta_star` | recomputed seed scores | mean/`stdev` over seeds, argmax | yes |
|
||||
| test metrics (beta 0, original) | `results/.../test_confusion_matrix.json` | independent recompute + mean/std | yes |
|
||||
| test descriptive (all candidates) | `artifacts/analysis/...__test_descriptive.csv` | direct read | yes |
|
||||
| beta=0 equivalence | original vs beta0 artifacts/snapshots | sha256 + independent aggregation + history diff | yes |
|
||||
| direct-effect displacement | snapshots + `...__prototype_effect_pooled.csv` | independent `direct_prototype` recompute | yes |
|
||||
| client improve/degrade | `results/.../val_per_client.csv` + `...__client_validation_{summary,detail}.csv` | deltas vs beta0 | yes |
|
||||
| rare-class identity | `artifacts/analysis/...__rare_unseen.csv` + manifest | recompute definition | yes |
|
||||
| partition characterization | manifests + `results/.../characterization.csv` | independent numpy recompute | yes |
|
||||
| runtime / RSS | `artifacts/analysis/xiiotid__alpha0.25__multiseed__runtime.csv` (source `/tmp/opencode/s7_logs/*.time`, present) | parse logs; sum = 43,799.03 s = 12.166 h | yes |
|
||||
|
||||
No scientific number was found to exist only in Markdown without a machine-readable or
|
||||
deterministic source.
|
||||
|
||||
---
|
||||
|
||||
## O. Deviations / corrections found
|
||||
|
||||
1. **Cross-alpha label disambiguation (documentation-only).** The reported external phrasing
|
||||
"Selected beta changes (0.50 -> 0.00)" is **not present anywhere in the repository**; the
|
||||
repo already states alpha=0.50 pooled = 0.75 and alpha=0.25 pooled = 0.00. An explicit
|
||||
disambiguation note was added to `docs/research/ALPHA025_BETA_VALIDATION.md` §L and
|
||||
`docs/handoff/SESSION_HANDOFF.md` §11 so the Session-5 seed-0 `0.50` and Session-6 pooled
|
||||
`0.75` cannot be conflated. No scientific value changed.
|
||||
2. **Wording nuance (clarified).** Handoff §14 item 6 ("the beta direction may reverse under
|
||||
stronger heterogeneity … monotone in heterogeneity") is a speculation, not a settled trend;
|
||||
reworded to a neutral "one more condition; not yet a trend across alpha" framing.
|
||||
3. **No arithmetic, metric-label, std-convention, or provenance defect was found** in the
|
||||
Session 7 report or analysis artifacts. No derived CSV required correction.
|
||||
|
||||
No training outputs, aggregation, model, manifests, or scientific protocol were altered.
|
||||
|
||||
---
|
||||
|
||||
## P. Final audit verdict
|
||||
|
||||
**PASS WITH MINOR CORRECTIONS.**
|
||||
|
||||
- Run inventory: **PASS** (18/18, no duplicates/partials; no alpha=0.75 or 5G-NIDD run).
|
||||
- Pairing / manifest reproduction: **PASS** (hash-identical; raw-data reproduction exact).
|
||||
- Partition characterization: **PASS** (19/27, 38, 15.6/20.8/7.8, 6/6/5; alpha=0.50 7/27, 9).
|
||||
- Validation recomputation (seed-level and global): **PASS** (exact).
|
||||
- Rare/unseen integrity: **PASS** (0 mismatches; training-derived; no val/test influence).
|
||||
- Pooled selection: **PASS**; independent `beta_star = 0.00` using validation only.
|
||||
- Validation/test leakage: **PASS**; selection reads validation only and is reproducible.
|
||||
- Test summaries (selected beta and `original_protean`): **PASS** (exact).
|
||||
- beta=0 equivalence: **PASS**, class **B** (semantically equivalent + float divergence).
|
||||
- Direct-effect and client-level analyses: **PASS** (independent recompute exact).
|
||||
- Cross-alpha comparison: **PASS**; alpha=0.50 pooled = 0.75, alpha=0.25 pooled = 0.00.
|
||||
- Source/data integrity, pytest (75 passed, 0 failed, 0 skipped, 0 warnings): **PASS**.
|
||||
|
||||
The corrections are documentation/wording only and do not affect `beta_star` or the scientific
|
||||
interpretation. Session 7 is **safe to checkpoint** with the documented limitations (single
|
||||
dataset, two alphas, three project-defined seeds, weak effect relative to seed variance, `S_j`
|
||||
reproduction assumption, no global-optimum claim).
|
||||
Reference in new issue
Block a user