experiment: validate beta selection at alpha 0.25
This commit is contained in:
willtanoe committed
1 parent
6b97a0d38b
commit
ff68d380e2
22 files changed
+33471
-209
No files matched your search
+253
-209
@@ -3,25 +3,27 @@
|
||||
This is the canonical continuation point for the next agent session. It is written to be
|
||||
readable without prior chat history, terminal scrollback, or git history.
|
||||
|
||||
Last updated: 2026-09-25 (Session 6 + post-experiment audit). Session 6 type:
|
||||
**protocol-scaling / stability experiment**. The single Session-5 condition was extended
|
||||
from seed 0 to seeds `{0,1,2}`; beta was selected by the **pooled-across-seed validation**
|
||||
rule and the selected beta evaluated on test. **No full grid was launched; no
|
||||
alpha/dataset expansion was run.** A strict independent audit
|
||||
(`docs/research/SESSION6_AUDIT.md`) returned **PASS WITH MINOR CORRECTIONS**; `beta_star =
|
||||
0.75` was independently reproduced from validation artifacts.
|
||||
Last updated: 2026-09-26 (Session 7, post-experiment audit complete). Session 7 type:
|
||||
**protocol-scaling / non-IID-severity experiment**. Session 6 completed X-IIoTID `alpha=0.50`
|
||||
for seeds `{0,1,2}`. Session 7 completed X-IIoTID **`alpha=0.25`** for seeds `{0,1,2}` across
|
||||
the five frozen beta candidates plus `original_protean`, using the **same pooled-across-seed
|
||||
validation-only** selection rule. **alpha=0.75 and 5G-NIDD were not run.** No commit or push
|
||||
was made.
|
||||
|
||||
> **Audit summary (Session 6).** Verdict: **PASS WITH MINOR CORRECTIONS**. Independently
|
||||
> recomputed `beta_star = 0.75` from `val_per_client.csv`/`val_selection.json`; all 15
|
||||
> seed×beta seed-level scores and the pooled mean/sample-std match the report exactly.
|
||||
> Validation-only selection confirmed in code (`analyze_multiseed_beta.py` reads only
|
||||
> `val_selection.json`; test read in a later block); `test_used_for_selection=false`
|
||||
> upheld. Rare-class flags match the training-derived definition for all 30 cells;
|
||||
> manifests reproduce from raw data with disjoint splits. beta=0 vs `original_protean` =
|
||||
> semantically equivalent + floating-point divergence (class B). Minor corrections:
|
||||
> persisted `artifacts/analysis/xiiotid__alpha0.5__multiseed__runtime.csv` (runtime
|
||||
> provenance) and clarified the client improve/degrade counts (unchanged/tie clients).
|
||||
> **Session 6 is safe to checkpoint.**
|
||||
> **Audit status (Session 7).** Independent audit verdict: **PASS WITH MINOR CORRECTIONS**
|
||||
> (`docs/research/SESSION7_AUDIT.md`). `beta_star(alpha=0.25) = 0.00` confirmed from validation
|
||||
> only; leakage audit PASS; `beta=0` class **B**; source/data integrity PASS; pytest 75 passed,
|
||||
> 0 failed, 0 skipped, 0 warnings. Corrections were documentation-only (cross-alpha label
|
||||
> disambiguation + one wording nuance); no `beta_star` or scientific interpretation changed.
|
||||
> **Session 7 is safe to checkpoint.**
|
||||
|
||||
> **Headline (Session 7).** PRIMARY pooled-validation selected beta for X-IIoTID
|
||||
> `alpha=0.25` = **0.00** (validation only). The pooled validation curve is monotonically
|
||||
> **decreasing** in beta at alpha=0.25, whereas at alpha=0.50 it peaked at 0.75. Per-seed
|
||||
> diagnostics were 0.75 / 0.50 / 0.00. beta=0 is class **B** (semantically equivalent +
|
||||
> floating-point divergence) for all three seeds; no semantic discrepancy. The realized
|
||||
> alpha=0.25 partitions are materially more heterogeneous (19/27 class-condition cells with
|
||||
> `|S_j| < M` vs 7/27 at alpha=0.50). Effect remains small relative to between-seed variance.
|
||||
|
||||
---
|
||||
|
||||
@@ -29,8 +31,8 @@ alpha/dataset expansion was run.** A strict independent audit
|
||||
|
||||
Build the code and experiment infrastructure for a thesis study that modifies **only the
|
||||
server-side prototype aggregation** of PROTEAN (Chennoufi et al., 2025), a federated
|
||||
intrusion-detection framework. The planned mechanism weights each contributing client's
|
||||
class prototype by its class sample support:
|
||||
intrusion-detection framework. The mechanism weights each contributing client's class
|
||||
prototype by its class sample support:
|
||||
|
||||
```
|
||||
Cbar_j(beta) = ( sum_{i in S_j} N_ij^beta * C_ij ) / ( sum_{i in S_j} N_ij^beta )
|
||||
@@ -55,303 +57,345 @@ bitwise identical to the baseline (see §7).
|
||||
- Frozen baseline (`original_protean`) source unchanged; no `equal_available` mode added;
|
||||
beta candidate set unchanged (`{0, .25, .5, .75, 1}`).
|
||||
|
||||
## 3. Session 6 objective and outcome
|
||||
## 3. Session 7 objective and outcome
|
||||
|
||||
Objective: extend the validated paired protocol from one seed to seeds `{0,1,2}` for
|
||||
X-IIoTID alpha=0.50, reuse the seed-0 Session-5 results, run the two new seeds across the
|
||||
five beta candidates plus `original_protean`, select beta with **pooled validation only**,
|
||||
evaluate the selected beta on test, and produce cross-seed direct-effect / client-level /
|
||||
stability analyses + docs. **All acceptance items were met.** No commit or push was made.
|
||||
Objective: evaluate one additional non-IID severity (X-IIoTID `alpha=0.25`, seeds `{0,1,2}`)
|
||||
across the five beta candidates using the exact Session-6 three-seed pooled-validation
|
||||
protocol, and test whether the pooled-selected beta changes under a more heterogeneous
|
||||
condition. All 18 runs (15 `support_weighted` + 3 `original_protean`) completed with rc=0.
|
||||
Report: `docs/research/ALPHA025_BETA_VALIDATION.md`.
|
||||
|
||||
Outcome: **PRIMARY pooled-validation selected beta = 0.75** (validation only). Per-seed
|
||||
diagnostic selections varied (0.50 / 1.00 / 0.75), and between-seed variance is comparable
|
||||
to the between-beta spread, so the selection is arithmetically clear but statistically
|
||||
weak. The protocol ran cleanly end to end.
|
||||
Outcome: **PRIMARY pooled-validation selected beta = 0.00** (validation only);
|
||||
`test_used_for_selection = false` for all 15 support_weighted runs. Pooled means: 0.00 →
|
||||
0.591704, 0.25 → 0.588691, 0.50 → 0.585851, 0.75 → 0.579463, 1.00 → 0.532099 (monotone
|
||||
decreasing). Between-seed std 0.045–0.090; the effect is small relative to seed variance.
|
||||
|
||||
## 4. Condition and pairing
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| dataset / alpha / seeds | `xiiotid` / 0.50 / `{0,1,2}` |
|
||||
| runs | 15 `support_weighted` + 3 `original_protean` (seed 0 reused; 12 run this session) |
|
||||
| dataset / alpha / seeds | `xiiotid` / 0.25 / `{0,1,2}` |
|
||||
| runs | 15 `support_weighted` + 3 `original_protean`, all new this session |
|
||||
| training | 10 clients, 10 rounds, 3 local epochs, batch 256, lr 1e-3, mu 0.1, lambda 1.0, Adam |
|
||||
| code revision | unchanged from HEAD `50f2da72166e4f609077d65b4916f745da5e8e3b`; only read-only analysis code added |
|
||||
| code revision | unchanged from HEAD `6b97a0d38b839c737c49f41006817503d0672871`; only docs/analysis artifacts added |
|
||||
|
||||
Manifests (reused, never regenerated; one content hash per seed across all six runs):
|
||||
Manifests (reused, never regenerated; one content hash per seed across all six runs; each was
|
||||
reproduced from the raw dataset with split/`N_ij`/partition hashes verified):
|
||||
|
||||
| seed | manifest | content hash | partition hash | split hash | rng seed |
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | `artifacts/partitions/xiiotid__alpha0.5__seed0.json` | `e81a6538…` | `2465485539…` | `d0acc0c1…` | 2009062324 |
|
||||
| 1 | `artifacts/partitions/xiiotid__alpha0.5__seed1.json` | `54ea1df1…` | `4061c5a1…` | `880a9f82…` | 621679668 |
|
||||
| 2 | `artifacts/partitions/xiiotid__alpha0.5__seed2.json` | `be881ac6…` | `1118e7fa…` | `8e0db0be…` | 1438483982 |
|
||||
| 0 | `artifacts/partitions/xiiotid__alpha0.25__seed0.json` | `839ba18e…` | `60faf299…` | `d0acc0c1…` | 1617371214 |
|
||||
| 1 | `artifacts/partitions/xiiotid__alpha0.25__seed1.json` | `84c20b4b…` | `6ffb10ae…` | `880a9f82…` | 1670706564 |
|
||||
| 2 | `artifacts/partitions/xiiotid__alpha0.25__seed2.json` | `d6e68ee1…` | `c8e1ad31…` | `8e0db0be…` | 795318573 |
|
||||
|
||||
Pairing proof: round-0 snapshot byte-identical across the five betas **and**
|
||||
`original_protean` within each seed — seed 0 `e27f41b3a62eb83c…`, seed 1
|
||||
`b7f61c995e97474c…`, seed 2 `5b749f4185e11fc7…`. `train_support_Nij.csv`,
|
||||
`characterization.csv`, `rare_unseen.csv` identical across each seed's runs.
|
||||
`manifest_reload_ok = true` everywhere.
|
||||
`original_protean` within each seed — seed 0 `cd821beb…`, seed 1 `59d2d2bc…`, seed 2
|
||||
`47befeff…`. `train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv` identical
|
||||
across each seed's runs. `manifest_reload_ok = true`, `smoke = false` everywhere.
|
||||
|
||||
## 5. Validation results (USED FOR SELECTION) and pooled selection
|
||||
|
||||
Primary metric: validation per-client-mean rare-class macro-F1. All 10 clients eligible
|
||||
in every seed. Full per-seed table:
|
||||
`artifacts/analysis/xiiotid__alpha0.5__multiseed__validation_per_seed.csv`.
|
||||
Primary metric: validation per-client-mean rare-class macro-F1. All 10 clients eligible in
|
||||
every seed. Full tables: `artifacts/analysis/xiiotid__alpha0.25__multiseed__validation_per_seed.csv`.
|
||||
|
||||
Per-seed validation per-client-mean rare-class macro-F1:
|
||||
|
||||
| beta | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| 0.00 | 0.557340 | 0.628624 | 0.447118 |
|
||||
| 0.25 | 0.571787 | 0.636205 | 0.458310 |
|
||||
| 0.50 | 0.585762 | 0.645189 | 0.471015 |
|
||||
| 0.75 | 0.578003 | 0.655160 | 0.504937 |
|
||||
| 1.00 | 0.551443 | 0.683117 | 0.485877 |
|
||||
| 0.00 | 0.616380 | 0.492425 | 0.666307 |
|
||||
| 0.25 | 0.609913 | 0.514237 | 0.641922 |
|
||||
| 0.50 | 0.595247 | 0.537040 | 0.625267 |
|
||||
| 0.75 | 0.654033 | 0.535672 | 0.548685 |
|
||||
| 1.00 | 0.578385 | 0.530942 | 0.486970 |
|
||||
|
||||
Pooled across seeds (`...__validation_pooled.csv`):
|
||||
|
||||
| beta | pooled mean | std (sample) | min | max | selected |
|
||||
|---|---|---|---|---|---|
|
||||
| 0.00 | 0.544361 | 0.091447 | 0.447118 | 0.628624 | |
|
||||
| 0.25 | 0.555434 | 0.090068 | 0.458310 | 0.636205 | |
|
||||
| 0.50 | 0.567322 | 0.088539 | 0.471015 | 0.645189 | |
|
||||
| **0.75** | **0.579367** | **0.075121** | 0.504937 | 0.655160 | **yes** |
|
||||
| 1.00 | 0.573479 | 0.100450 | 0.485877 | 0.683117 | |
|
||||
| **0.00** | **0.591704** | 0.089529 | 0.492425 | 0.666307 | **yes** |
|
||||
| 0.25 | 0.588691 | 0.066435 | 0.514237 | 0.641922 | |
|
||||
| 0.50 | 0.585851 | 0.044858 | 0.537040 | 0.625267 | |
|
||||
| 0.75 | 0.579463 | 0.064906 | 0.535672 | 0.654033 | |
|
||||
| 1.00 | 0.532099 | 0.045718 | 0.486970 | 0.578385 | |
|
||||
|
||||
- **PRIMARY selected beta (`beta_star`) = 0.75**, max pooled validation per-client-mean
|
||||
rare-class macro-F1; **no tie-break used**; `test_used_for_selection = false` for all 15
|
||||
runs. The secondary pooled recall is highest at beta=0.25 (recall is only a tie-break).
|
||||
- **SECONDARY per-seed diagnostics only:** seed 0 → 0.50, seed 1 → 1.00, seed 2 → 0.75.
|
||||
These are stability diagnostics, not the condition's selection rule.
|
||||
- **PRIMARY selected beta (`beta_star_alpha025`) = 0.00**, max pooled validation
|
||||
per-client-mean rare-class macro-F1; **no tie-break used**; selection is validation-only.
|
||||
- **SECONDARY per-seed diagnostics only:** seed 0 → 0.75, seed 1 → 0.50, seed 2 → 0.00.
|
||||
|
||||
## 6. Final test evaluation of the pooled-selected beta
|
||||
|
||||
`beta_star = 0.75` was fixed from pooled validation before test was read.
|
||||
`beta_star = 0.00` was fixed from pooled validation before test was read.
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std |
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.636189 | 0.671937 | 0.688381 | 0.665503 ± 0.026685 |
|
||||
| rare-class recall | 0.857543 | 0.851883 | 0.878321 | 0.862583 ± 0.013921 |
|
||||
| macro-F1 | 0.726298 | 0.729015 | 0.738194 | 0.731169 ± 0.006234 |
|
||||
| accuracy | 0.888670 | 0.925493 | 0.905012 | 0.906392 ± 0.018450 |
|
||||
| weighted-F1 | 0.922278 | 0.944734 | 0.933884 | 0.933632 ± 0.011230 |
|
||||
| rare-class macro-F1 | 0.641362 | 0.580555 | 0.674489 | 0.632136 ± 0.047642 |
|
||||
| rare-class recall | 0.830701 | 0.825699 | 0.807370 | 0.821257 ± 0.012283 |
|
||||
| macro-F1 | 0.674558 | 0.665039 | 0.704433 | 0.681343 ± 0.020555 |
|
||||
| accuracy | 0.874551 | 0.863490 | 0.884002 | 0.874015 ± 0.010267 |
|
||||
| weighted-F1 | 0.906156 | 0.899864 | 0.914627 | 0.906882 ± 0.007408 |
|
||||
|
||||
`original_protean` for the same seeds/condition:
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std |
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.609620 | 0.655665 | 0.662275 | 0.642520 ± 0.028683 |
|
||||
| rare-class recall | 0.871487 | 0.889365 | 0.870018 | 0.876957 ± 0.010771 |
|
||||
| macro-F1 | 0.707701 | 0.709176 | 0.710017 | 0.708965 ± 0.001172 |
|
||||
| accuracy | 0.890432 | 0.900885 | 0.907045 | 0.899454 ± 0.008398 |
|
||||
| weighted-F1 | 0.917585 | 0.929132 | 0.927481 | 0.924733 ± 0.006245 |
|
||||
| rare-class macro-F1 | 0.639810 | 0.582522 | 0.676039 | 0.632790 ± 0.047152 |
|
||||
| rare-class recall | 0.829601 | 0.826700 | 0.809595 | 0.821965 ± 0.010811 |
|
||||
| macro-F1 | 0.673103 | 0.666981 | 0.705969 | 0.682018 ± 0.020967 |
|
||||
| accuracy | 0.873722 | 0.867341 | 0.885231 | 0.875431 ± 0.009067 |
|
||||
| weighted-F1 | 0.905466 | 0.902075 | 0.915416 | 0.907652 ± 0.006934 |
|
||||
|
||||
Test columns are **descriptive only** and played no part in selection. No superiority or
|
||||
generalisation is claimed: single dataset, single alpha, three project-defined seeds, and
|
||||
between-seed spread ~ between-method difference.
|
||||
Descriptive difference for test rare-class macro-F1 (`beta_star` − `original_protean`) is
|
||||
+0.001552 / −0.001967 / −0.001550 (mean −0.000655) — i.e. essentially identical, as expected
|
||||
because `beta = 0` is mathematically the equal mean over `S_j`. Test is descriptive only; no
|
||||
superiority or generalisation is claimed.
|
||||
|
||||
## 7. `beta = 0` equivalence result (seeds 1 and 2)
|
||||
## 7. `beta = 0` equivalence result (all three seeds)
|
||||
|
||||
For seeds 1 and 2, `support_weighted beta=0` vs frozen `original_protean`:
|
||||
For seeds 0, 1, 2, `support_weighted beta=0` vs frozen `original_protean`:
|
||||
|
||||
- Same manifest/split/hashes: **yes**.
|
||||
- Round-0 snapshot byte-identical: **yes** (seed 1 `b7f61c99…`, seed 2 `5b749f41…`).
|
||||
- Round-0 snapshot byte-identical: **yes** (`cd821beb…`, `59d2d2bc…`, `47befeff…`).
|
||||
- `train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv`: **identical**.
|
||||
- Aggregation: **mathematically equivalent, not bitwise** (`Σ(C/count)` vs
|
||||
`Σ((1/count)·C)`, ~1 ULP ordering difference that feeds back through round≥1).
|
||||
- Test/history/per-client/per-class: **differ slightly**.
|
||||
- Classification: **mathematically equivalent + floating-point numerical divergence;
|
||||
no semantic discrepancy** in either seed. Same result as Session 5 seed 0.
|
||||
- Independent aggregation (`ΣC/n` vs `(1/n)ΣC`): ≤ 4.4e-16 (1 ULP ordering).
|
||||
- Test/history/per-client/per-class: **differ slightly** (history rare-macro-F1 max diff
|
||||
3.6e-3–6.6e-3; test metric max diff 1.6e-3–3.9e-3).
|
||||
- Classification: **B — mathematically equivalent + floating-point numerical divergence; no
|
||||
semantic discrepancy** in any seed. Machine-readable:
|
||||
`artifacts/analysis/xiiotid__alpha0.25__multiseed__beta0_equivalence.csv`.
|
||||
|
||||
Conditional on the shared `S_j` reproduction assumption; not proof `beta=0` reproduces
|
||||
original PROTEAN.
|
||||
|
||||
## 8. Direct prototype-effect findings (fixed round-0 snapshots, no retraining)
|
||||
## 8. Partition characterization (realized)
|
||||
|
||||
`||Cbar_j(1) - Cbar_j(0)||`, cross-seed mean (std):
|
||||
`|S_j| < M` class-condition cells and missing client-class cells (training split):
|
||||
|
||||
- Largest means: `Lateral_movement` 2.724 (1.035), `RDOS` 2.631 (1.165),
|
||||
`Weaponization` 2.561 (1.228), `Tampering` 2.354 (1.165), `Exploitation` 2.344 (0.412).
|
||||
- Smallest: `Reconnaissance` 1.002 (0.261), `Normal` 1.557 (0.548), `C&C` 1.694 (0.505).
|
||||
- Every class moves for beta>0 (all classes have unequal support); the **rank order is
|
||||
seed-sensitive** (seed 0 top = Weaponization, seed 1 top = RDOS, seed 2 top =
|
||||
Lateral_movement). Displacement is a direct-aggregation measure only, not a detection
|
||||
result. Outputs: `...__prototype_effect_pooled.csv` and the per-seed
|
||||
`...__prototype_snapshot__prototype_effect.csv` / `..._weights.csv`.
|
||||
| condition | `|S_j| < M` cells | missing client-class cells | client-size max/min (seeds) |
|
||||
|---|---|---|---|
|
||||
| alpha=0.25 | **19 / 27** | **38** | 15.6, 20.8, 7.8 |
|
||||
| alpha=0.50 | 7 / 27 | 9 | 5.1, 7.2, 9.9 |
|
||||
|
||||
## 9. Client-level findings (validation, vs beta = 0)
|
||||
Union unseen classes per seed: alpha=0.25 = 6/6/5 (max 4 on a client); alpha=0.50 = 3/3/1
|
||||
(max 2). Union rare classes per seed: alpha=0.25 = 8/7/8; alpha=0.50 = 6/7/7. All clients have
|
||||
exactly two rare classes at both alphas. Full per-class `N_ij`, CV and max/min in
|
||||
`docs/research/ALPHA025_BETA_VALIDATION.md` §B. The lower alpha does **not** uniformly increase
|
||||
every realized statistic (e.g. alpha=0.25 seed 2 client-size ratio 7.8 < alpha=0.50 seed 2's
|
||||
9.9), so interpretation uses the realized partitions, not alpha alone.
|
||||
|
||||
Full detail: `...__multiseed__client_validation_detail.csv`; summary:
|
||||
## 9. Direct prototype-effect findings (fixed round-0 snapshots, no retraining)
|
||||
|
||||
`||Cbar_j(1) - Cbar_j(0)||`, cross-seed mean (std), alpha=0.25 vs alpha=0.50:
|
||||
|
||||
| class | a25 mean (std) | a50 mean (std) |
|
||||
|---|---|---|
|
||||
| Exfiltration | 4.316 (0.898) | 2.020 (0.502) |
|
||||
| Tampering | 4.248 (0.241) | 2.354 (1.165) |
|
||||
| C&C | 4.020 (0.290) | 1.694 (0.505) |
|
||||
| Lateral_movement | 3.726 (1.383) | 2.724 (1.035) |
|
||||
| Exploitation | 3.464 (1.113) | 2.344 (0.412) |
|
||||
| Reconnaissance | 3.339 (1.107) | 1.002 (0.261) |
|
||||
| Weaponization | 3.124 (0.585) | 2.561 (1.228) |
|
||||
| RDOS | 2.566 (1.219) | 2.631 (1.165) |
|
||||
| Normal | 1.997 (0.533) | 1.557 (0.548) |
|
||||
|
||||
Mean displacement across classes is larger at alpha=0.25 (3.42) than alpha=0.50 (2.10), because
|
||||
support is more imbalanced and more classes are missing contributors. Displacement is a
|
||||
direct-aggregation measure only, not a detection result. Outputs:
|
||||
`artifacts/analysis/xiiotid__alpha0.25__multiseed__prototype_effect_pooled.csv` and per-seed
|
||||
`...__seed{0,1,2}__support_weighted__beta0__prototype_snapshot__prototype_effect.csv`.
|
||||
|
||||
## 10. Client-level findings (validation, vs beta = 0)
|
||||
|
||||
Detail: `...__multiseed__client_validation_detail.csv`; summary:
|
||||
`...__client_validation_summary.csv`.
|
||||
|
||||
- At `beta = 0.75`, improve/degrade/**unchanged** counts are 7/3/0 (seed 0), 8/1/1
|
||||
(seed 1, tie = client 7), 8/1/1 (seed 2, tie = client 4); mean Δ is positive in all
|
||||
three (+0.021 / +0.027 / +0.058).
|
||||
- At `beta = 1.00`, seed 0 flips to 4 improve / 6 degrade / 0 unchanged, while seed 1
|
||||
improves 9/10 with 1 unchanged (client 7) — the high-beta response is seed-dependent.
|
||||
- `min positive attack N_ij` vs ΔF1 is **non-monotonic**: N=1 clients range −0.021…+0.041
|
||||
at beta=0.75; the largest gain (+0.177) is an N=10 client; low support neither
|
||||
systematically benefits nor degrades. Client ids are different realisations across
|
||||
seeds and are not treated as one population. Descriptive only.
|
||||
- At alpha=0.25 the higher-beta response is **seed-dependent and often negative**:
|
||||
improve/degrade/unchanged at beta=0.75 = 5/4/1 (seed 0, mean Δ +0.038), 5/4/1 (seed 1,
|
||||
+0.043), 3/5/2 (seed 2, −0.118). At beta=1.00 seed 2 degrades 6/10 clients with a worst
|
||||
client Δ = −0.678, while seed 1 still gains on average.
|
||||
- This **contrasts with alpha=0.50**, where beta=0.75 improved the majority of clients in every
|
||||
seed. Client ids are per-seed realisations and are not one population; the analysis is
|
||||
descriptive and non-causal, and `N_ij` vs ΔF1 remains non-monotonic.
|
||||
|
||||
## 10. Runtime / resources
|
||||
## 11. Comparison with alpha=0.50 (RQ3)
|
||||
|
||||
CPU only, no GPU, no swap, exit 0 on all 18 runs (15 new-session + reused). Per-run
|
||||
wall-clock 40:23–42:21; CPU 733–745 %; peak RSS 6.21 GiB (seed 0), 7.48 GiB (seed 1),
|
||||
6.45 GiB (seed 2). New compute this session: 12 runs, ≈ 8 h 11 m wall-clock, run
|
||||
sequentially (8 cores; concurrent runs would not reduce total CPU work).
|
||||
Seed-0 numbers were reused and not re-run. Machine-readable per-run values:
|
||||
`artifacts/analysis/xiiotid__alpha0.5__multiseed__runtime.csv` (added by the audit;
|
||||
seed 0 sourced from Session-5 docs, seeds 1–2 from `/usr/bin/time -v` logs).
|
||||
| | alpha=0.50 | alpha=0.25 |
|
||||
|---|---|---|
|
||||
| pooled-selected beta | 0.75 | **0.00** |
|
||||
| pooled validation curve | rises to 0.75 | monotone decreasing |
|
||||
| pooled mean range | 0.544–0.579 | 0.532–0.592 |
|
||||
| pooled std range | 0.075–0.100 | 0.045–0.090 |
|
||||
| `|S_j| < M` cells | 7/27 | 19/27 |
|
||||
| mean `||Cbar(1)−Cbar(0)||` | 2.10 | 3.42 |
|
||||
|
||||
## 11. Methodological decisions / assumptions
|
||||
The selection **changes** with non-IID severity. This is documented as a descriptive
|
||||
association accompanied by the realized partition characteristics; **no causal claim is made
|
||||
that alpha by itself caused the difference**. The effect is small relative to between-seed
|
||||
variance at both settings.
|
||||
|
||||
> **Audited label (do not confuse).** Condition-level **pooled** selections are alpha=0.50 →
|
||||
> **0.75** and alpha=0.25 → **0.00**. The value **0.50** is only the Session-5 **seed-0
|
||||
> single-condition** choice and the Session-6 alpha=0.50 **per-seed diagnostic** for seed 0;
|
||||
> it is not the alpha=0.50 pooled selection. Any "0.50 → 0.00" phrasing for condition-level
|
||||
> pooled selection is wrong. Verified in `docs/research/SESSION7_AUDIT.md` §M.
|
||||
|
||||
## 12. Runtime / resources
|
||||
|
||||
CPU only, no GPU, no swap, exit 0 on all 18 runs. Per-run wall-clock 39:49–42:12; CPU
|
||||
741–746 %; peak RSS 8.38 GiB (seed 0), 10.80 GiB (seed 1), 7.51 GiB (seed 2). New compute:
|
||||
18 runs, ≈ 43,799 s ≈ **12.17 h** wall-clock, sequential (8 cores). Machine-readable:
|
||||
`artifacts/analysis/xiiotid__alpha0.25__multiseed__runtime.csv`.
|
||||
|
||||
## 13. Methodological decisions / assumptions
|
||||
|
||||
1. **`S_j` participant/denominator rule** (`S_j = {i | N_ij > 0 and valid C_ij}`, equal
|
||||
weights in the baseline) is a **REPRODUCTION ASSUMPTION**; material in 53/162
|
||||
class-condition cells (32.7 %), 15/18 conditions (`BASELINE_VALIDATION.md` §D).
|
||||
Carry this caveat in all reporting.
|
||||
weights in the baseline) is a **REPRODUCTION ASSUMPTION**; material in 19/27
|
||||
class-condition cells at alpha=0.25 (7/27 at alpha=0.50). Carry this caveat in all reporting.
|
||||
2. `N_ij` is class-specific sample support, not prototype quality.
|
||||
3. Beta is one value for all classes/clients; candidates frozen at `{0,.25,.5,.75,1}`.
|
||||
4. **PRIMARY selection = pooled validation across clients and seeds**; per-seed selection
|
||||
is secondary stability diagnostics; test used only after the pooled beta is fixed.
|
||||
Documented in `EXPERIMENT_PROTOCOL.md` §8 (updated this session).
|
||||
5. The union of rare classes (and global rare support) differs by seed because partitions
|
||||
differ; only the per-client-mean metric is directly comparable across seeds.
|
||||
6. `lambda=1.0`, Adam, lr 1e-3, batch 256 remain PROJECT DECISIONS. No `equal_available`
|
||||
mode was added.
|
||||
4. **PRIMARY selection = pooled validation across clients and seeds**; per-seed selection is
|
||||
secondary stability diagnostics; test used only after the pooled beta is fixed.
|
||||
5. The `scripts/analyze_multiseed_beta.py` already supported `--alpha` (default 0.5) and
|
||||
already includes alpha in output names; **no code change was needed**. alpha=0.50 outputs
|
||||
were re-run and are byte-identical to the Session-6 artifacts (reproducibility verified).
|
||||
6. `lambda=1.0`, Adam, lr 1e-3, batch 256 remain PROJECT DECISIONS. No `equal_available` mode.
|
||||
|
||||
## 12. Unresolved questions / blockers
|
||||
## 14. Unresolved questions / blockers
|
||||
|
||||
1. `S_j` reproduction ambiguity — unresolved; no official PROTEAN code exists.
|
||||
2. **Pooled beta selection is statistically weak here:** pooled means span only
|
||||
0.544→0.579 while within-beta across-seed std is 0.075–0.100; three seeds give a
|
||||
standard error of the same order as the between-beta spread. The beta effect is small
|
||||
relative to seed variance.
|
||||
3. `beta = 0` is not bitwise identical to `original_protean`; documentation decision on how
|
||||
to report this remains open.
|
||||
2. **Pooled beta selection remains statistically weak at both alphas (descriptive only):**
|
||||
the across-seed sample std (0.045–0.090) is comparable to, or larger than, the total
|
||||
between-beta pooled-mean spread (0.035 at alpha=0.50, 0.060 at alpha=0.25). No inferential
|
||||
test was performed, so this is a descriptive comparison, not evidence of insignificance.
|
||||
3. `beta = 0` is not bitwise identical to `original_protean`; documentation decision on how to
|
||||
report this remains open.
|
||||
4. Session-2 manifests still store `preprocessing_hash = null` / `config_hash = null`.
|
||||
5. Only alpha=0.50, X-IIoTID, three seeds measured.
|
||||
5. Only alpha=0.25 and 0.50, X-IIoTID, three seeds measured. No 5G-NIDD beta result.
|
||||
6. The alpha=0.25 result (pooled-selected beta = 0) is **one additional condition, not yet a
|
||||
trend across alpha**. Whether any relationship between non-IID severity and the selected beta
|
||||
is monotone would require alpha=0.75 (and is still not implied by two conditions).
|
||||
|
||||
**Audit status (see `docs/research/SESSION6_AUDIT.md`).** No unresolved issue that could
|
||||
change `beta_star` or the scientific interpretation. One inherent audit limitation: the
|
||||
trained model is not persisted, so per-client prediction metrics cannot be recomputed
|
||||
without retraining (forbidden); they were verified via defining source code, exact
|
||||
agreement of their mean with `val_selection.json`, and exact agreement of the global
|
||||
union-rare metric recomputed from `val_confusion_matrix.json`. Pooled std uses the
|
||||
**sample** convention (ddof=1), labelled "(sample)" in the report.
|
||||
## 15. Git status
|
||||
|
||||
## 13. Git status
|
||||
|
||||
No commit or push was made. `git status --short` (results/ is ignored):
|
||||
No commit or push was made. `git status --short` shows only untracked docs/analysis/run
|
||||
artifacts (`results/` is gitignored; `artifacts/analysis/` is intentionally tracked):
|
||||
|
||||
```
|
||||
M docs/handoff/SESSION_HANDOFF.md
|
||||
M docs/research/EXPERIMENT_PROTOCOL.md
|
||||
?? docs/research/MULTISEED_BETA_VALIDATION.md
|
||||
?? docs/research/SESSION6_AUDIT.md
|
||||
?? scripts/analyze_multiseed_beta.py
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__multiseed__*.csv / __selection_summary.json
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__seed{1,2}__*__prototype_snapshot.json
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__seed{1,2}__support_weighted__beta0__prototype_snapshot__{prototype_effect,prototype_weights}.csv
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__seed0__support_weighted__beta{0.25,0.5,0.75,1}__prototype_snapshot.json (pre-existing from Session 5)
|
||||
?? docs/research/ALPHA025_BETA_VALIDATION.md
|
||||
?? docs/research/SESSION7_AUDIT.md
|
||||
?? artifacts/analysis/xiiotid__alpha0.25__* (runs' snapshots, effect CSVs, multiseed tables)
|
||||
?? (pre-existing Session-5/6 untracked alpha=0.5 views, unchanged)
|
||||
```
|
||||
|
||||
`planningawal/` and both raw datasets are unchanged; `results/` is gitignored.
|
||||
`HEAD = 50f2da72166e4f609077d65b4916f745da5e8e3b`. No commit or push was made.
|
||||
`planningawal/` and both raw datasets are unchanged (audit re-verified SHA-256:
|
||||
X-IIoTID `7b929005…`, 5G-NIDD `fa36f808…`). No model/training/aggregation source changed
|
||||
(working-tree blobs equal HEAD). `artifacts/partitions/` unmodified. No alpha=0.75 run and no
|
||||
5G-NIDD `support_weighted` run exists. `equal_available` absent.
|
||||
`HEAD = 6b97a0d38b839c737c49f41006817503d0672871`. No commit or push was made.
|
||||
|
||||
## 14. Files created / modified this session
|
||||
## 15b. Session 7 audit result (independent)
|
||||
|
||||
- Verdict: **PASS WITH MINOR CORRECTIONS** (`docs/research/SESSION7_AUDIT.md`).
|
||||
- Run inventory 18/18, pairing hash-identical, manifests reproduced from raw data.
|
||||
- Validation recomputed exactly; **pooled `beta_star(alpha=0.25) = 0.00`** (validation only,
|
||||
monotone decreasing in beta). `beta_star` test and `original_protean` test means/stds exact.
|
||||
- Leakage audit: selection reads validation only; `test_used_for_selection=false` upheld.
|
||||
- `beta=0` equivalence: class **B** for all three seeds (no semantic discrepancy).
|
||||
- Source/data integrity: PASS. pytest: **75 passed, 0 failed, 0 skipped, 0 warnings**.
|
||||
- Corrections: cross-alpha label disambiguation (documentation-only) and one wording nuance.
|
||||
No arithmetic, metric-label, std-convention, or provenance defect was found.
|
||||
|
||||
## 16. Files created / modified this session
|
||||
|
||||
Created:
|
||||
|
||||
- `docs/research/MULTISEED_BETA_VALIDATION.md`
|
||||
- `docs/research/SESSION6_AUDIT.md` (post-experiment audit)
|
||||
- `scripts/analyze_multiseed_beta.py`
|
||||
- `artifacts/analysis/xiiotid__alpha0.5__multiseed__*` (validation per-seed/pooled,
|
||||
client summary/detail/deltas, prototype-effect pooled, test descriptive, selection
|
||||
summary, and `...__runtime.csv` added by the audit)
|
||||
- seed-1/seed-2 `results/...` run dirs (gitignored) and their snapshots / effect CSVs
|
||||
- `docs/research/ALPHA025_BETA_VALIDATION.md`
|
||||
- `docs/research/SESSION7_AUDIT.md` (independent post-experiment audit)
|
||||
- `artifacts/analysis/xiiotid__alpha0.25__seed{0,1,2}__*__prototype_snapshot.json`
|
||||
- `artifacts/analysis/xiiotid__alpha0.25__seed{0,1,2}__support_weighted__beta0__prototype_snapshot__{prototype_effect,prototype_weights}.csv`
|
||||
- `artifacts/analysis/xiiotid__alpha0.25__multiseed__{validation_per_seed,validation_pooled,client_validation_deltas,client_validation_detail,client_validation_summary,prototype_effect_pooled,test_descriptive,runtime,beta0_equivalence}.csv` and `...__selection_summary.json`
|
||||
- `results/xiiotid__alpha0.25__seed{0,1,2}__*` (18 run dirs, gitignored)
|
||||
|
||||
Modified:
|
||||
|
||||
- `docs/research/EXPERIMENT_PROTOCOL.md` (§8 selection hierarchy clarified)
|
||||
- `docs/handoff/SESSION_HANDOFF.md` (this file)
|
||||
|
||||
Reused (not regenerated): all seed-0 Session-5 runs and artifacts.
|
||||
No source or test code was changed. `scripts/analyze_multiseed_beta.py` was re-run for
|
||||
alpha=0.50 and produced byte-identical outputs.
|
||||
|
||||
## 15. Exact recommended next phase
|
||||
## 17. Exact recommended next phase
|
||||
|
||||
**Session 6 is safe to checkpoint** (audit PASS WITH MINOR CORRECTIONS; no issue affects
|
||||
`beta_star`). The protocol is **mechanically** ready to scale, but Session 6 shows the beta
|
||||
effect at alpha=0.50 is small relative to seed-to-seed variance. Recommended, in order:
|
||||
1. **Session 7 is audited and safe to checkpoint** (verdict PASS WITH MINOR CORRECTIONS;
|
||||
see §15b and `docs/research/SESSION7_AUDIT.md`). alpha=0.25 behaved cleanly: pairing valid,
|
||||
beta=0 class B, pooled selection clean, artifacts complete, no test leakage, runtime
|
||||
manageable (≈12 h per alpha). Before checkpointing, stage only the intended docs/analysis
|
||||
artifacts; do not commit `results/` or the raw datasets.
|
||||
2. **Next experiment:** run X-IIoTID `alpha=0.75` under the same protocol (seeds `{0,1,2}`,
|
||||
five betas + `original_protean`, sequential, reusing frozen manifests, pooled
|
||||
validation-only selection), then test evaluation, direct-effect, client-level, and
|
||||
cross-alpha comparison. This would complete the three planned non-IID severities. Do **not**
|
||||
infer a monotone alpha→beta relation from two conditions.
|
||||
3. Do **not** treat `beta = 0.00` (alpha=0.25) or `0.75` (alpha=0.50) as optima. Report each as
|
||||
"best among tested candidates under pooled validation for that condition".
|
||||
4. **Do not launch 5G-NIDD or the full grid** until alpha=0.75 is inspected.
|
||||
5. Use `.venv/bin/python`. Run one alpha at a time; measure runtime as before.
|
||||
6. Update `docs/research/ALPHA025_BETA_VALIDATION.md` (or add `ALPHA075_BETA_VALIDATION.md`),
|
||||
`EXPERIMENT_PROTOCOL.md` only if a real ambiguity appears, and this handoff. Do not commit or
|
||||
push unless asked.
|
||||
|
||||
1. **Do not treat beta=0.75 as a settled optimum.** Report it as "best among tested
|
||||
candidates under pooled validation for X-IIoTID alpha=0.50".
|
||||
2. Before expanding alpha, decide the interpretation question: is the goal to characterise
|
||||
small beta effects against seed noise (requires more seeds), or to run alpha=0.25/0.75
|
||||
as a paired characterisation (accepting the same weak-effect caveat)?
|
||||
3. If expanding, run alpha=0.25 then alpha=0.75 for seeds `{0,1,2}`, the five betas each,
|
||||
reusing frozen manifests, sequentially (~8 h per alpha), with the same pooled
|
||||
validation-only selection. Extend `scripts/analyze_multiseed_beta.py` with `--alpha`.
|
||||
4. Do **not** launch 5G-NIDD or the full grid until the alpha results are inspected.
|
||||
|
||||
## 16. Important commands
|
||||
## 18. Important commands
|
||||
|
||||
```bash
|
||||
# Tests (75 expected)
|
||||
.venv/bin/python -m pytest
|
||||
|
||||
# One full paired beta run (CPU, ~41 min, ~6.5-7.5 GiB)
|
||||
.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.5 --seed 1 \
|
||||
# One full paired beta run (CPU, ~40 min, ~7.5-11.3 GiB)
|
||||
.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.25 --seed 1 \
|
||||
--clients 10 --rounds 10 --local-epochs 3 \
|
||||
--aggregation-mode support_weighted --beta 0.5
|
||||
|
||||
# Pooled multi-seed validation analysis (read-only, no retraining)
|
||||
.venv/bin/python scripts/analyze_multiseed_beta.py --dataset xiiotid --alpha 0.5 \
|
||||
--seeds 0 1 2
|
||||
# Pooled multi-seed validation analysis (read-only, no retraining; --alpha already supported)
|
||||
.venv/bin/python scripts/analyze_multiseed_beta.py --dataset xiiotid --alpha 0.25 --seeds 0 1 2
|
||||
|
||||
# Direct prototype-effect analysis (no retraining)
|
||||
.venv/bin/python scripts/analyze_prototype_effect.py \
|
||||
--snapshot artifacts/analysis/xiiotid__alpha0.5__seed1__support_weighted__beta0__prototype_snapshot.json \
|
||||
--snapshot artifacts/analysis/xiiotid__alpha0.25__seed0__support_weighted__beta0__prototype_snapshot.json \
|
||||
--class-names "C&C,Exfiltration,Exploitation,Lateral_movement,Normal,RDOS,Reconnaissance,Tampering,Weaponization" \
|
||||
--output-dir artifacts/analysis
|
||||
```
|
||||
|
||||
## 17. Recommended next-session prompt
|
||||
## 19. Recommended next-session prompt
|
||||
|
||||
```
|
||||
You are continuing the perdalam-protean research repository (Session 7).
|
||||
You are continuing the perdalam-protean research repository (Session 8).
|
||||
|
||||
FIRST, read completely:
|
||||
1. docs/handoff/SESSION_HANDOFF.md
|
||||
2. docs/research/MULTISEED_BETA_VALIDATION.md
|
||||
3. docs/research/SESSION6_AUDIT.md
|
||||
2. docs/research/ALPHA025_BETA_VALIDATION.md
|
||||
3. docs/research/MULTISEED_BETA_VALIDATION.md
|
||||
4. docs/research/EXPERIMENT_PROTOCOL.md
|
||||
5. docs/baseline/BASELINE_VALIDATION.md
|
||||
|
||||
Critical rules:
|
||||
- planningawal/ is immutable. Do not edit/create/move/rename/delete anything inside it.
|
||||
- ~/datasets/X-IIoTID/X-IIoTID-dataset.csv and ~/datasets/5G-NIDD/5G-NIDD.csv are
|
||||
read-only external source data. Do not modify or copy them into the repo.
|
||||
- The S_j participant rule is a REPRODUCTION ASSUMPTION (material in 32.7% of
|
||||
class-condition cells). Carry this caveat in all reporting.
|
||||
- planningawal/ is immutable.
|
||||
- ~/datasets/X-IIoTID/X-IIoTID-dataset.csv and ~/datasets/5G-NIDD/5G-NIDD.csv are read-only.
|
||||
- The S_j participant rule is a REPRODUCTION ASSUMPTION (material in 19/27 cells at alpha=0.25).
|
||||
- Do NOT add an executable equal_available mode.
|
||||
- Do NOT launch the full alpha x seed x dataset x beta grid.
|
||||
- Beta selection uses validation metrics only; test never influences selection.
|
||||
- PRIMARY selection is pooled across clients and seeds; per-seed selection is
|
||||
diagnostic only.
|
||||
- Beta selection uses validation only; PRIMARY is pooled across clients and seeds.
|
||||
- Do NOT launch 5G-NIDD or the full grid.
|
||||
|
||||
Context: Session 6 completed X-IIoTID alpha=0.50 for seeds {0,1,2} across beta in
|
||||
{0,0.25,0.5,0.75,1}. Seed 0 was reused from Session 5. Pooled validation selected
|
||||
beta=0.75 (val per-client-mean rare macro-F1 0.579367); per-seed diagnostics were
|
||||
0.50/1.00/0.75. Pooled-selected test rare macro-F1 0.665503 +/- 0.026685 vs
|
||||
original_protean 0.642520 +/- 0.028683. beta=0 is mathematically but not bitwise equal
|
||||
to original_protean (float ordering), no semantic discrepancy. Between-seed variance is
|
||||
comparable to the between-beta spread, so the selection is weak. A strict independent
|
||||
audit (docs/research/SESSION6_AUDIT.md) returned PASS WITH MINOR CORRECTIONS and
|
||||
independently reproduced beta_star=0.75 from validation artifacts alone.
|
||||
Context: Session 7 completed X-IIoTID alpha=0.25 for seeds {0,1,2} (18 runs, ~12.17 h) and was
|
||||
independently audited (PASS WITH MINOR CORRECTIONS; docs/research/SESSION7_AUDIT.md). Pooled
|
||||
validation selected beta=0.00 (pooled mean 0.591704; monotone decreasing in beta), unlike
|
||||
alpha=0.50 which selected 0.75. Per-seed diagnostics 0.75/0.50/0.00. beta=0 equivalence class B
|
||||
(no semantic discrepancy). Realized partitions more heterogeneous (19/27 |S_j|<M vs 7/27).
|
||||
Condition-level pooled labels: alpha=0.50 -> 0.75, alpha=0.25 -> 0.00 (the value 0.50 is only the
|
||||
Session-5 seed-0 choice, not a condition-level selection).
|
||||
|
||||
Task: decide and document the alpha-expansion plan (recommended: alpha=0.25 then 0.75,
|
||||
seeds {0,1,2}, five betas each, sequential, reusing frozen manifests, pooled
|
||||
validation-only selection) OR argue for more seeds at alpha=0.50 first. Do not launch
|
||||
5G-NIDD or the full grid. Update docs and docs/handoff/SESSION_HANDOFF.md. Use
|
||||
.venv/bin/python. Do not commit or push unless asked.
|
||||
Task: run X-IIoTID alpha=0.75 for seeds {0,1,2} across the five frozen betas plus
|
||||
original_protean, sequential, reusing frozen manifests, pooled validation-only selection, then
|
||||
test evaluation, direct-effect, client-level, and cross-alpha comparison. Produce
|
||||
docs/research/ALPHA075_BETA_VALIDATION.md and update the handoff. Use .venv/bin/python. Do not
|
||||
commit or push unless asked.
|
||||
```
|
||||
Reference in new issue
Block a user