experiment: validate 5gnidd beta selection at alpha 0.75
This commit is contained in:
1 parent
a7f2b0049e
commit
bbd3508506
36 files changed
+38705
-189
No files matched your search
+189
-189
@@ -3,26 +3,26 @@
|
||||
This is the canonical continuation point for the next agent session. It is written to be
|
||||
readable without prior chat history, terminal scrollback, or git history.
|
||||
|
||||
Last updated: 2026-09-28 (Session 9). Session 9 type: **first cross-dataset (5G-NIDD)
|
||||
beta-validation condition**. Sessions 6–8 completed the X-IIoTID three-alpha study
|
||||
(`alpha=0.25/0.50/0.75`, seeds `{0,1,2}`). Session 9 ran **only** 5G-NIDD `alpha=0.50` for seeds
|
||||
`{0,1,2}` across the five frozen beta candidates plus `original_protean`, using the same
|
||||
pooled-across-seed validation-only selection rule. **5G-NIDD alpha=0.25 and alpha=0.75 were NOT
|
||||
run.** No commit or push was made.
|
||||
Last updated: 2026-09-29 (Session 10). Session 10 type: **second 5G-NIDD beta-validation
|
||||
condition**. Sessions 6–8 completed the X-IIoTID three-alpha study (`alpha=0.25/0.50/0.75`,
|
||||
seeds `{0,1,2}`); Session 9 ran 5G-NIDD `alpha=0.50` for seeds `{0,1,2}`. Session 10 ran **only**
|
||||
5G-NIDD `alpha=0.75` for seeds `{0,1,2}` across the five frozen beta candidates plus
|
||||
`original_protean`, using the same pooled-across-seed validation-only selection rule.
|
||||
**5G-NIDD alpha=0.25 was NOT run.** No commit or push was made.
|
||||
|
||||
> **Audit status (Session 9).** Independent audit verdict: **PASS**
|
||||
> (`docs/research/SESSION9_AUDIT.md`). `beta_star(5gnidd, alpha=0.50) = 1.00` confirmed from
|
||||
> validation only; reuse decision resolved; run inventory 18/18; pairing hash-identical and
|
||||
> reproduced from raw data; rare/unseen 0 mismatches over 270 cells; leakage audit PASS; `beta=0`
|
||||
> class **B**; source/data integrity PASS; pytest 75 passed. **Session 9 is safe to checkpoint.**
|
||||
> **Audit status (Session 10).** Independent audit verdict: **PASS**
|
||||
> (`docs/research/SESSION10_AUDIT.md`). `beta_star(5gnidd, alpha=0.75) = 0.50` confirmed from
|
||||
> validation only; run inventory 18/18; pairing hash-identical and reproduced from raw data;
|
||||
> rare/unseen 0 mismatches over 1,620 cells; leakage audit PASS; `beta=0` class **B**; source/data
|
||||
> integrity PASS; pytest 75 passed. **Session 10 is safe to checkpoint.**
|
||||
|
||||
> **Headline (Session 9).** PRIMARY pooled-validation selected beta for 5G-NIDD `alpha=0.50` =
|
||||
> **1.00** (validation only; per-seed diagnostics 1.00 / 0.00 / 1.00). The pooled validation curve
|
||||
> is **non-monotone** (0.687 → 0.645 → 0.675 → 0.688 → 0.692) and the total between-beta spread
|
||||
> (≈ 0.047) is comparable to the between-seed std (0.053–0.139), so the selection is statistically
|
||||
> weak. `beta=0` is class **B** for all three seeds. The pre-existing seed-0 baseline was **not
|
||||
> reused** (missing validation artifacts); the fresh rerun reproduces its training outputs
|
||||
> byte-identically. X-IIoTID alpha=0.50 selected 0.75; the two datasets differ descriptively.
|
||||
> **Headline (Session 10).** PRIMARY pooled-validation selected beta for 5G-NIDD `alpha=0.75` =
|
||||
> **0.50** (validation only; per-seed diagnostics 0.50 / 1.00 / 1.00). The pooled validation curve
|
||||
> is **non-monotone** (0.778 → 0.790 → 0.809 → 0.792 → 0.804) and the total between-beta spread
|
||||
> (≈ 0.031) is smaller than the between-seed std (0.051–0.065), so the selection is statistically
|
||||
> weak. `beta=0` is class **B** for all three seeds. At alpha=0.75 the rare-class union is
|
||||
> **identical across seeds** and realized support imbalance is lower than at alpha=0.50, yet the
|
||||
> number of `|S_j|<M` cells is unchanged (4/27) and missing cells are slightly higher (5 vs 4).
|
||||
|
||||
---
|
||||
|
||||
@@ -55,39 +55,40 @@ bitwise identical to the baseline (see §7).
|
||||
`7b9290057ee42e784da3c0d84b781815502c9205c74175c96374e71a5ffd98a0`.
|
||||
- Frozen baseline (`original_protean`) source unchanged; no `equal_available` mode added; beta
|
||||
candidate set unchanged (`{0, .25, .5, .75, 1}`). No tracked source/test/config file differs
|
||||
from HEAD. The only modified tracked file is this handoff.
|
||||
from HEAD. The only new tracked-class file candidates are the new docs/artifacts listed in §16.
|
||||
|
||||
## 3. Session 9 objective and outcome
|
||||
## 3. Session 10 objective and outcome
|
||||
|
||||
Objective: begin the 5G-NIDD phase conservatively — Stage A verification, then (if clean) **one**
|
||||
condition: 5G-NIDD `alpha=0.50`, seeds `{0,1,2}`, `original_protean` + five betas.
|
||||
Objective: run exactly **one** new condition — 5G-NIDD `alpha=0.75`, seeds `{0,1,2}`,
|
||||
`original_protean` + five betas — after a Stage-A verification gate. Do not run alpha=0.25, do not
|
||||
launch the grid, do not change methodology.
|
||||
|
||||
Stage A result: raw hash valid; all nine 5G-NIDD manifests (alpha 0.25/0.50/0.75 × seeds) reproduce
|
||||
exactly from raw data; label semantics and rare/unseen logic valid; pre-existing seed-0 baseline
|
||||
not reusable. Gate = **GO**.
|
||||
Stage A result: raw hash valid; all three 5G-NIDD `alpha=0.75` manifests reproduce exactly from raw
|
||||
data (support match, partition-hash match); label semantics and rare/unseen logic valid; pairing
|
||||
framework valid. Gate = **GO**.
|
||||
|
||||
Outcome: 18/18 runs rc=0. **PRIMARY pooled-validation selected beta = 1.00** (validation only);
|
||||
`test_used_for_selection = false` for all 15 runs. Pooled means: 0.00 → 0.687230,
|
||||
0.25 → 0.645078, 0.50 → 0.674669, 0.75 → 0.688368, 1.00 → 0.692370. Reports:
|
||||
`docs/research/5GNIDD_ALPHA050_BETA_VALIDATION.md`, audit `docs/research/SESSION9_AUDIT.md`.
|
||||
Outcome: 18/18 runs rc=0. **PRIMARY pooled-validation selected beta = 0.50** (validation only);
|
||||
`test_used_for_selection = false` for all 15 runs. Pooled means: 0.00 → 0.778094, 0.25 → 0.790483,
|
||||
0.50 → 0.809081, 0.75 → 0.791530, 1.00 → 0.804306. Reports:
|
||||
`docs/research/5GNIDD_ALPHA075_BETA_VALIDATION.md`, audit `docs/research/SESSION10_AUDIT.md`.
|
||||
|
||||
## 4. Condition and pairing
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| dataset / alpha / seeds | `5gnidd` / 0.50 / `{0,1,2}` |
|
||||
| dataset / alpha / seeds | `5gnidd` / 0.75 / `{0,1,2}` |
|
||||
| runs | 15 `support_weighted` + 3 `original_protean`, all new this session |
|
||||
| training | 10 clients, 10 rounds, 3 local epochs, batch 256, lr 1e-3, mu 0.1, lambda 1.0, Adam |
|
||||
| code revision | unchanged from HEAD `2d27abc35026456de765c929dbcf0503a72ac572`; only docs/analysis added |
|
||||
| code revision | unchanged from HEAD `a7f2b0049e99fb66021b4660b82a48d5dbbef072`; only docs/analysis added |
|
||||
|
||||
Manifests (reused, never regenerated; one content hash per seed across all six runs; each
|
||||
reproduced from the raw dataset with split/`N_ij`/partition hash verified):
|
||||
|
||||
| seed | manifest content hash | partition hash | split hash | rng seed | round-0 snapshot SHA-256 |
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | `0732582cf4510089…` | `8352d4ff…` | `18f89f48…` | 673621352 | `eb0839f2595727…` |
|
||||
| 1 | `f43906e111a8342c…` | `0815c91e…` | `3d05b7fe…` | 1403944623 | `6cbd04e0dd97bb…` |
|
||||
| 2 | `fe8b4fc6f0486cf4…` | `fdcf388e…` | `fc758d06…` | 633461056 | `f012a02b543b92…` |
|
||||
| 0 | `96cd9b90b71895da…` | `74a91845627dbe57…` | `18f89f48…` | 892370747 | `266a6e9976d5c138…` |
|
||||
| 1 | `007c23564a44919b…` | `fc04ba00f4942bf9…` | `3d05b7fe…` | 1817661572 | `acecab9a60852e45…` |
|
||||
| 2 | `3134bea2d47a7336…` | `4ee33d0a28326ae1…` | `fc758d06…` | 306220443 | `ef90c51aa9998ec8…` |
|
||||
|
||||
Splits: train/validation/test = 851,123 / 121,589 / 243,177 for every seed (10 clients, 9 classes,
|
||||
`Benign`). `train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv` identical across each
|
||||
@@ -97,60 +98,61 @@ hashes.
|
||||
## 5. Validation results (USED FOR SELECTION) and pooled selection
|
||||
|
||||
Primary metric: validation per-client-mean rare-class macro-F1. All 10 clients eligible in every
|
||||
seed. Full tables: `artifacts/analysis/5gnidd__alpha0.5__multiseed__validation_per_seed.csv`.
|
||||
seed. Full tables: `artifacts/analysis/5gnidd__alpha0.75__multiseed__validation_per_seed.csv`.
|
||||
|
||||
Per-seed validation per-client-mean rare-class macro-F1:
|
||||
|
||||
| beta | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| 0.00 | 0.738597 | 0.633243 | 0.689849 |
|
||||
| 0.25 | 0.661461 | 0.573717 | 0.700055 |
|
||||
| 0.50 | 0.764487 | 0.563388 | 0.696131 |
|
||||
| 0.75 | 0.750661 | 0.543319 | 0.771124 |
|
||||
| 1.00 | 0.768877 | 0.531764 | 0.776468 |
|
||||
| 0.00 | 0.834681 | 0.763787 | 0.735812 |
|
||||
| 0.25 | 0.823388 | 0.821190 | 0.726871 |
|
||||
| 0.50 | **0.836792** | 0.848836 | 0.741617 |
|
||||
| 0.75 | 0.824667 | 0.833255 | 0.716668 |
|
||||
| 1.00 | 0.815774 | **0.852469** | **0.744677** |
|
||||
|
||||
Pooled across seeds (`...__validation_pooled.csv`):
|
||||
|
||||
| beta | pooled mean | std (sample) | min | max | selected |
|
||||
|---|---|---|---|---|---|
|
||||
| 0.00 | 0.687230 | 0.052726 | 0.633243 | 0.738597 | |
|
||||
| 0.25 | 0.645078 | 0.064743 | 0.573717 | 0.700055 | |
|
||||
| 0.50 | 0.674669 | 0.102253 | 0.563388 | 0.764487 | |
|
||||
| 0.75 | 0.688368 | 0.126032 | 0.543319 | 0.771124 | |
|
||||
| **1.00** | **0.692370** | 0.139140 | 0.531764 | 0.776468 | **yes** |
|
||||
| 0.00 | 0.778094 | 0.050963 | 0.735812 | 0.834681 | |
|
||||
| 0.25 | 0.790483 | 0.055100 | 0.726871 | 0.823388 | |
|
||||
| **0.50** | **0.809081** | 0.058735 | 0.741617 | 0.848836 | **yes** |
|
||||
| 0.75 | 0.791530 | 0.064974 | 0.716668 | 0.833255 | |
|
||||
| 1.00 | 0.804306 | 0.054803 | 0.744677 | 0.852469 | |
|
||||
|
||||
- **PRIMARY selected beta (`beta_star_5gnidd_alpha050`) = 1.00**, max pooled validation
|
||||
- **PRIMARY selected beta (`beta_star_5gnidd_alpha075`) = 0.50**, max pooled validation
|
||||
per-client-mean rare-class macro-F1; **no tie-break used**; validation-only.
|
||||
- **SECONDARY per-seed diagnostics only:** seed 0 → 1.00, seed 1 → 0.00, seed 2 → 1.00.
|
||||
- Between-beta spread ≈ 0.047; between-seed std 0.053–0.139 → statistically weak, descriptive.
|
||||
- **SECONDARY per-seed diagnostics only:** seed 0 → 0.50, seed 1 → 1.00, seed 2 → 1.00.
|
||||
- Between-beta spread ≈ 0.031; between-seed std 0.051–0.065 → statistically weak, descriptive.
|
||||
|
||||
## 6. Final test evaluation of the pooled-selected beta
|
||||
|
||||
`beta_star = 1.00` was fixed from pooled validation before test was read.
|
||||
`beta_star = 0.50` was fixed from pooled validation before test was read.
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.880962 | 0.840075 | 0.894914 | 0.871984 ± 0.028501 |
|
||||
| rare-class recall | 0.974355 | 0.968341 | 0.958616 | 0.967104 ± 0.007942 |
|
||||
| macro-F1 | 0.850470 | 0.815941 | 0.849378 | 0.838596 ± 0.019627 |
|
||||
| accuracy | 0.742838 | 0.744026 | 0.742907 | 0.743257 ± 0.000667 |
|
||||
| weighted-F1 | 0.716950 | 0.720106 | 0.717530 | 0.718195 ± 0.001680 |
|
||||
| rare-class macro-F1 | 0.904447 | 0.905721 | 0.880045 | 0.896738 ± 0.014470 |
|
||||
| rare-class recall | 0.958666 | 0.959723 | 0.960715 | 0.959701 ± 0.001025 |
|
||||
| macro-F1 | 0.857309 | 0.859939 | 0.842201 | 0.853150 ± 0.009572 |
|
||||
| accuracy | 0.749508 | 0.754381 | 0.752933 | 0.752274 ± 0.002503 |
|
||||
| weighted-F1 | 0.723885 | 0.730109 | 0.728408 | 0.727467 ± 0.003217 |
|
||||
|
||||
`original_protean` for the same seeds/condition:
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.871964 | 0.921136 | 0.848525 | 0.880542 ± 0.037058 |
|
||||
| rare-class recall | 0.974314 | 0.968566 | 0.975859 | 0.972913 ± 0.003843 |
|
||||
| macro-F1 | 0.836881 | 0.865437 | 0.824867 | 0.842395 ± 0.020840 |
|
||||
| accuracy | 0.734416 | 0.747377 | 0.750133 | 0.743975 ± 0.008393 |
|
||||
| weighted-F1 | 0.708765 | 0.722802 | 0.724622 | 0.718730 ± 0.008678 |
|
||||
| rare-class macro-F1 | 0.905589 | 0.860758 | 0.873356 | 0.879901 ± 0.023121 |
|
||||
| rare-class recall | 0.961285 | 0.959036 | 0.963321 | 0.961214 ± 0.002144 |
|
||||
| macro-F1 | 0.857109 | 0.829867 | 0.837238 | 0.841405 ± 0.014091 |
|
||||
| accuracy | 0.747048 | 0.754072 | 0.751745 | 0.750955 ± 0.003578 |
|
||||
| weighted-F1 | 0.721059 | 0.729950 | 0.727133 | 0.726047 ± 0.004544 |
|
||||
|
||||
Descriptive difference for test rare-class macro-F1 (`beta_star` − `original_protean`) is
|
||||
+0.008998 / −0.081061 / +0.046389 (mean −0.008558). Test is descriptive only; no superiority or
|
||||
generalisation claimed. Note 5G-NIDD test weighted-F1 (`≈0.718`) sits **below** accuracy
|
||||
(`≈0.744`) — the opposite of X-IIoTID — because the high-support classes are the harder ones for
|
||||
nearest-prototype inference.
|
||||
−0.001141 / +0.044963 / +0.006689 (mean +0.016837). Test is descriptive only; no superiority or
|
||||
generalisation claimed. The global rare evaluation support is **28,000 for every seed** (rare-class
|
||||
union identical across seeds at alpha=0.75), so cross-seed test levels are directly comparable at
|
||||
this condition. Test weighted-F1 (`≈0.727`) again sits **below** accuracy (`≈0.752`), as at
|
||||
alpha=0.50.
|
||||
|
||||
## 7. `beta = 0` equivalence result (all three seeds)
|
||||
|
||||
@@ -159,157 +161,152 @@ For seeds 0, 1, 2, `support_weighted beta=0` vs frozen `original_protean`:
|
||||
- Same manifest/split/hashes, round-0 snapshot byte-identical, support/char/rare_unseen
|
||||
identical: **yes** for all three.
|
||||
- Independent aggregation (`ΣC/n` vs `(1/n)ΣC`): 4.4e-16 / 4.4e-16 / 4.4e-16.
|
||||
- History rare-macro-F1 max diff (rounds ≥ 1): 1.93e-2 / 6.01e-2 / 2.60e-2; test metric max diff
|
||||
2.33e-3 / 1.86e-4 / 8.70e-3.
|
||||
- History rare-macro-F1 max diff (rounds ≥ 1): 3.22e-2 / 5.04e-2 / 2.72e-2; test metric max diff
|
||||
5.22e-3 / 5.32e-3 / 2.74e-4.
|
||||
- Classification: **B** in all seeds — semantic equivalence + floating-point divergence; no
|
||||
semantic discrepancy. Machine-readable:
|
||||
`artifacts/analysis/5gnidd__alpha0.5__multiseed__beta0_equivalence.csv`.
|
||||
`artifacts/analysis/5gnidd__alpha0.75__multiseed__beta0_equivalence.csv`.
|
||||
|
||||
Conditional on the shared `S_j` reproduction assumption; not proof `beta=0` reproduces original
|
||||
PROTEAN.
|
||||
|
||||
## 8. Partition characterization (realized) and dataset structure
|
||||
## 8. Partition characterization (realized) and comparison with alpha=0.50
|
||||
|
||||
5G-NIDD `alpha=0.50` (training split), `M = 10`:
|
||||
5G-NIDD `alpha=0.75` (training split), `M = 10`:
|
||||
|
||||
| seed | `|S_j| < M` cells | missing client-class cells | client-size ratio | union rare | union unseen | max unseen/client |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0 | 2 / 9 | 2 | 11.63 | 6 | 2 | 1 |
|
||||
| 1 | 0 / 9 | 0 | 4.43 | 5 | 0 | 0 |
|
||||
| 2 | 2 / 9 | 2 | 12.72 | 6 | 2 | 1 |
|
||||
| **sum/mean** | **4 / 27** | **4** | mean **9.59** | 6/5/6 | 2/0/2 | 1 |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 0 | 2 / 9 | 3 | 3.70 | 6 | 2 | 1 |
|
||||
| 1 | 2 / 9 | 2 | 11.36 | 6 | 2 | 1 |
|
||||
| 2 | 0 / 9 | 0 | 3.87 | 6 | 0 | 0 |
|
||||
| **sum/mean** | **4 / 27** | **5** | mean **6.31** | 6/6/6 | 2/2/0 | 1 |
|
||||
|
||||
All 10 clients have exactly two rare classes and are eligible in every seed; mean class-support
|
||||
CV = **1.225**. Full tables:
|
||||
`artifacts/analysis/5gnidd__alpha0.5__partition_heterogeneity_*.csv` and
|
||||
All 10 clients have exactly two rare classes and are eligible in every seed. **The rare-class union
|
||||
is identical across all three seeds** (`{ICMPFlood, SYNFlood, SYNScan, SlowrateDoS, TCPConnectScan,
|
||||
UDPScan}`), so test rare support is constant (28,000). Mean class-support CV = **0.916**. Missing
|
||||
cells are single-client gaps: seed 0 (`ICMPFlood` c0/c3, `SYNFlood` c4), seed 1 (`ICMPFlood` c3,
|
||||
`SYNScan` c6). Full tables:
|
||||
`artifacts/analysis/5gnidd__alpha0.75__partition_heterogeneity_*` and
|
||||
`...__partition_characterization.csv`.
|
||||
|
||||
### Comparison with X-IIoTID alpha=0.50 (descriptive)
|
||||
### Comparison with 5G-NIDD alpha=0.50 (descriptive)
|
||||
|
||||
| statistic | 5G-NIDD a0.50 | X-IIoTID a0.50 |
|
||||
| statistic | 5G-NIDD a0.50 | 5G-NIDD a0.75 |
|
||||
|---|---|---|
|
||||
| pooled-selected beta | **1.00** | **0.75** |
|
||||
| pooled validation mean range (spread) | 0.645–0.692 (0.047) | 0.544–0.579 (0.035) |
|
||||
| pooled validation std range | 0.053–0.139 | 0.075–0.100 |
|
||||
| `|S_j| < M` cells (sum/27) | **4/27** | 7/27 |
|
||||
| missing client-class cells | **4** | 9 |
|
||||
| client-size ratio mean | **9.59** | 7.41 |
|
||||
| mean class-support CV | 1.225 | 1.183 |
|
||||
| union rare per seed | 6/5/6 | 6/7/7 |
|
||||
| union unseen per seed | 2/0/2 | 3/3/1 |
|
||||
| mean displacement ‖Cbar(1)−Cbar(0)‖ | 2.340 | 2.099 |
|
||||
| selected-beta client improve/degrade | 6/3, 3/5, 7/2 | 7/3, 8/1, 8/1 |
|
||||
| pooled-selected beta | **1.00** | **0.50** |
|
||||
| pooled validation mean range (spread) | 0.645–0.692 (0.047) | 0.778–0.809 (0.031) |
|
||||
| pooled validation std range | 0.053–0.139 | 0.051–0.065 |
|
||||
| `|S_j| < M` cells (sum/27) | 4/27 | 4/27 |
|
||||
| missing client-class cells | 4 | 5 |
|
||||
| client-size ratio mean | 9.59 | 6.31 |
|
||||
| mean class-support CV | 1.225 | 0.916 |
|
||||
| union rare per seed | 6/5/6 | 6/6/6 |
|
||||
| union unseen per seed | 2/0/2 | 2/2/0 |
|
||||
| test rare support per seed | 104,843 / 13,375 / 128,995 | 28,000 / 28,000 / 28,000 |
|
||||
| mean displacement `‖Cbar(1)−Cbar(0)‖` | 2.340 | 1.385 |
|
||||
| selected-beta client improve/degrade | 6/3, 3/5, 7/2 | 4/6, 6/3, 4/2 |
|
||||
| `beta=0` class | B/B/B | B/B/B |
|
||||
|
||||
5G-NIDD has fewer missing class cells but more imbalanced client sizes; the selected beta differs
|
||||
descriptively. No causal or monotone claim; only two datasets, one condition each.
|
||||
Higher alpha lowers **mean** support imbalance and mean displacement, but the number of
|
||||
`|S_j|<M` cells is unchanged and missing cells rise. No causal or monotone claim (two alpha levels).
|
||||
Machine-readable cross-alpha artifacts: `artifacts/analysis/5gnidd__cross_alpha__*`.
|
||||
|
||||
## 9. Direct prototype-effect findings (fixed round-0 snapshots, no retraining)
|
||||
|
||||
`||Cbar_j(1) − Cbar_j(0)||`, cross-seed mean (std), 5G-NIDD alpha=0.50:
|
||||
`||Cbar_j(1) − Cbar_j(0)||`, cross-seed mean (std), 5G-NIDD alpha=0.75:
|
||||
|
||||
| class | mean disp | std |
|
||||
|---|---|---|
|
||||
| UDPScan | 3.461 | 2.011 |
|
||||
| HTTPFlood | 3.120 | 1.289 |
|
||||
| SYNScan | 2.875 | 0.959 |
|
||||
| SlowrateDoS | 2.763 | 1.454 |
|
||||
| SYNFlood | 2.262 | 0.748 |
|
||||
| TCPConnectScan | 2.131 | 0.213 |
|
||||
| Benign | 1.690 | 0.699 |
|
||||
| UDPFlood | 1.598 | 0.428 |
|
||||
| ICMPFlood | 1.159 | 0.093 |
|
||||
| UDPScan | 2.445 | 0.484 |
|
||||
| SYNFlood | 1.897 | 0.171 |
|
||||
| SlowrateDoS | 1.565 | 0.654 |
|
||||
| SYNScan | 1.563 | 0.640 |
|
||||
| HTTPFlood | 1.366 | 0.544 |
|
||||
| TCPConnectScan | 1.226 | 0.476 |
|
||||
| UDPFlood | 0.903 | 0.103 |
|
||||
| Benign | 0.824 | 0.272 |
|
||||
| ICMPFlood | 0.673 | 0.331 |
|
||||
|
||||
Mean across classes **2.340** at beta=1.0 (monotone: 0.953 / 1.567 / 2.003 / 2.340 for beta
|
||||
0.25/0.50/0.75/1.00). Seed-sensitive (e.g. UDPScan 1.42 / 3.52 / 5.44). Displacement is a
|
||||
direct-aggregation measure only, not a detection result. Outputs:
|
||||
`...__prototype_effect_pooled.csv` and per-seed `...__prototype_effect.csv` / `...__prototype_weights.csv`.
|
||||
Mean across classes **1.385** at beta=1.0 (monotone: 0.497 / 0.875 / 1.163 / 1.385 for beta
|
||||
0.25/0.50/0.75/1.00), smaller than at alpha=0.50 (2.340). Displacement is a direct-aggregation
|
||||
measure only, not a detection result. Outputs:
|
||||
`artifacts/analysis/5gnidd__alpha0.75__multiseed__prototype_effect_pooled.csv` and per-seed
|
||||
`...__prototype_effect.csv` / `...__prototype_weights.csv`.
|
||||
|
||||
## 10. Client-level findings (validation, vs beta = 0)
|
||||
|
||||
Detail: `...__client_validation_detail.csv`; summary: `...__client_validation_summary.csv`.
|
||||
|
||||
At the selected `beta=1.00`: **6/3/1** (seed 0, mean Δ +0.030), **3/5/2** (seed 1, mean Δ −0.101;
|
||||
worst client −0.656), **7/2/1** (seed 2, mean Δ +0.087). The response is seed-dependent and
|
||||
`N_ij` vs ΔF1 is non-monotonic. Client ids are per-seed realisations and are not pooled.
|
||||
At the selected `beta=0.50`: **4/6/0** (seed 0, mean Δ +0.0021, range −0.0434…+0.0744),
|
||||
**6/3/1** (seed 1, mean Δ +0.0850, range −0.0172…+0.2667), **4/2/4** (seed 2, mean Δ +0.0058).
|
||||
The response is seed-dependent and `N_ij` vs ΔF1 is non-monotonic. Client ids are per-seed
|
||||
realisations and are not pooled.
|
||||
|
||||
## 11. Runtime / resources
|
||||
|
||||
CPU only, no GPU, no swap, exit 0 on all 18 runs. Per-run wall-clock 62–68 min; CPU 734–745 %;
|
||||
peak RSS 10.36 GiB (seed 0), 8.67 GiB (seed 1), 14.04 GiB (seed 2). New compute: 18 runs,
|
||||
69,336 s ≈ **19.26 h** wall-clock, sequential (8 cores). Machine-readable:
|
||||
`artifacts/analysis/5gnidd__alpha0.5__multiseed__runtime.csv`.
|
||||
CPU only, no GPU, no swap, exit 0 on all 18 runs. Per-run wall-clock 61:55–66:37; CPU 738–745 %;
|
||||
peak RSS ~8.69 GiB (seed 0), ~11.78 GiB (seed 1), ~11.00 GiB (seed 2). New compute: 18 runs,
|
||||
**69,064 s ≈ 19.18 h** wall-clock, sequential (8 cores). Machine-readable:
|
||||
`artifacts/analysis/5gnidd__alpha0.75__multiseed__runtime.csv`.
|
||||
|
||||
## 12. Methodological decisions / assumptions
|
||||
|
||||
1. **`S_j` participant/denominator rule** (`S_j = {i | N_ij > 0 and valid C_ij}`, equal weights in
|
||||
the baseline) is a **REPRODUCTION ASSUMPTION**; material in only **4/27** class-condition cells
|
||||
at 5G-NIDD alpha=0.50. Carry this caveat in all reporting.
|
||||
the baseline) is a **REPRODUCTION ASSUMPTION**; material in **4/27** class-condition cells at
|
||||
5G-NIDD alpha=0.75. Carry this caveat in all reporting.
|
||||
2. `N_ij` is class-specific sample support, not prototype quality.
|
||||
3. Beta is one value for all classes/clients; candidates frozen at `{0,.25,.5,.75,1}`.
|
||||
4. **PRIMARY selection = pooled validation across clients and seeds**; per-seed selection is
|
||||
secondary stability diagnostics; test used only after the pooled beta is fixed.
|
||||
5. No new source scripts this session; analysis used the existing read-only
|
||||
`analyze_partition_heterogeneity.py`, `analyze_prototype_effect.py`, `analyze_multiseed_beta.py`,
|
||||
`verify_beta0_equivalence.py`. Ad-hoc Stage-A/audit helpers live only under
|
||||
`/tmp/opencode/s9/` (outside the repo).
|
||||
`verify_beta0_equivalence.py`, `analyze_cross_alpha.py`. Ad-hoc Stage-A/audit helpers live only
|
||||
under `/tmp/opencode/s10/` (outside the repo).
|
||||
6. `lambda=1.0`, Adam, lr 1e-3, batch 256 remain PROJECT DECISIONS. No `equal_available` mode.
|
||||
|
||||
## 13. Unresolved questions / blockers
|
||||
|
||||
1. `S_j` reproduction ambiguity — unresolved; no official PROTEAN code exists.
|
||||
2. Pooled beta selection remains statistically weak at all measured conditions: across-seed std is
|
||||
comparable to or larger than the between-beta spread. No inferential test was performed.
|
||||
comparable to the between-beta spread. No inferential test was performed.
|
||||
3. `beta = 0` is not bitwise identical to `original_protean`; documentation decision still open.
|
||||
4. Session-2 manifests still store `preprocessing_hash = null` / `config_hash = null`.
|
||||
5. Only 5G-NIDD alpha=0.50 (one condition) measured; no 5G-NIDD alpha=0.25/0.75 result; no
|
||||
cross-alpha 5G-NIDD or cross-dataset causal claim.
|
||||
6. The selected beta differs across datasets (5G-NIDD 1.00 vs X-IIoTID 0.75) but no trend or
|
||||
causality is claimed.
|
||||
5. 5G-NIDD now has alpha=0.50 (1.00) and alpha=0.75 (0.50); **alpha=0.25 remains unmeasured**, so
|
||||
no 5G-NIDD alpha→beta trend can be stated.
|
||||
6. The selected beta differs across datasets/conditions (5G-NIDD a0.50 = 1.00, a0.75 = 0.50;
|
||||
X-IIoTID a0.75 = 0.50) but no trend or causality is claimed.
|
||||
|
||||
## 14. Git status
|
||||
|
||||
No commit or push was made. `HEAD = 2d27abc35026456de765c929dbcf0503a72ac572`, branch `main`,
|
||||
in sync with `origin/main`. `git status --short` shows only untracked docs/analysis artifacts and
|
||||
the modified handoff; `results/` is gitignored:
|
||||
No commit or push was made. `HEAD = a7f2b0049e99fb66021b4660b82a48d5dbbef072`, branch `main`,
|
||||
in sync with `origin/main`. `git status --short` shows only untracked docs/analysis artifacts (no
|
||||
tracked modification); `results/` is gitignored. `planningawal/` and both raw datasets are
|
||||
unchanged. `equal_available` absent. No test-based selection. The 18 runs are the only
|
||||
`results/5gnidd__alpha0.75__*` directories.
|
||||
|
||||
```
|
||||
M docs/handoff/SESSION_HANDOFF.md
|
||||
?? docs/research/5GNIDD_ALPHA050_BETA_VALIDATION.md
|
||||
?? docs/research/SESSION9_AUDIT.md
|
||||
?? artifacts/analysis/5gnidd__alpha0.5__* (46 files: snapshots, multiseed tables, effect/weights,
|
||||
partition characterization, beta0 equivalence, runtime,
|
||||
audit_recomputation)
|
||||
?? (pre-existing Session-5/6/7/8 untracked views, unchanged)
|
||||
```
|
||||
## 15. Session 10 audit result
|
||||
|
||||
No tracked source/test/config file differs from HEAD. `planningawal/` and both raw datasets are
|
||||
unchanged. `artifacts/partitions/` unmodified. `equal_available` absent. No 5G-NIDD
|
||||
`support_weighted` run before this session; no 5G-NIDD alpha=0.25/0.75 run. The pre-existing
|
||||
`5gnidd__alpha0.5__seed0__original_protean` was overwritten in place by the fresh rerun (backed up
|
||||
to `/tmp/opencode/s9/old_baseline_backup/`).
|
||||
|
||||
## 15. Session 9 audit result
|
||||
|
||||
- Verdict: **PASS** (`docs/research/SESSION9_AUDIT.md`).
|
||||
- Verdict: **PASS** (`docs/research/SESSION10_AUDIT.md`).
|
||||
- Run inventory 18/18, pairing hash-identical, manifests reproduced from raw data.
|
||||
- Validation recomputed exactly; **pooled `beta_star(5gnidd, alpha=0.50) = 1.00`** (validation
|
||||
- Validation recomputed exactly; **pooled `beta_star(5gnidd, alpha=0.75) = 0.50`** (validation
|
||||
only). Selected-beta test and `original_protean` test means/stds exact.
|
||||
- Leakage audit PASS; `test_used_for_selection=false` upheld.
|
||||
- Rare/unseen integrity PASS (0 mismatches / 270 cells). `beta=0` equivalence class **B**.
|
||||
- Reuse decision PASS (not reused; rerun reproduces old baseline exactly).
|
||||
- Direct-effect, client-level, cross-dataset labels: PASS.
|
||||
- Source/data integrity PASS. pytest: **75 passed, 0 failed, 0 skipped, 0 warnings**.
|
||||
- Rare/unseen integrity PASS (0 mismatches / 1,620 cells). `beta=0` equivalence class **B**.
|
||||
- Direct-effect, client-level, cross-condition labels: PASS.
|
||||
- Source/data integrity PASS. pytest: **75 passed, 0 failed, 0 skipped**.
|
||||
|
||||
## 16. Files created / modified this session
|
||||
|
||||
Created:
|
||||
|
||||
- `docs/research/5GNIDD_ALPHA050_BETA_VALIDATION.md`
|
||||
- `docs/research/SESSION9_AUDIT.md`
|
||||
- `artifacts/analysis/5gnidd__alpha0.5__*` (46 files: 18 prototype snapshots, 6 prototype
|
||||
effect/weights CSVs, multiseed validation/client/test/effect/beta0/runtime tables,
|
||||
partition heterogeneity + characterization, audit recomputation)
|
||||
- `results/5gnidd__alpha0.5__seed{0,1,2}__*` (18 run dirs, gitignored)
|
||||
- `docs/research/5GNIDD_ALPHA075_BETA_VALIDATION.md`
|
||||
- `docs/research/SESSION10_AUDIT.md`
|
||||
- `artifacts/analysis/5gnidd__alpha0.75__*` — 18 prototype snapshots + 6 per-seed
|
||||
effect/weights CSVs + `5gnidd__alpha0.75__multiseed__*` (validation per-seed/pooled,
|
||||
client detail/summary/deltas, test descriptive, prototype effect pooled, beta0 equivalence,
|
||||
runtime, selection summary, audit recomputation) + `5gnidd__alpha0.75__partition_heterogeneity_*`
|
||||
+ `5gnidd__alpha0.75__partition_characterization.csv` + `5gnidd__cross_alpha__*`
|
||||
- `results/5gnidd__alpha0.75__seed{0,1,2}__*` (18 run dirs, gitignored)
|
||||
|
||||
Modified:
|
||||
|
||||
@@ -319,20 +316,20 @@ No source or test code was changed.
|
||||
|
||||
## 17. Exact recommended next phase
|
||||
|
||||
1. **Session 9 is audited and safe to checkpoint** (verdict PASS; see §15 and
|
||||
`docs/research/SESSION9_AUDIT.md`). Before checkpointing, stage only the intended
|
||||
1. **Session 10 is audited and safe to checkpoint** (verdict PASS; see §15 and
|
||||
`docs/research/SESSION10_AUDIT.md`). Before checkpointing, stage only the intended
|
||||
docs/analysis artifacts; do not commit `results/` or the raw datasets.
|
||||
2. **5G-NIDD alpha=0.50 is complete.** Next 5G-NIDD condition (alpha=0.75 or alpha=0.25) is
|
||||
**safe to run** — its manifest already exists and reproduces from raw data, the pipeline is
|
||||
stable, and the same frozen protocol applies. Budget **≈19 h per condition** (18 runs × ~64 min)
|
||||
and run **one condition at a time**; do not launch the full grid blindly. A single condition,
|
||||
then inspect, then proceed is recommended.
|
||||
2. **5G-NIDD alpha=0.50 and alpha=0.75 are complete.** The final 5G-NIDD condition, **alpha=0.25**,
|
||||
is **safe to run** — its manifest already exists and reproduces from raw data, the pipeline is
|
||||
stable, and the same frozen protocol applies. Budget **≈19 h** (18 runs × ~64 min) and run
|
||||
**one condition at a time**; do not launch the grid blindly.
|
||||
3. Follow the same pooled validation-only selection rule; report each selected beta as "best among
|
||||
the tested candidates under pooled validation for that condition" — never a global optimum.
|
||||
4. Do not infer a monotone alpha→beta relation from three X-IIoTID conditions or from a single
|
||||
5G-NIDD condition; do not claim dataset causality from two datasets.
|
||||
5. Update `EXPERIMENT_PROTOCOL.md` only if a real ambiguity appears; update this handoff.
|
||||
6. Do not commit or push unless asked.
|
||||
4. After alpha=0.25, build the 5G-NIDD cross-alpha comparison (alpha=0.25/0.50/0.75) analogous to
|
||||
`XIIOTID_CROSS_ALPHA_ANALYSIS.md`, and only then consider cross-dataset discussion.
|
||||
5. Do not infer a monotone alpha→beta relation; do not claim dataset causality.
|
||||
6. Update `EXPERIMENT_PROTOCOL.md` only if a real ambiguity appears; update this handoff.
|
||||
7. Do not commit or push unless asked.
|
||||
|
||||
## 18. Important commands
|
||||
|
||||
@@ -340,39 +337,42 @@ No source or test code was changed.
|
||||
# Tests (75 expected)
|
||||
.venv/bin/python -m pytest
|
||||
|
||||
# One full paired beta run (CPU, ~63-68 min, ~8.7-14.1 GiB for 5G-NIDD)
|
||||
.venv/bin/python scripts/run_baseline.py --dataset 5gnidd --alpha 0.5 --seed 1 \
|
||||
# One full paired beta run (CPU, ~62-67 min, ~8.7-11.8 GiB for 5G-NIDD)
|
||||
.venv/bin/python scripts/run_baseline.py --dataset 5gnidd --alpha 0.75 --seed 1 \
|
||||
--clients 10 --rounds 10 --local-epochs 3 \
|
||||
--aggregation-mode support_weighted --beta 1.0
|
||||
|
||||
# Pooled multi-seed validation analysis (read-only, no retraining)
|
||||
.venv/bin/python scripts/analyze_multiseed_beta.py --dataset 5gnidd --alpha 0.5 --seeds 0 1 2
|
||||
.venv/bin/python scripts/analyze_multiseed_beta.py --dataset 5gnidd --alpha 0.75 --seeds 0 1 2
|
||||
|
||||
# Direct prototype-effect analysis (no retraining)
|
||||
.venv/bin/python scripts/analyze_prototype_effect.py \
|
||||
--snapshot artifacts/analysis/5gnidd__alpha0.5__seed0__support_weighted__beta0__prototype_snapshot.json \
|
||||
--snapshot artifacts/analysis/5gnidd__alpha0.75__seed0__support_weighted__beta0__prototype_snapshot.json \
|
||||
--class-names "Benign,HTTPFlood,ICMPFlood,SYNFlood,SYNScan,SlowrateDoS,TCPConnectScan,UDPFlood,UDPScan" \
|
||||
--output-dir artifacts/analysis
|
||||
|
||||
# Realized partition characterization
|
||||
.venv/bin/python scripts/analyze_partition_heterogeneity.py --dataset 5gnidd --alphas 0.5
|
||||
# Realized partition characterization (cross-alpha)
|
||||
.venv/bin/python scripts/analyze_partition_heterogeneity.py --dataset 5gnidd --alphas 0.5 0.75
|
||||
|
||||
# beta=0 equivalence audit
|
||||
.venv/bin/python scripts/verify_beta0_equivalence.py --dataset 5gnidd --alpha 0.5 --seeds 0 1 2
|
||||
.venv/bin/python scripts/verify_beta0_equivalence.py --dataset 5gnidd --alpha 0.75 --seeds 0 1 2
|
||||
|
||||
# Cross-alpha comparison table (5G-NIDD)
|
||||
.venv/bin/python scripts/analyze_cross_alpha.py --dataset 5gnidd --alphas 0.5 0.75 --tag 5gnidd__cross_alpha
|
||||
```
|
||||
|
||||
## 19. Recommended next-session prompt
|
||||
|
||||
```
|
||||
You are continuing the perdalam-protean research repository (Session 10).
|
||||
You are continuing the perdalam-protean research repository (Session 11).
|
||||
|
||||
FIRST, read completely:
|
||||
1. docs/handoff/SESSION_HANDOFF.md
|
||||
2. docs/research/5GNIDD_ALPHA050_BETA_VALIDATION.md
|
||||
3. docs/research/SESSION9_AUDIT.md
|
||||
4. docs/research/EXPERIMENT_PROTOCOL.md
|
||||
5. docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md
|
||||
6. docs/research/MULTISEED_BETA_VALIDATION.md
|
||||
2. docs/research/5GNIDD_ALPHA075_BETA_VALIDATION.md
|
||||
3. docs/research/SESSION10_AUDIT.md
|
||||
4. docs/research/5GNIDD_ALPHA050_BETA_VALIDATION.md
|
||||
5. docs/research/EXPERIMENT_PROTOCOL.md
|
||||
6. docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md
|
||||
|
||||
Critical rules:
|
||||
- planningawal/ is immutable.
|
||||
@@ -380,18 +380,18 @@ Critical rules:
|
||||
- The S_j participant rule is a REPRODUCTION ASSUMPTION.
|
||||
- Do NOT add an executable equal_available mode.
|
||||
- Beta selection uses validation only; PRIMARY is pooled across clients and seeds.
|
||||
- Do NOT launch the full grid; run one condition at a time (~19 h per 5G-NIDD condition).
|
||||
- Run one condition at a time (~19 h per 5G-NIDD condition).
|
||||
|
||||
Context: Session 9 completed 5G-NIDD alpha=0.50 for seeds {0,1,2} (18 runs, ~19.26 h) and was
|
||||
independently audited (PASS; docs/research/SESSION9_AUDIT.md). Pooled validation selected
|
||||
beta=1.00 (per-seed diagnostics 1.00 / 0.00 / 1.00). beta=0 equivalence class B. Realized
|
||||
heterogeneity: 4/27 cells |S_j|<M, 4 missing cells, client ratio mean 9.59, mean displacement
|
||||
2.340. X-IIoTID alpha=0.50 selected 0.75. The pre-existing seed-0 baseline was not reusable and
|
||||
was re-run; it reproduces exactly.
|
||||
Context: Session 10 completed 5G-NIDD alpha=0.75 for seeds {0,1,2} (18 runs, ~19.18 h) and was
|
||||
independently audited (PASS; docs/research/SESSION10_AUDIT.md). Pooled validation selected
|
||||
beta=0.50 (per-seed diagnostics 0.50 / 1.00 / 1.00). beta=0 equivalence class B. Realized
|
||||
heterogeneity: 4/27 cells |S_j|<M, 5 missing cells, client ratio mean 6.31, mean displacement
|
||||
1.385, rare-class union identical across seeds. 5G-NIDD alpha=0.50 selected 1.00; X-IIoTID
|
||||
alpha=0.75 selected 0.50.
|
||||
|
||||
Task: run the next single 5G-NIDD condition (recommended alpha=0.75, then alpha=0.25) under the
|
||||
same frozen protocol (seeds {0,1,2}, five betas + original_protean, frozen manifests, pooled
|
||||
validation-only selection). Verify the manifest reproduces, run the 18 runs sequentially with
|
||||
runtime/status files, produce a condition report + audit, and extend the cross-dataset/cross-alpha
|
||||
comparison. Use .venv/bin/python. Do not commit or push unless asked.
|
||||
Task: run the final 5G-NIDD condition alpha=0.25 under the same frozen protocol (seeds {0,1,2},
|
||||
five betas + original_protean, frozen manifests, pooled validation-only selection). Verify the
|
||||
manifest reproduces, run the 18 runs sequentially with runtime/status files, produce a condition
|
||||
report + audit, and extend the 5G-NIDD cross-alpha comparison to alpha=0.25/0.50/0.75. Use
|
||||
.venv/bin/python. Do not commit or push unless asked.
|
||||
```
|
||||
Reference in new issue
Block a user