experiment: complete two-dataset beta study
This commit is contained in:
willtanoe committed
1 parent
bbd3508506
commit
d30e9831e6
37 files changed
+36038
-195
No files matched your search
@@ -0,0 +1,460 @@
|
||||
# 5G-NIDD Alpha-0.25 Beta Validation (Session 11)
|
||||
|
||||
Status: **final 5G-NIDD beta-validation condition**. This report covers a single 5G-NIDD
|
||||
condition — `alpha = 0.25`, seeds `{0,1,2}`, the frozen `original_protean` baseline plus the five
|
||||
`support_weighted` beta candidates — using the same pooled-across-seed validation-only selection
|
||||
rule already applied to X-IIoTID (`alpha=0.25/0.50/0.75`) and 5G-NIDD `alpha=0.50/0.75`. It is
|
||||
**not** a sweep, not a hyper-parameter search, and not evidence of a global optimum. **No other
|
||||
5G-NIDD alpha was run**; `alpha=0.50` (Session 9) and `alpha=0.75` (Session 10) are reused only as
|
||||
audited references. No condition was run after this one.
|
||||
|
||||
Companion documents:
|
||||
|
||||
- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen protocol and selection hierarchy.
|
||||
- `docs/research/5GNIDD_ALPHA050_BETA_VALIDATION.md` — audited 5G-NIDD alpha=0.50 reference.
|
||||
- `docs/research/5GNIDD_ALPHA075_BETA_VALIDATION.md` — audited 5G-NIDD alpha=0.75 reference.
|
||||
- `docs/research/SESSION9_AUDIT.md`, `docs/research/SESSION10_AUDIT.md` — prior audits.
|
||||
- `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md` — X-IIoTID three-condition reference.
|
||||
- `docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md` — the resulting three-alpha 5G-NIDD comparison.
|
||||
- `docs/research/CROSS_DATASET_BETA_ANALYSIS.md` — two-dataset, three-alpha descriptive synthesis.
|
||||
- `docs/research/SESSION11_AUDIT.md` — independent audit of this condition.
|
||||
- `docs/handoff/SESSION_HANDOFF.md` — continuation point.
|
||||
|
||||
---
|
||||
|
||||
## A. Scope
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| dataset | `5gnidd` (5G-NIDD) |
|
||||
| alpha | 0.25 |
|
||||
| seeds | 0, 1, 2 |
|
||||
| aggregation | `original_protean` (per seed) + `support_weighted` for `beta ∈ {0, 0.25, 0.50, 0.75, 1.00}` |
|
||||
| runs | 18: 15 `support_weighted` + 3 `original_protean`, **all run new this session** |
|
||||
| compute | sequential CPU, one run at a time, 68,798 s ≈ **19.11 h** wall-clock |
|
||||
| selection | pooled validation-only across seeds `{0,1,2}` |
|
||||
| test | read only after the pooled beta was fixed |
|
||||
|
||||
No 5G-NIDD `alpha=0.50` or `alpha=0.75` run was re-executed; the full grid was deliberately not
|
||||
launched. `equal_available` was not added. No training/aggregation/preprocessing source code was
|
||||
changed (HEAD `bbd35085063194db58e4472e70b204d4be4a3607`, unchanged).
|
||||
|
||||
---
|
||||
|
||||
## B. Dataset and manifests
|
||||
|
||||
### B.1 Raw-data provenance (Stage A)
|
||||
|
||||
- Path `~/datasets/5G-NIDD/5G-NIDD.csv`; SHA-256
|
||||
`fa36f80859585f474504ca69eb951a079b35145537976c86b00aae3aab46ee59` — matches the audited
|
||||
Session-9/10 value; **unmutated**.
|
||||
- Raw 1,215,890 rows; **1** exact duplicate dropped → 1,215,889; **0** label rows dropped.
|
||||
- **Label semantics (canonicalised):** label column `Attack Type`; **9 classes** — benign
|
||||
`Benign` (index 0) plus **8 attack classes** (`HTTPFlood, ICMPFlood, SYNFlood, SYNScan,
|
||||
SlowrateDoS, TCPConnectScan, UDPFlood, UDPScan`). Full-data counts unchanged from Session 9
|
||||
(Benign 477,736; UDPFlood 457,340; HTTPFlood 140,812; SlowrateDoS 73,124; TCPConnectScan
|
||||
20,052; SYNScan 20,043; UDPScan 15,906; SYNFlood 9,721; ICMPFlood 1,155).
|
||||
- **46 feature columns** (41 numeric median-imputed + train-standardized; 5 categorical one-hot
|
||||
with an explicit `__MISSING__` category). Preprocessing fits on the train split only and is
|
||||
identical to the previously audited pipeline (no source change).
|
||||
- Rare/unseen logic verified against this class list: attack classes only, `N_ij > 0`, two
|
||||
smallest positive training supports per client, `N_ij = 0` is unseen (never rare), benign never
|
||||
rare/unseen. All 10 clients are eligible in all three seeds (each has exactly two rare classes).
|
||||
|
||||
### B.2 Manifest identity (frozen, never regenerated)
|
||||
|
||||
Each `(alpha=0.25, seed)` manifest was independently reproduced from the raw dataset during Stage A
|
||||
(`prepare_dataset` → `reproduce_train_assignments`): exact `N_ij` match, partition-hash match and
|
||||
split-hash match. Train/validation/test sizes are 851,123 / 121,589 / 243,177 for every seed.
|
||||
|
||||
| seed | manifest | manifest content hash | partition hash | split hash | rng seed |
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | `artifacts/partitions/5gnidd__alpha0.25__seed0.json` | `ab6d8a8b63d478c4…` | `e628c17369f25e35…` | `18f89f4807de5493…` | 708356900 |
|
||||
| 1 | `artifacts/partitions/5gnidd__alpha0.25__seed1.json` | `74df102e4edb4509…` | `05aec12ceb4c94bd…` | `3d05b7fe48d5a06c…` | 902143161 |
|
||||
| 2 | `artifacts/partitions/5gnidd__alpha0.25__seed2.json` | `061c094d1913f942…` | `d9631ea6b2d3907e…` | `fc758d060dc51dec…` | 218373793 |
|
||||
|
||||
`preprocessing_hash` / `config_hash` remain `null` in these Session-2 manifests (known limitation,
|
||||
carry-over).
|
||||
|
||||
### B.3 Pairing (from content hashes, not filenames)
|
||||
|
||||
Within each seed all six runs share one manifest content hash and a **byte-identical round-0
|
||||
snapshot**; `train_support_Nij.csv`, `characterization.csv` and `rare_unseen.csv` each have exactly
|
||||
one SHA-256 across the six runs; `manifest_reload_ok = true` and `smoke = false` everywhere.
|
||||
|
||||
| seed | manifest content hash | round-0 snapshot SHA-256 (all 6 runs) |
|
||||
|---|---|---|
|
||||
| 0 | `ab6d8a8b63d478c4f6cfd88f96ccc7e16e1902f5e2c0ceac0f38ea26cfcda788` | `e2243ac4ed4391ff42ae358ab47ae4b98177227ac6e3ff0cdfd3a9147bf0d7c7` |
|
||||
| 1 | `74df102e4edb4509ba32b2625aa6a99f4bbd1dbbac0fc61d3bcccd50f21475ba` | `26a162991b7cbde947040c32c3d01b91841b6a58edc917038f813c8cbbd335b7` |
|
||||
| 2 | `061c094d1913f942267bdb5654d3c0d8cfef3fa592be6cd5a1845a5eb5cdc8f2` | `e287bff5ca71f8d2a677263ed17b67e909baaaada66cd9e1cb0c7f9e0c829b62` |
|
||||
|
||||
18/18 runs present with `dataset=5gnidd, alpha=0.25`, 10 clients, 10 rounds, 3 local epochs,
|
||||
batch 256, lr 1e-3, mu 0.1, lambda 1.0; all exit status 0.
|
||||
|
||||
---
|
||||
|
||||
## C. Frozen protocol
|
||||
|
||||
Identical to `EXPERIMENT_PROTOCOL.md` §14: 10 clients, 10 rounds, 3 local epochs, batch 256, Adam
|
||||
`lr=1e-3`, `mu=0.1`, `lambda=1.0`; unchanged architecture, preprocessing, split recipe, Dirichlet
|
||||
partition recipe (`per_class_dirichlet_multinomial`), loss and nearest-prototype inference. All 18
|
||||
runs used `scripts/run_baseline.py` at HEAD `bbd35085063194db58e4472e70b204d4be4a3607`.
|
||||
|
||||
Selection rule (unchanged):
|
||||
|
||||
- **PRIMARY:** pooled-across-seed validation per-client-mean rare-class macro-F1, validation only.
|
||||
- **SECONDARY (diagnostic):** per-seed validation selection.
|
||||
- **TEST:** read only after the pooled beta was fixed; `test_used_for_selection = false` for every
|
||||
`support_weighted` run.
|
||||
|
||||
---
|
||||
|
||||
## D. Partition characterization (realized, before interpretation)
|
||||
|
||||
All statistics are from the training-split `N_ij`; `M = 10` clients; `|S_j| < M` means fewer than
|
||||
10 clients contribute a prototype for a class.
|
||||
|
||||
### D.1 Per-seed summary (alpha=0.25)
|
||||
|
||||
| seed | `|S_j| < M` cells | missing client-class cells | client-size min/max (ratio) | union rare | union unseen | max unseen on a client |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 0 | 6 / 9 | 10 | 23,943 / 237,693 (9.93) | 6 | 6 | 3 |
|
||||
| 1 | 5 / 9 | 10 | 7,399 / 205,900 (27.83) | 6 | 5 | 3 |
|
||||
| 2 | 5 / 9 | 10 | 19,817 / 298,192 (15.05) | 8 | 5 | 2 |
|
||||
| **total / mean** | **16 / 27** | **30** | ratio mean **17.60** | 6/6/8 | 6/5/5 | 3 |
|
||||
|
||||
- `alpha=0.25` is the **most heterogeneous** of the three 5G-NIDD conditions: 16/27 class-condition
|
||||
cells have `|S_j| < M`, 30 missing `(client,class)` cells, mean client-size ratio 17.60.
|
||||
- The **rare-class union is not identical across seeds**: seed 0 = `{ICMPFlood, HTTPFlood,
|
||||
SYNFlood, TCPConnectScan, SlowrateDoS, UDPScan}`; seed 1 = `{ICMPFlood, HTTPFlood, SYNFlood,
|
||||
SYNScan, TCPConnectScan, UDPFlood}`; seed 2 = all eight attack classes except none — seed 2 has
|
||||
**8** union-rare classes. Consequently the global test rare support differs by seed
|
||||
(141,677 / 129,824 / 147,630) and cross-seed test levels are **not** directly comparable.
|
||||
- All 10 clients have exactly two rare classes in every seed and are eligible.
|
||||
- Union unseen classes: seed 0 = 6 distinct (`HTTPFlood, ICMPFlood, SYNFlood, SYNScan,
|
||||
SlowrateDoS, TCPConnectScan`), seed 1 = 5, seed 2 = 5; up to **3** unseen classes on a single
|
||||
client (vs max 1 at alpha=0.50/0.75). This is a qualitative shift: at low alpha whole classes
|
||||
vanish from a client.
|
||||
- Machine-readable: `artifacts/analysis/5gnidd__alpha0.25__partition_characterization.csv` and the
|
||||
cross-alpha `5gnidd__cross_alpha__partition_heterogeneity_*` files.
|
||||
|
||||
### D.2 Per-class support inequality
|
||||
|
||||
Mean class-support CV across seeds = **1.302** (seed 0 1.440, seed 1 1.140, seed 2 1.327); the
|
||||
maximum per-class `max/min` ratio across seeds is 185,504 (Benign seed 0, driven by one client with
|
||||
a single sample). The rare/attack classes are strongly imbalanced at every seed. Complete per-class
|
||||
tables: `5gnidd__cross_alpha__partition_heterogeneity_per_class.csv`.
|
||||
|
||||
### D.3 Comparison with the audited 5G-NIDD alpha=0.50 and alpha=0.75
|
||||
|
||||
| statistic (training split) | 5G-NIDD a0.25 | 5G-NIDD a0.50 | 5G-NIDD a0.75 |
|
||||
|---|---|---|---|
|
||||
| `|S_j| < M` cells (sum / 27) | **16 / 27** | 4 / 27 | 4 / 27 |
|
||||
| missing client-class cells | **30** | 4 | 5 |
|
||||
| client-size ratio (seeds) | 9.93 / 27.83 / 15.05 | 11.63 / 4.43 / 12.72 | 3.70 / 11.36 / 3.87 |
|
||||
| client-size ratio mean | **17.60** | 9.59 | 6.31 |
|
||||
| mean class-support CV | **1.302** | 1.225 | 0.916 |
|
||||
| union rare per seed | 6 / 6 / 8 | 6 / 5 / 6 | 6 / 6 / 6 |
|
||||
| union unseen per seed | 6 / 5 / 5 | 2 / 0 / 2 | 2 / 2 / 0 |
|
||||
| max unseen on a client | **3** | 1 | 1 |
|
||||
|
||||
Realized heterogeneity increases markedly as alpha falls. Note the client-size ratio is **not**
|
||||
uniformly ordered by seed (seed 0's ratio at alpha=0.50 is *higher* than at alpha=0.25), so these
|
||||
are realized statistics and are used for interpretation instead of alpha alone.
|
||||
|
||||
---
|
||||
|
||||
## E. Per-seed validation results
|
||||
|
||||
`val rareMacroF1 (pc mean)` is the primary metric; all 10 clients eligible in every seed.
|
||||
Machine-readable: `artifacts/analysis/5gnidd__alpha0.25__multiseed__validation_per_seed.csv`.
|
||||
|
||||
### Seed 0
|
||||
|
||||
| beta | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|
|
||||
| **0.00** | **0.736360** | 0.863272 | 0.984162 | 0.831244 | 0.746359 |
|
||||
| 0.25 | **0.747299** | 0.883896 | 0.978293 | 0.847782 | 0.741490 |
|
||||
| 0.50 | 0.717450 | 0.822592 | 0.981845 | 0.806414 | 0.746383 |
|
||||
| 0.75 | 0.713262 | 0.803765 | 0.974314 | 0.804768 | 0.750808 |
|
||||
| 1.00 | 0.734455 | 0.804729 | 0.935751 | 0.820900 | 0.754986 |
|
||||
|
||||
### Seed 1
|
||||
|
||||
| beta | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|
|
||||
| **0.00** | **0.771268** | 0.820509 | 0.974835 | 0.807348 | 0.737476 |
|
||||
| 0.25 | 0.767300 | 0.827350 | 0.970492 | 0.828781 | 0.751515 |
|
||||
| 0.50 | 0.738223 | 0.727485 | 0.844435 | 0.720389 | 0.666919 |
|
||||
| 0.75 | 0.728478 | 0.682389 | 0.759538 | 0.702449 | 0.639844 |
|
||||
| 1.00 | 0.739297 | 0.716268 | 0.799153 | 0.729077 | 0.671845 |
|
||||
|
||||
### Seed 2
|
||||
|
||||
| beta | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|
|
||||
| **0.00** | **0.776732** | 0.842370 | 0.922437 | 0.810709 | 0.708074 |
|
||||
| 0.25 | 0.733259 | 0.778933 | 0.901942 | 0.754454 | 0.688138 |
|
||||
| 0.50 | 0.721471 | 0.744221 | 0.893070 | 0.723604 | 0.679272 |
|
||||
| 0.75 | 0.724966 | 0.774682 | 0.912452 | 0.750839 | 0.697703 |
|
||||
| 1.00 | 0.739157 | 0.825132 | 0.951026 | 0.795799 | 0.734014 |
|
||||
|
||||
The per-seed curves are **non-monotone and seed-dependent**, but differ from the higher-alpha
|
||||
conditions: seed 0 peaks at 0.25 and again at 1.00 with a dip at 0.50–0.75; seed 1 is highest at
|
||||
0.00 and then falls; seed 2 is highest at 0.00. Increasing beta generally **reduces** the primary
|
||||
metric at this condition, with the notable exception of seed 0's small peak at 0.25. Seed 1 at
|
||||
beta ≥ 0.50 shows a large global-metric collapse (val accuracy 0.67–0.64), consistent with a
|
||||
rare-class-dominated partition where emphasizing high-support Benign/UDPFlood prototypes hurts
|
||||
nearest-prototype inference.
|
||||
|
||||
---
|
||||
|
||||
## F. Per-seed diagnostic beta choices (SECONDARY, stability only)
|
||||
|
||||
| seed | selected beta | val rareMacroF1 (pc mean) | tie-break used | test_used_for_selection |
|
||||
|---|---|---|---|---|
|
||||
| 0 | 0.25 | 0.747299 | no | false |
|
||||
| 1 | 0.00 | 0.771268 | no | false |
|
||||
| 2 | 0.00 | 0.776732 | no | false |
|
||||
|
||||
These per-seed choices are **diagnostic only**; they are not the condition-level selection rule.
|
||||
`val_selection.json` records `"selection_split": "validation"` and
|
||||
`"test_used_for_selection": false` for all 15 `support_weighted` runs.
|
||||
|
||||
---
|
||||
|
||||
## G. PRIMARY pooled-across-seed validation selection
|
||||
|
||||
| beta | pooled mean | std (sample, ddof=1) | min | max | pooled val rareRecall (pc mean) | selected |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **0.00** | **0.761453** | 0.021903 | 0.736360 | 0.776732 | 0.908577 | **yes** |
|
||||
| 0.25 | 0.749286 | 0.017107 | 0.733259 | 0.767300 | 0.907137 | |
|
||||
| 0.50 | 0.725715 | 0.011017 | 0.717450 | 0.738223 | 0.894947 | |
|
||||
| 0.75 | 0.722235 | 0.007967 | 0.713262 | 0.728478 | 0.879711 | |
|
||||
| 1.00 | 0.737636 | 0.002756 | 0.734455 | 0.739297 | 0.888991 | |
|
||||
|
||||
- **PRIMARY selected beta (`beta_star_5gnidd_alpha025`) = 0.00.**
|
||||
- **Criterion:** maximum pooled validation per-client-mean rare-class macro-F1 (validation only);
|
||||
tie-break order higher pooled rare recall → lower across-seed std → smaller beta.
|
||||
- **Tie-break used:** no. Top-two beta = 0.00 (0.761453) vs 0.25 (0.749286); gap ≈ **1.2e-2**.
|
||||
- **`test_used_for_selection = false`.** Test metrics were not inspected until the pooled beta was
|
||||
fixed.
|
||||
- **Interpretation:** `beta = 0.00` is the *best among the tested candidates under pooled
|
||||
validation across seeds `{0,1,2}` for 5G-NIDD alpha=0.25*. It is **not** a global optimum. The
|
||||
pooled validation curve decreases from 0.761 at beta=0 to a minimum of 0.722 at beta=0.75, then
|
||||
rises slightly to 0.738 at beta=1.00, so it is **non-monotone but overall declining**. The total
|
||||
between-beta spread is ≈ **0.039**, while the across-seed sample std ranges 0.0028–0.0219. Unlike
|
||||
alpha=0.50/0.75, at alpha=0.25 the across-seed std at beta=1.00 (0.0028) is *smaller* than the
|
||||
between-beta spread, i.e. the seeds agree more closely at high beta; at beta=0.00 the std
|
||||
(0.0219) is comparable to the spread. The pooled win of beta=0 is driven mainly by seed 2
|
||||
(0.777 vs 0.733 at beta=0.25) and seed 1 (0.771 vs 0.767).
|
||||
- Selecting `beta=0.00` means the selected aggregation is the **equal mean over the shared `S_j`**,
|
||||
i.e. mathematically the same mechanism as the `beta=0` control and numerically ~equal to the
|
||||
frozen baseline (see §J). This makes the client-level "vs beta=0" delta trivially zero at the
|
||||
selected beta.
|
||||
|
||||
Machine-readable: `artifacts/analysis/5gnidd__alpha0.25__multiseed__validation_pooled.csv`,
|
||||
`...__selection_summary.json`.
|
||||
|
||||
---
|
||||
|
||||
## H. Final selected-beta test metrics (descriptive only)
|
||||
|
||||
`beta_star = 0.00` was fixed from pooled validation; only then were test metrics read.
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.859893 | 0.813889 | 0.849925 | 0.841236 ± 0.024202 |
|
||||
| rare-class recall | 0.985234 | 0.972659 | 0.925756 | 0.961216 ± 0.031347 |
|
||||
| macro-F1 | 0.829680 | 0.802987 | 0.817254 | 0.816640 ± 0.013357 |
|
||||
| accuracy | 0.744844 | 0.737779 | 0.708052 | 0.730225 ± 0.019525 |
|
||||
| weighted-F1 | 0.717404 | 0.712302 | 0.683684 | 0.704463 ± 0.018175 |
|
||||
|
||||
The global rare evaluation support differs by seed (**141,677 / 129,824 / 147,630**) because the
|
||||
training-derived union of rare classes differs at this alpha; cross-seed absolute test levels are
|
||||
therefore **not** directly comparable. Test columns:
|
||||
`artifacts/analysis/5gnidd__alpha0.25__multiseed__test_descriptive.csv`.
|
||||
|
||||
---
|
||||
|
||||
## I. `original_protean` comparison
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.864903 | 0.814848 | 0.847072 | 0.842274 ± 0.025370 |
|
||||
| rare-class recall | 0.985444 | 0.972333 | 0.923890 | 0.960556 ± 0.032423 |
|
||||
| macro-F1 | 0.835043 | 0.808373 | 0.814800 | 0.819405 ± 0.013919 |
|
||||
| accuracy | 0.745280 | 0.741295 | 0.706329 | 0.730968 ± 0.021431 |
|
||||
| weighted-F1 | 0.718041 | 0.716327 | 0.681939 | 0.705435 ± 0.020367 |
|
||||
|
||||
Descriptive difference (`beta_star` − `original_protean`) for test rare-class macro-F1 is
|
||||
**−0.005010** (seed 0), **−0.000959** (seed 1), **+0.002853** (seed 2); mean **−0.001039**. As
|
||||
expected for a selected beta of 0, the selected-beta test metrics are numerically very close to the
|
||||
baseline, with small floating-point divergence (see §J). No superiority or generalisation is
|
||||
claimed. Descriptive dataset-level observation (consistent with alpha=0.50/0.75): on 5G-NIDD test
|
||||
weighted-F1 (`≈0.704`) sits **below** accuracy (`≈0.730`), the opposite of X-IIoTID.
|
||||
|
||||
---
|
||||
|
||||
## J. `beta = 0` equivalence vs `original_protean`
|
||||
|
||||
| property | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| same manifest / split / hashes | yes | yes | yes |
|
||||
| round-0 snapshot byte-identical | yes (`e2243ac4…`) | yes (`26a16299…`) | yes (`e287bff5…`) |
|
||||
| support / characterization / rare_unseen identical | yes | yes | yes |
|
||||
| independent aggregation diff (`ΣC/n` vs `(1/n)ΣC`) | 4.4e-16 | 4.4e-16 | 6.7e-16 |
|
||||
| history rare-macro-F1 round-0 diff | 0.0 | 0.0 | 0.0 |
|
||||
| history rare-macro-F1 max diff (rounds ≥ 1) | 2.17e-2 | 6.85e-2 | 2.74e-2 |
|
||||
| test metric max abs diff | 5.36e-3 | 5.39e-3 | 2.85e-3 |
|
||||
| **classification** | **B** | **B** | **B** |
|
||||
|
||||
Class **B**: semantic equivalence + floating-point numerical divergence; **no semantic
|
||||
discrepancy**. Round 0 is byte-identical; a ~1 ULP ordering difference enters round-1
|
||||
prototype-alignment training and the trajectories then diverge. Because the PRIMARY selection at
|
||||
this condition chose `beta=0.00`, the *selected* run is itself the `support_weighted beta=0` run,
|
||||
and the selected-beta test metrics differ from `original_protean` only by this floating-point
|
||||
divergence. Equivalence remains conditional on the shared `S_j` reproduction assumption and is
|
||||
**not** proof that `beta=0` reproduces original PROTEAN. Machine-readable:
|
||||
`artifacts/analysis/5gnidd__alpha0.25__multiseed__beta0_equivalence.csv`.
|
||||
|
||||
---
|
||||
|
||||
## K. Direct prototype-effect findings (fixed round-0 snapshots, no retraining)
|
||||
|
||||
Computed from each seed's fixed `{C_ij, N_ij}` round-0 snapshot via the production aggregation.
|
||||
Cross-seed mean `||Cbar_j(beta) − Cbar_j(0)||` at `beta = 1.0` (sample std across seeds):
|
||||
|
||||
| class | mean disp | std | mean rel disp |
|
||||
|---|---|---|---|
|
||||
| SYNScan | 3.815 | 1.402 | 0.544 |
|
||||
| HTTPFlood | 3.760 | 1.539 | 0.558 |
|
||||
| TCPConnectScan | 3.258 | 2.574 | 0.887 |
|
||||
| SlowrateDoS | 3.179 | 1.157 | 0.562 |
|
||||
| SYNFlood | 3.176 | 1.007 | 0.917 |
|
||||
| UDPScan | 2.816 | 1.604 | 0.371 |
|
||||
| Benign | 2.648 | 0.519 | 0.482 |
|
||||
| UDPFlood | 2.408 | 0.578 | 0.477 |
|
||||
| ICMPFlood | 1.215 | 0.151 | 0.347 |
|
||||
|
||||
- Mean displacement across the nine classes is **2.919** at beta=1.0, monotone in beta
|
||||
(0.25 → 1.295, 0.50 → 2.055, 0.75 → 2.559, 1.00 → 2.919).
|
||||
- Displacement is largest at this condition (2.919) among the three 5G-NIDD alphas and highest for
|
||||
the imbalanced rare/scan classes; `ICMPFlood` (very low support) and `UDPFlood` (very high
|
||||
support) move least.
|
||||
- Displacement is a **direct-aggregation measure only** and is **not** interpreted as improved or
|
||||
degraded detection. The end-to-end effect is reported separately in §E/§H.
|
||||
|
||||
Outputs: `artifacts/analysis/5gnidd__alpha0.25__seed{0,1,2}__support_weighted__beta0__prototype_snapshot__prototype_effect.csv`
|
||||
and `...__prototype_weights.csv`; cross-seed aggregate
|
||||
`artifacts/analysis/5gnidd__alpha0.25__multiseed__prototype_effect_pooled.csv`.
|
||||
|
||||
---
|
||||
|
||||
## L. Client-level effect (validation, relative to beta = 0)
|
||||
|
||||
Detail: `...__client_validation_detail.csv`; summary: `...__client_validation_summary.csv`. Client
|
||||
ids are per-seed realisations and are **not** pooled across seeds. Tolerance 1e-9.
|
||||
|
||||
**At the selected `beta = 0.00` the delta is trivially 0 for all clients in all seeds** (selected
|
||||
beta = reference beta). To describe the response, here is the full curve of mean Δ per seed:
|
||||
|
||||
| beta | seed 0 (imp/deg/unch, mean Δ) | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| 0.00 | 0/0/10, 0.0000 | 0/0/10, 0.0000 | 0/0/10, 0.0000 |
|
||||
| 0.25 | 3/3/4, +0.0109 | 3/3/4, −0.0040 | 0/4/6, −0.0435 |
|
||||
| 0.50 | 3/4/3, −0.0189 | 2/4/4, −0.0330 | 1/3/6, −0.0553 |
|
||||
| 0.75 | 3/4/3, −0.0231 | 2/5/3, −0.0428 | 1/4/5, −0.0518 |
|
||||
| 1.00 | 3/4/3, −0.0019 | 3/4/3, −0.0320 | 4/3/3, −0.0376 |
|
||||
|
||||
- At a common non-zero `beta = 0.75`, improve/degrade/unchanged = **3/4/3** (seed 0, mean
|
||||
−0.023), **2/5/3** (seed 1, mean −0.043), **1/4/5** (seed 2, mean −0.052) — i.e. at low alpha
|
||||
emphasizing support generally **degrades** the rare-class metric, the opposite of the higher
|
||||
alpha conditions and consistent with the pooled selection of beta=0.
|
||||
- `N_ij` vs ΔF1 is **non-monotonic**: at beta=0.75 the worst seed-1 degradation (c5, −0.406) has
|
||||
min positive attack `N_ij = 2`, but the best seed-0 gain (c5, +0.081) has min `N_ij = 1`; the
|
||||
worst seed-2 degradation (c9, −0.268) has min `N_ij = 33`.
|
||||
- This analysis is descriptive and non-causal.
|
||||
|
||||
---
|
||||
|
||||
## M. Runtime and resources
|
||||
|
||||
CPU only, no GPU, no swap, exit status 0 for every run (`/usr/bin/time -v`). Full table:
|
||||
`artifacts/analysis/5gnidd__alpha0.25__multiseed__runtime.csv`.
|
||||
|
||||
| group | wall range | CPU % | peak RSS |
|
||||
|---|---|---|---|
|
||||
| seed 0 (6 runs) | 62:00 – 63:28 | 743 | 13.18 – 13.20 GiB |
|
||||
| seed 1 (6 runs) | 64:16 – 64:48 | 740–742 | 11.92 – 11.95 GiB |
|
||||
| seed 2 (6 runs) | 62:57 – 64:28 | 739–741 | 16.09 – 16.11 GiB |
|
||||
|
||||
- New compute this session: **18 runs, 68,798 s ≈ 19.11 h** wall-clock, sequential on an 8-core
|
||||
host.
|
||||
- Per-run wall time is homogeneous (62:00–64:48). Peak RSS is partition-dependent, not
|
||||
beta-dependent: ~13.2 GiB (seed 0), ~11.9 GiB (seed 1), ~16.1 GiB (seed 2), all within the host
|
||||
memory. 5G-NIDD uses more memory than X-IIoTID because the training split is ~1.5× larger.
|
||||
- No warnings, no client failures, 9/9 global prototypes in every round, no retries.
|
||||
|
||||
---
|
||||
|
||||
## N. Comparison with 5G-NIDD alpha=0.50 and alpha=0.75 (descriptive only)
|
||||
|
||||
| statistic | 5G-NIDD a0.25 | 5G-NIDD a0.50 | 5G-NIDD a0.75 |
|
||||
|---|---|---|---|
|
||||
| pooled-selected beta | **0.00** | 1.00 | 0.50 |
|
||||
| pooled validation mean range (spread) | 0.722–0.761 (0.039) | 0.645–0.692 (0.047) | 0.778–0.809 (0.031) |
|
||||
| pooled validation std range | 0.003–0.022 | 0.053–0.139 | 0.051–0.065 |
|
||||
| `|S_j| < M` cells (sum / 27) | **16 / 27** | 4 / 27 | 4 / 27 |
|
||||
| missing client-class cells | **30** | 4 | 5 |
|
||||
| client-size ratio mean | **17.60** | 9.59 | 6.31 |
|
||||
| mean class-support CV | **1.302** | 1.225 | 0.916 |
|
||||
| union rare per seed | 6 / 6 / 8 | 6 / 5 / 6 | 6 / 6 / 6 |
|
||||
| union unseen per seed | 6 / 5 / 5 | 2 / 0 / 2 | 2 / 2 / 0 |
|
||||
| test rare support per seed | 141,677 / 129,824 / 147,630 | 104,843 / 13,375 / 128,995 | 28,000 / 28,000 / 28,000 |
|
||||
| mean displacement `‖Cbar(1)−Cbar(0)‖` | **2.919** | 2.340 | 1.385 |
|
||||
| selected-beta client improve/degrade | 0/0 (trivial at beta=0) | 6/3, 3/5, 7/2 | 4/6, 6/3, 4/2 |
|
||||
| `beta=0` equivalence class (per seed) | B / B / B | B / B / B | B / B / B |
|
||||
|
||||
Descriptive observations (three alpha levels, one dataset, three project-defined seeds — **no
|
||||
monotone or causal claim**):
|
||||
|
||||
- All three pooled validation curves are **non-monotone**, and the selected beta differs across
|
||||
conditions (0.00 at alpha=0.25, 1.00 at alpha=0.50, 0.50 at alpha=0.75). The three selected
|
||||
values are **not** monotone in alpha.
|
||||
- Realized support imbalance is largest at alpha=0.25 (CV 1.302, client ratio 17.60, 30 missing
|
||||
cells, 16/27 `|S_j|<M` cells) and mean prototype displacement is also largest (2.919), tracking
|
||||
the mechanical `N_ij^beta` sensitivity.
|
||||
- At alpha=0.25 the selection of `beta=0` coincides with the best pooled validation rarity, i.e.
|
||||
the support-weighted mechanism does **not** help rare-class validation at this condition; this is
|
||||
the opposite of alpha=0.50. Selection remains statistically weak (between-beta spread 0.039 vs
|
||||
across-seed std up to 0.022).
|
||||
- Validation **levels** are not directly comparable across alpha (different rare-class sets and
|
||||
per-client eval partitions); only within-condition contrasts are interpreted.
|
||||
|
||||
Machine-readable: `artifacts/analysis/5gnidd__cross_alpha__cross_alpha_beta_table.csv`,
|
||||
`..._displacement.csv`, `..._client_summary.csv`, `..._summary.json` and
|
||||
`5gnidd__cross_alpha__partition_heterogeneity_*`.
|
||||
|
||||
---
|
||||
|
||||
## O. Limitations
|
||||
|
||||
- **One new condition** (5G-NIDD alpha=0.25). This is the final 5G-NIDD condition, so the 5G-NIDD
|
||||
three-alpha grid is complete; a cross-alpha comparison is now possible but remains descriptive.
|
||||
No X-IIoTID/5G-NIDD causal or monotone claim.
|
||||
- **Three project-defined seeds** (`{0,1,2}`); the pooled selection at this condition has
|
||||
between-beta spread larger than the across-seed std, but the result is still a weak descriptive
|
||||
preference, not a strong effect. `beta_star = 0.00` is only "best among the five tested
|
||||
candidates under pooled validation for this condition".
|
||||
- The `S_j` participant/denominator rule (`S_j = {i | N_ij > 0 and valid C_ij}`, equal weights in
|
||||
the baseline) remains a **REPRODUCTION ASSUMPTION**. At 5G-NIDD alpha=0.25 it is material in
|
||||
**16/27** class-condition cells, the largest of the three conditions; the `beta=0` comparison is
|
||||
conditional on both modes sharing this rule and does **not** prove `beta=0` reproduces original
|
||||
PROTEAN.
|
||||
- Train/validation/test adaptation (70/10/20 stratified) differs from the paper's literal 80/20;
|
||||
the validation eval partition is a project decision.
|
||||
- `preprocessing_hash`/`config_hash` remain `null` in the 5G-NIDD manifests.
|
||||
- Selecting `beta=0` makes the selected mode arithmetically the equal-`S_j` mean, so this condition
|
||||
yields no evidence that support weighting improves rare-class detection.
|
||||
- **No global beta optimum claim**; test metrics did not influence selection.
|
||||
@@ -0,0 +1,262 @@
|
||||
# 5G-NIDD Cross-Alpha Analysis (alpha = 0.25 / 0.50 / 0.75)
|
||||
|
||||
Status: **descriptive three-condition comparison** of the frozen beta protocol on 5G-NIDD, using
|
||||
the audited condition-level pooled results of Session 11 (`alpha=0.25`), Session 9 (`alpha=0.50`)
|
||||
and Session 10 (`alpha=0.75`). This document does **not** run anything and does not make a causal
|
||||
or monotonicity claim. Machine-readable counterpart:
|
||||
`artifacts/analysis/5gnidd__cross_alpha__cross_alpha_beta_table.csv`, `..._summary.json`,
|
||||
`..._displacement.csv`, `..._client_summary.csv`, and
|
||||
`artifacts/analysis/5gnidd__cross_alpha__partition_heterogeneity_*.csv`.
|
||||
|
||||
Companion documents: `docs/research/5GNIDD_ALPHA025_BETA_VALIDATION.md`,
|
||||
`docs/research/5GNIDD_ALPHA050_BETA_VALIDATION.md`,
|
||||
`docs/research/5GNIDD_ALPHA075_BETA_VALIDATION.md`,
|
||||
`docs/research/EXPERIMENT_PROTOCOL.md`, `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md`,
|
||||
`docs/research/CROSS_DATASET_BETA_ANALYSIS.md`, `docs/handoff/SESSION_HANDOFF.md`.
|
||||
|
||||
---
|
||||
|
||||
## A. Experimental scope
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| dataset | 5G-NIDD (`5gnidd`) |
|
||||
| alpha | 0.25, 0.50, 0.75 |
|
||||
| seeds | 0, 1, 2 |
|
||||
| betas | 0.00, 0.25, 0.50, 0.75, 1.00 |
|
||||
| aggregation | `original_protean` + `support_weighted` |
|
||||
| selection | pooled validation-only per condition, seeds `{0,1,2}` |
|
||||
| test | read only after each condition's pooled beta is fixed |
|
||||
|
||||
All three conditions share the frozen training protocol: 10 clients, 10 rounds, 3 local epochs,
|
||||
batch 256, Adam `lr=1e-3`, `mu=0.1`, `lambda=1.0`, unchanged architecture/preprocessing/split
|
||||
recipe/loss/inference. For each `(alpha, seed)` all six runs share one manifest and a
|
||||
byte-identical round-0 snapshot. 54 runs total (18 per condition).
|
||||
|
||||
---
|
||||
|
||||
## B. Realized heterogeneity
|
||||
|
||||
`|S_j| < M` counts class-condition cells with fewer than 10 contributing clients, summed over
|
||||
the three seeds (out of 27). Missing cells count `(client, class)` pairs with `N_ij = 0`.
|
||||
|
||||
| statistic | alpha=0.25 | alpha=0.50 | alpha=0.75 |
|
||||
|---|---|---|---|
|
||||
| `|S_j| < M` cells (sum over 3 seeds) | **16 / 27** | 4 / 27 | 4 / 27 |
|
||||
| missing client-class cells | **30** | 4 | 5 |
|
||||
| client-size ratio (seed 0/1/2) | 9.93 / 27.83 / 15.05 | 11.63 / 4.43 / 12.72 | 3.70 / 11.36 / 3.87 |
|
||||
| client-size ratio mean | **17.60** | 9.59 | 6.31 |
|
||||
| union rare classes per seed | 6 / 6 / 8 | 6 / 5 / 6 | 6 / 6 / 6 |
|
||||
| union unseen classes per seed | 6 / 5 / 5 | 2 / 0 / 2 | 2 / 2 / 0 |
|
||||
| max unseen classes on a client | **3** | 1 | 1 |
|
||||
| mean class-support CV | **1.302** | 1.225 | 0.916 |
|
||||
| max per-class max/min ratio (over seeds) | 185,504 | 88,943 | 7,642.5 |
|
||||
|
||||
Realized heterogeneity increases as alpha falls, but **not uniformly per seed** (e.g. at
|
||||
alpha=0.50 seed 0 the client ratio 11.63 exceeds alpha=0.25 seed 0's 9.93). Interpretation uses
|
||||
these realized partitions, not alpha alone. The rarest attack class (`ICMPFlood`, 1,155 records
|
||||
full-data) is always rare; the rare-set union is seed-dependent only at alpha=0.25, where seed 2
|
||||
has 8 union-rare classes.
|
||||
|
||||
---
|
||||
|
||||
## C. Pooled validation curves (alpha × beta)
|
||||
|
||||
Pooled validation per-client-mean rare-class macro-F1 (selection metric; validation only):
|
||||
|
||||
| beta | alpha=0.25 mean (std) | alpha=0.50 mean (std) | alpha=0.75 mean (std) |
|
||||
|---|---|---|---|
|
||||
| **0.00** | **0.761453** (0.021903) | 0.687230 (0.052726) | 0.778094 (0.050963) |
|
||||
| 0.25 | 0.749286 (0.017107) | 0.645078 (0.064743) | 0.790483 (0.055100) |
|
||||
| 0.50 | 0.725715 (0.011017) | 0.674669 (0.102253) | **0.809081** (0.058735) |
|
||||
| 0.75 | 0.722235 (0.007967) | 0.688368 (0.126032) | 0.791530 (0.064974) |
|
||||
| 1.00 | 0.737636 (0.002756) | **0.692370** (0.139140) | 0.804306 (0.054803) |
|
||||
| **curve shape** | declines to 0.75, slight rise at 1.0 | rises to peak at 1.0 | peaks at 0.50, declines both sides |
|
||||
| **total between-beta spread** | 0.0392 | 0.0473 | 0.0308 |
|
||||
|
||||
The across-seed sample std (0.003–0.139) is comparable to or larger than the between-beta spread at
|
||||
alpha=0.50 and alpha=0.75; at alpha=0.25 the between-beta spread (0.039) exceeds the across-seed
|
||||
std at every beta except beta=0.00, so the alpha=0.25 selection is relatively less seed-noise
|
||||
dominated — but it is still only three project-defined seeds and needs no causal reading.
|
||||
|
||||
---
|
||||
|
||||
## D. Selected beta by condition
|
||||
|
||||
Condition-level **pooled** selections only (per-seed diagnostics are excluded):
|
||||
|
||||
| condition | pooled-selected beta | pooled validation mean | tie-break used | per-seed diagnostic betas |
|
||||
|---|---|---|---|---|
|
||||
| alpha=0.25 | **0.00** | 0.761453 | no | 0.25 / 0.00 / 0.00 |
|
||||
| alpha=0.50 | **1.00** | 0.692370 | no | 1.00 / 0.00 / 1.00 |
|
||||
| alpha=0.75 | **0.50** | 0.809081 | no | 0.50 / 1.00 / 1.00 |
|
||||
|
||||
The selected beta **changes with the non-IID condition but is not monotone** in alpha
|
||||
(0.00 → 1.00 → 0.50). At alpha=0.25 and alpha=0.75 the per-seed diagnostics partly disagree with
|
||||
the pooled choice, confirming seed sensitivity.
|
||||
|
||||
---
|
||||
|
||||
## E. Selected-beta test summaries (descriptive)
|
||||
|
||||
Selected beta vs `original_protean`, mean ± sample std across seeds `{0,1,2}`.
|
||||
|
||||
| condition | selected beta | test rare macro-F1 (selected) | test rare macro-F1 (original) | test macro-F1 (selected) | test macro-F1 (original) | test acc (selected) | test acc (original) |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| alpha=0.25 | 0.00 | 0.841236 ± 0.024202 | 0.842274 ± 0.025370 | 0.816640 ± 0.013357 | 0.819405 ± 0.013919 | 0.730225 ± 0.019525 | 0.730968 ± 0.021431 |
|
||||
| alpha=0.50 | 1.00 | 0.871984 ± 0.028501 | 0.880542 ± 0.037058 | 0.838596 ± 0.019627 | 0.842395 ± 0.020840 | 0.743257 ± 0.000667 | 0.743975 ± 0.008393 |
|
||||
| alpha=0.75 | 0.50 | 0.896738 ± 0.014470 | 0.879901 ± 0.023121 | 0.853150 ± 0.009572 | 0.841405 ± 0.014091 | 0.752274 ± 0.002503 | 0.750955 ± 0.003578 |
|
||||
|
||||
**Do not compare the absolute test rare-class macro-F1 across conditions**: the training-derived
|
||||
union of rare classes (and hence the global rare evaluation support) differs by alpha and seed
|
||||
(alpha=0.25 support 141,677 / 129,824 / 147,630; alpha=0.50 104,843 / 13,375 / 128,995;
|
||||
alpha=0.75 28,000 / 28,000 / 28,000). Only the selected-vs-original difference *within* a condition
|
||||
is meaningful, and even that is dominated by between-seed variance. At alpha=0.25 and alpha=0.50
|
||||
the selected beta is numerically essentially the baseline (selected−original rare macro-F1 mean
|
||||
−0.0010 and −0.0086); at alpha=0.75 it is +0.0168. No superiority or generalisation is claimed.
|
||||
|
||||
---
|
||||
|
||||
## F. `original_protean` summaries
|
||||
|
||||
`original_protean` test means (± sample std) across seeds:
|
||||
|
||||
| condition | rare macro-F1 | rare recall | macro-F1 | accuracy | weighted-F1 |
|
||||
|---|---|---|---|---|---|
|
||||
| alpha=0.25 | 0.842274 ± 0.025370 | 0.960556 ± 0.032423 | 0.819405 ± 0.013919 | 0.730968 ± 0.021431 | 0.705435 ± 0.020367 |
|
||||
| alpha=0.50 | 0.880542 ± 0.037058 | 0.972913 ± 0.003843 | 0.842395 ± 0.020840 | 0.743975 ± 0.008393 | 0.718730 ± 0.008678 |
|
||||
| alpha=0.75 | 0.879901 ± 0.023121 | 0.961214 ± 0.002144 | 0.841405 ± 0.014091 | 0.750955 ± 0.003578 | 0.726047 ± 0.004544 |
|
||||
|
||||
At every 5G-NIDD alpha, test weighted-F1 is **below** accuracy, i.e. the high-support
|
||||
`Benign`/`UDPFlood` classes are the harder ones for nearest-prototype inference — a dataset
|
||||
characteristic, not a beta effect.
|
||||
|
||||
---
|
||||
|
||||
## G. Direct prototype displacement (beta = 1 vs beta = 0)
|
||||
|
||||
Cross-seed mean `||Cbar_j(1) − Cbar_j(0)||` on the fixed round-0 snapshots (no retraining):
|
||||
|
||||
| class | alpha=0.25 | alpha=0.50 | alpha=0.75 |
|
||||
|---|---|---|---|
|
||||
| UDPScan | 2.816 | 3.461 | 2.445 |
|
||||
| SYNFlood | 3.176 | 2.262 | 1.897 |
|
||||
| SlowrateDoS | 3.179 | 2.763 | 1.565 |
|
||||
| SYNScan | 3.815 | 2.875 | 1.563 |
|
||||
| HTTPFlood | 3.760 | 3.120 | 1.366 |
|
||||
| TCPConnectScan | 3.258 | 2.131 | 1.226 |
|
||||
| UDPFlood | 2.408 | 1.598 | 0.903 |
|
||||
| Benign | 2.648 | 1.690 | 0.824 |
|
||||
| ICMPFlood | 1.215 | 1.159 | 0.673 |
|
||||
| **mean across classes** | **2.919** | **2.340** | **1.385** |
|
||||
|
||||
Displacement by beta (cross-class mean), showing monotone growth in beta within each condition:
|
||||
|
||||
| beta | alpha=0.25 | alpha=0.50 | alpha=0.75 |
|
||||
|---|---|---|---|
|
||||
| 0.25 | 1.295 | 0.953 | 0.497 |
|
||||
| 0.50 | 2.055 | 1.567 | 0.875 |
|
||||
| 0.75 | 2.559 | 2.003 | 1.163 |
|
||||
| 1.00 | 2.919 | 2.340 | 1.385 |
|
||||
|
||||
Mean displacement tracks realized support imbalance (alpha=0.25 > 0.50 > 0.75), but the per-class
|
||||
rank order is seed-sensitive and differs across alpha (e.g. `UDPScan` is largest at alpha=0.50 but
|
||||
not at alpha=0.25/0.75). Displacement is a direct-aggregation measure only; it is **not**
|
||||
interpreted as improved or degraded detection.
|
||||
|
||||
---
|
||||
|
||||
## H. Client-level behaviour
|
||||
|
||||
Improve/degrade/unchanged counts (validation, relative to `beta = 0`).
|
||||
|
||||
At each condition's pooled-selected beta:
|
||||
|
||||
| condition | selected beta | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|---|
|
||||
| alpha=0.25 | 0.00 | 0/0/10 (trivial) | 0/0/10 (trivial) | 0/0/10 (trivial) |
|
||||
| alpha=0.50 | 1.00 | 6/3/1 (+0.030) | 3/5/2 (−0.101) | 7/2/1 (+0.087) |
|
||||
| alpha=0.75 | 0.50 | 4/6/0 (+0.002) | 6/3/1 (+0.085) | 4/2/4 (+0.006) |
|
||||
|
||||
(mean Δ in parentheses.) At a common `beta = 0.75`:
|
||||
|
||||
| condition | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| alpha=0.25 | 3/4/3 | 2/5/3 | 1/4/5 |
|
||||
| alpha=0.50 | 5/4/1 | 3/5/2 | 7/2/1 |
|
||||
| alpha=0.75 | 3/7/0 | 9/0/1 | 5/5/0 |
|
||||
|
||||
The client-level response is seed-dependent at every alpha. At alpha=0.25 the response is
|
||||
predominantly negative at non-zero beta (support weighting degrades rare-class validation), whereas
|
||||
alpha=0.50/0.75 are mixed-to-positive. `N_ij` vs ΔF1 is non-monotonic everywhere. Client ids are
|
||||
per-seed realisations and are never pooled across seeds.
|
||||
|
||||
---
|
||||
|
||||
## I. Relation to RQ1 (support imbalance ↔ displacement ↔ rare-class performance)
|
||||
|
||||
Observed, descriptive:
|
||||
|
||||
- Realized support imbalance is highest at alpha=0.25 (mean class CV 1.302; `|S_j| < M` cells
|
||||
16/27; 30 missing cells; client ratio 17.60) and lowest at alpha=0.75 (0.916; 4/27; 5; 6.31),
|
||||
and mean prototype displacement follows the same ordering (2.919 / 2.340 / 1.385).
|
||||
Directionally consistent across the three conditions.
|
||||
- Rare-class validation *levels* do **not** follow the same ordering (pooled means ≈ 0.76 / 0.65–0.69
|
||||
/ 0.78–0.81). Absolute cross-condition levels are not controlled because the training-derived
|
||||
rare-class sets and per-client eval partitions differ.
|
||||
|
||||
**No causal conclusion** is drawn: the association between support imbalance and displacement is
|
||||
mechanical under the beta formula, while the relationship to rare-class performance is confounded
|
||||
by which classes are rare and how much support they have in each partition.
|
||||
|
||||
---
|
||||
|
||||
## J. Relation to RQ2 (how beta changes prototypes and rare-class performance)
|
||||
|
||||
- **Prototype position:** increasing beta moves every class prototype monotonically away from its
|
||||
beta=0 equal mean (by construction, because supports are unequal); the magnitude grows with
|
||||
support imbalance (mean class displacement at beta=1: 1.385 / 2.340 / 2.919 as imbalance grows).
|
||||
`|S_j| = 1` would leave the aggregate unchanged, but no class has a single contributor at any of
|
||||
these conditions.
|
||||
- **Rare-class performance:** the validation response is **non-monotone in beta** and
|
||||
condition-dependent: overall declining at alpha=0.25 (best at beta=0), peaking at beta=1.00 at
|
||||
alpha=0.50, peaking at beta=0.50 at alpha=0.75. The effect magnitude is comparable to or smaller
|
||||
than between-seed variance at alpha=0.50/0.75; at alpha=0.25 the between-beta spread exceeds the
|
||||
across-seed std, but the sample is only three seeds.
|
||||
|
||||
---
|
||||
|
||||
## K. Relation to RQ3 (does selected beta differ across non-IID conditions?)
|
||||
|
||||
Yes, descriptively: the pooled-selected beta is **0.00 (alpha=0.25), 1.00 (alpha=0.50),
|
||||
0.50 (alpha=0.75)**. It is **not monotone** in alpha.
|
||||
|
||||
Caveats that forbid a trend claim:
|
||||
|
||||
- only **three** alpha levels and **three** project-defined seeds, **one** dataset;
|
||||
- one selection per condition, each statistically weak or only modestly stronger than seed noise,
|
||||
and with **no tie-break** exercised at any condition;
|
||||
- the realized partitions differ, so the non-IID "condition" is only partially summarized by
|
||||
alpha;
|
||||
- **no inferential monotonic-trend test** has been performed.
|
||||
|
||||
RQ3 asks whether beta changes with conditions; it does not require a monotonic answer.
|
||||
|
||||
---
|
||||
|
||||
## L. Limitations
|
||||
|
||||
- One dataset (5G-NIDD), three alpha values, three seeds; cross-dataset claims are deferred to
|
||||
`CROSS_DATASET_BETA_ANALYSIS.md`.
|
||||
- The `S_j` participant/denominator rule remains a **REPRODUCTION ASSUMPTION** (material in
|
||||
16/27 cells at alpha=0.25, 4/27 at alpha=0.50, 4/27 at alpha=0.75); all comparisons are
|
||||
conditional on both modes sharing it.
|
||||
- Train/validation/test adaptation (70/10/20 stratified) differs from the paper's literal
|
||||
80/20; the validation eval partition is a project decision.
|
||||
- **No inferential significance testing**; no global beta optimum. Each selected beta is
|
||||
"best among the tested candidates under pooled validation for that condition".
|
||||
- Cross-alpha absolute metrics are not directly comparable because the rare-class sets, global
|
||||
rare support, and per-client eval partitions differ by condition/seed; only within-condition
|
||||
contrasts and relative patterns are interpreted.
|
||||
- Selecting beta=0 at alpha=0.25 makes the selected mode arithmetically the equal-`S_j` mean, so
|
||||
that condition cannot show a benefit of support weighting.
|
||||
@@ -0,0 +1,240 @@
|
||||
# Cross-Dataset Beta Analysis: X-IIoTID vs 5G-NIDD (alpha = 0.25 / 0.50 / 0.75)
|
||||
|
||||
Status: **descriptive six-condition comparison** of the frozen server-side prototype-aggregation
|
||||
beta protocol across two datasets and three Dirichlet non-IID levels, using only the audited
|
||||
condition-level pooled results:
|
||||
|
||||
- X-IIoTID: Session 6 (`alpha=0.50`), Session 7 (`alpha=0.25`), Session 8 (`alpha=0.75`);
|
||||
- 5G-NIDD: Session 9 (`alpha=0.50`), Session 10 (`alpha=0.75`), Session 11 (`alpha=0.25`).
|
||||
|
||||
This document runs nothing, uses **validation-only** selections, and makes **no causal, universal,
|
||||
or monotone** claim. Machine-readable counterpart:
|
||||
`artifacts/analysis/cross_dataset__matched_alpha_summary.csv` / `.json`,
|
||||
`artifacts/analysis/cross_dataset__beta_curve.csv`, plus the per-dataset
|
||||
`xiiotid__cross_alpha__*` and `5gnidd__cross_alpha__*` artifacts.
|
||||
|
||||
Companion documents: `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md`,
|
||||
`docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md`, `docs/research/EXPERIMENT_PROTOCOL.md`,
|
||||
`docs/research/DATASET_INSPECTION.md`, `docs/handoff/SESSION_HANDOFF.md`.
|
||||
|
||||
---
|
||||
|
||||
## A. Scope
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| datasets | X-IIoTID (`xiiotid`), 5G-NIDD (`5gnidd`) |
|
||||
| alpha | 0.25, 0.50, 0.75 (both datasets) |
|
||||
| seeds | 0, 1, 2 |
|
||||
| betas | 0.00, 0.25, 0.50, 0.75, 1.00 |
|
||||
| aggregation | `original_protean` + `support_weighted` |
|
||||
| conditions | 6 (2 datasets × 3 alphas × 3 seeds = 108 runs total across all sessions) |
|
||||
| selection | pooled validation-only per condition, seeds `{0,1,2}`; test read after fixing |
|
||||
| test_used_for_selection | false at all six conditions |
|
||||
|
||||
Both datasets share the frozen training protocol (10 clients, 10 rounds, 3 local epochs, batch 256,
|
||||
Adam `lr=1e-3`, `mu=0.1`, `lambda=1.0`) and the same frozen preprocessing recipe; the only
|
||||
dataset-specific inputs are the raw CSV, label column, class mapping and dropped labels.
|
||||
|
||||
---
|
||||
|
||||
## B. Dataset structural differences
|
||||
|
||||
| property | X-IIoTID | 5G-NIDD |
|
||||
|---|---|---|
|
||||
| benign label | `Normal` | `Benign` |
|
||||
| attack classes (8) | `C&C, Exfiltration, Exploitation, Lateral_movement, RDOS, Reconnaissance, Tampering, Weaponization` | `HTTPFlood, ICMPFlood, SYNFlood, SYNScan, SlowrateDoS, TCPConnectScan, UDPFlood, UDPScan` |
|
||||
| dropped label | `crypto-ransomware` | — |
|
||||
| training rows | 569,909 | 851,123 |
|
||||
| raw feature columns | 59 (56 numeric + 3 categorical) | 46 (41 numeric + 5 categorical) |
|
||||
| one-hot expanded width | 78 | 81 |
|
||||
|
||||
The two **rare-set compositions are qualitatively different**, and this must be kept explicit:
|
||||
|
||||
- 5G-NIDD rare classes are low-volume **flood/scan** categories (`ICMPFlood` is always rare;
|
||||
partners vary: `SYNFlood`, `SYNScan`, `TCPConnectScan`, `UDPScan`, `HTTPFlood`, `SlowrateDoS`).
|
||||
- X-IIoTID rare classes are low-volume **behavioural** categories (`Tampering`, `Exfiltration`,
|
||||
`Exploitation`, `Lateral_movement`, `C&C`, `Weaponization`, `Reconnaissance`, `RDOS`), whose
|
||||
identities and per-seed unions vary more. A "rare class" is therefore not a comparable object
|
||||
across the two datasets; only the protocol and the aggregation mechanism are shared.
|
||||
|
||||
Both datasets have 9 classes (8 attack + benign) and all 10 clients have exactly two rare classes
|
||||
per seed, so all clients are eligible for rare-class evaluation.
|
||||
|
||||
---
|
||||
|
||||
## C. Matched-alpha realized heterogeneity
|
||||
|
||||
`|S_j| < M` cells are summed over the three seeds (out of 27). Missing cells count `N_ij = 0`.
|
||||
|
||||
| alpha | dataset | `|S_j|<M` cells | missing cells | client-size ratio mean | mean class-support CV | union unseen mean | max unseen/client |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 0.25 | 5G-NIDD | 16 / 27 | 30 | 17.60 | 1.302 | 5.33 | 3 |
|
||||
| 0.25 | X-IIoTID | 19 / 27 | 38 | 14.75 | 1.442 | 5.67 | 4 |
|
||||
| 0.50 | 5G-NIDD | 4 / 27 | 4 | 9.59 | 1.225 | 1.33 | 1 |
|
||||
| 0.50 | X-IIoTID | 7 / 27 | 9 | 7.41 | 1.183 | 2.33 | 2 |
|
||||
| 0.75 | 5G-NIDD | 4 / 27 | 4–5 | 6.31 | 0.916 | 1.33 | 1 |
|
||||
| 0.75 | X-IIoTID | 3 / 27 | 4 | 4.67 | 0.881 | 1.00 | 1 |
|
||||
|
||||
At every matched alpha the two datasets are **quantitatively close but not identical** in realized
|
||||
heterogeneity: X-IIoTID has slightly more missing class cells and a slightly higher support CV at
|
||||
alpha=0.25/0.50, while 5G-NIDD has a higher mean client-size ratio at alpha=0.25/0.50/0.75. Both
|
||||
datasets become far more homogeneous as alpha rises, with the mixture of `|S_j|<M` cells
|
||||
collapsing between alpha=0.25 and alpha=0.50.
|
||||
|
||||
---
|
||||
|
||||
## D. Pooled beta selection across six conditions
|
||||
|
||||
Condition-level pooled selections (validation only):
|
||||
|
||||
| dataset | alpha | selected beta | pooled validation mean | tie-break | curve shape |
|
||||
|---|---|---|---|---|---|
|
||||
| 5G-NIDD | 0.25 | **0.00** | 0.761453 | no | declines to 0.75, small rise at 1.0 |
|
||||
| 5G-NIDD | 0.50 | **1.00** | 0.692370 | no | rises to peak at 1.0, dip at 0.25 |
|
||||
| 5G-NIDD | 0.75 | **0.50** | 0.809081 | no | peaks at 0.50, declines both sides |
|
||||
| X-IIoTID | 0.25 | **0.00** | 0.591704 | no | monotone decreasing |
|
||||
| X-IIoTID | 0.50 | **0.75** | 0.579367 | no | rises to peak at 0.75 |
|
||||
| X-IIoTID | 0.75 | **0.50** | 0.476749 | no | peaks at 0.50, declines both sides |
|
||||
|
||||
Observations:
|
||||
|
||||
- **A common beta is selected by both datasets at alpha=0.25 (`0.00`) and alpha=0.75 (`0.50`)**;
|
||||
they differ at alpha=0.50 (5G-NIDD `1.00` vs X-IIoTID `0.75`).
|
||||
- The selected beta is **not a function of alpha alone** and is **not universal**: across the six
|
||||
conditions it takes the values `{0.00, 0.50, 1.00}`.
|
||||
- The alpha=0.25 result (`beta=0`, i.e. the equal-`S_j` mean) is the same descriptive outcome on
|
||||
both datasets, but at the other two alphas the selections diverge, and no tie-break was ever
|
||||
exercised.
|
||||
- All six pooled curves are non-monotone except X-IIoTID alpha=0.25 (monotone decreasing); the
|
||||
between-beta spread is generally comparable to or smaller than the across-seed std (see the
|
||||
per-dataset cross-alpha documents).
|
||||
|
||||
Full pooled curves (validation per-client-mean rare-class macro-F1):
|
||||
|
||||
| condition | beta 0.00 | 0.25 | 0.50 | 0.75 | 1.00 |
|
||||
|---|---|---|---|---|---|
|
||||
| 5G-NIDD a0.25 | **0.7615** | 0.7493 | 0.7257 | 0.7222 | 0.7376 |
|
||||
| 5G-NIDD a0.50 | 0.6872 | 0.6451 | 0.6747 | 0.6884 | **0.6924** |
|
||||
| 5G-NIDD a0.75 | 0.7781 | 0.7905 | **0.8091** | 0.7915 | 0.8043 |
|
||||
| X-IIoTID a0.25 | **0.5917** | 0.5887 | 0.5859 | 0.5795 | 0.5321 |
|
||||
| X-IIoTID a0.50 | 0.5444 | 0.5554 | 0.5673 | **0.5794** | 0.5735 |
|
||||
| X-IIoTID a0.75 | 0.4717 | 0.4744 | **0.4767** | 0.4725 | 0.4686 |
|
||||
|
||||
---
|
||||
|
||||
## E. Prototype displacement
|
||||
|
||||
Cross-seed mean `||Cbar_j(1) − Cbar_j(0)||` (fixed round-0 snapshots, no retraining):
|
||||
|
||||
| dataset | alpha=0.25 | alpha=0.50 | alpha=0.75 |
|
||||
|---|---|---|---|
|
||||
| 5G-NIDD | 2.919 | 2.340 | 1.385 |
|
||||
| X-IIoTID | 3.422 | 2.099 | 1.305 |
|
||||
|
||||
Within each dataset, mean displacement decreases monotonically as alpha rises, tracking realized
|
||||
support imbalance. At matched alpha, 5G-NIDD has *smaller* mean displacement at alpha=0.25
|
||||
(2.919 vs 3.422) while X-IIoTID has slightly smaller displacement at alpha=0.50/0.75 (2.099 vs
|
||||
2.340; 1.305 vs 1.385). The per-class rank order is seed-sensitive and dataset-specific, and the
|
||||
**rare-class identities differ**, so these are not a like-for-like class comparison. Displacement
|
||||
is a direct-aggregation measure only; it is not interpreted as detection quality.
|
||||
|
||||
---
|
||||
|
||||
## F. Rare-class performance patterns
|
||||
|
||||
Selected-beta test rare-class macro-F1 (mean ± sample std) vs `original_protean`:
|
||||
|
||||
| condition | selected beta | test rare macro-F1 (selected) | test rare macro-F1 (original) | within-condition difference |
|
||||
|---|---|---|---|---|
|
||||
| 5G-NIDD a0.25 | 0.00 | 0.841236 ± 0.024202 | 0.842274 ± 0.025370 | −0.001039 |
|
||||
| 5G-NIDD a0.50 | 1.00 | 0.871984 ± 0.028501 | 0.880542 ± 0.037058 | −0.008558 |
|
||||
| 5G-NIDD a0.75 | 0.50 | 0.896738 ± 0.014470 | 0.879901 ± 0.023121 | +0.016837 |
|
||||
| X-IIoTID a0.25 | 0.00 | 0.632136 ± 0.047642 | 0.632790 ± 0.047152 | −0.000654 |
|
||||
| X-IIoTID a0.50 | 0.75 | 0.665503 ± 0.026685 | 0.642520 ± 0.028683 | +0.022983 |
|
||||
| X-IIoTID a0.75 | 0.50 | 0.605246 ± 0.085489 | 0.603890 ± 0.077764 | +0.001356 |
|
||||
|
||||
**Do not compare absolute test levels across datasets or conditions.** The training-derived rare
|
||||
sets, global rare support and per-client eval partitions differ; 5G-NIDD rare macro-F1 levels
|
||||
(≈0.84–0.90) exceed X-IIoTID levels (≈0.60–0.67) largely because the rare classes are different
|
||||
and 5G-NIDD's are easier for nearest-prototype inference. Only the within-condition selected-vs-
|
||||
original difference is meaningful, and it is dominated by between-seed variance; the largest
|
||||
positive difference (+0.023 for X-IIoTID alpha=0.50) is still smaller than its between-seed std.
|
||||
No cross-dataset superiority is claimed.
|
||||
|
||||
A consistent dataset-level observation: on 5G-NIDD test weighted-F1 is **below** accuracy (hard
|
||||
high-support classes), whereas on X-IIoTID weighted-F1 is **above** accuracy. This is a dataset
|
||||
characteristic, not a beta effect.
|
||||
|
||||
At alpha=0.25 both datasets select `beta=0`, so at that condition the selected mode is
|
||||
arithmetically the equal-`S_j` mean and cannot demonstrate a benefit of support weighting.
|
||||
|
||||
---
|
||||
|
||||
## G. Client-level effects
|
||||
|
||||
Improve/degrade/unchanged (validation, vs `beta=0`) at each condition's selected beta:
|
||||
|
||||
| condition | selected beta | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|---|
|
||||
| 5G-NIDD a0.25 | 0.00 | 0/0/10 | 0/0/10 | 0/0/10 |
|
||||
| 5G-NIDD a0.50 | 1.00 | 6/3/1 | 3/5/2 | 7/2/1 |
|
||||
| 5G-NIDD a0.75 | 0.50 | 4/6/0 | 6/3/1 | 4/2/4 |
|
||||
| X-IIoTID a0.25 | 0.00 | 0/0/10 | 0/0/10 | 0/0/10 |
|
||||
| X-IIoTID a0.50 | 0.75 | 7/3/0 | 8/1/1 | 8/1/1 |
|
||||
| X-IIoTID a0.75 | 0.50 | 6/4/0 | 9/0/1 | 6/4/0 |
|
||||
|
||||
At the common `beta=0.75` (non-trivial comparison), the number of clients that improve (out of 10)
|
||||
is: 5G-NIDD **3 / 2 / 1** (a0.25), **5 / 3 / 7** (a0.50), **4 / 6 / 1** (a0.75); X-IIoTID
|
||||
**5 / 5 / 3** (a0.25), **7 / 8 / 8** (a0.50), **3 / 9 / 5** (a0.75). The broad shape is similar
|
||||
(mixed and seed-dependent), but the specific counts and mean deltas differ; at alpha=0.25, in
|
||||
particular, support weighting at `beta=0.75` helps more X-IIoTID clients than 5G-NIDD clients.
|
||||
`N_ij` vs ΔF1 is non-monotonic on both datasets. Client ids are per-seed realisations and are never
|
||||
pooled, so no client-level cross-dataset pairing is performed.
|
||||
|
||||
---
|
||||
|
||||
## H. RQ1 synthesis (support imbalance ↔ displacement ↔ rare-class performance)
|
||||
|
||||
- On both datasets, realized support imbalance and mean prototype displacement move together with
|
||||
alpha: higher alpha → lower CV / fewer missing cells / smaller displacement. Directionally
|
||||
consistent across all six conditions.
|
||||
- Rare-class validation/test **levels** do **not** track the same ordering, on either dataset; they
|
||||
are confounded by which classes are rare and their support in each partition.
|
||||
- The link between support imbalance and displacement is mechanical under the beta formula; the
|
||||
link to rare-class performance is not established causally. **No causal conclusion.**
|
||||
|
||||
## I. RQ2 synthesis (how beta changes prototypes and rare-class performance)
|
||||
|
||||
- Increasing beta moves class prototypes monotonically away from the beta=0 equal mean, by an
|
||||
amount that grows with realized support imbalance; this holds on both datasets.
|
||||
- Rare-class validation response is non-monotone in beta and condition-dependent on both datasets.
|
||||
The effect is small relative to between-seed variance at most conditions.
|
||||
- At alpha=0.25 on **both** datasets the pooled validation optimum sits at `beta=0`, i.e. support
|
||||
weighting gives no rare-class validation benefit under the strongest non-IID setting tested.
|
||||
|
||||
## J. RQ3 synthesis (does selected beta differ across conditions/datasets?)
|
||||
|
||||
- Across the six conditions the pooled-selected beta is `{0.00, 1.00, 0.50}` (5G-NIDD) and
|
||||
`{0.00, 0.75, 0.50}` (X-IIoTID): it differs by condition and by dataset.
|
||||
- It is **not** a function of alpha alone (both datasets give identical selections at alpha=0.25 and
|
||||
0.75 but differ at alpha=0.50), and it is **not** universal.
|
||||
- Caveats forbidding a trend/universality claim: two datasets, three alpha levels, three
|
||||
project-defined seeds, six weak selections, no tie-break exercised, different rare-class
|
||||
compositions, no inferential trend test.
|
||||
|
||||
## K. Limitations
|
||||
|
||||
- **Six conditions, two datasets, three project-defined seeds.** No inferential test; no global
|
||||
optimum; each selected beta is "best among the tested candidates under pooled validation for that
|
||||
condition".
|
||||
- The `S_j` participant/denominator rule remains a **REPRODUCTION ASSUMPTION**; it is material in
|
||||
16/27 cells at 5G-NIDD alpha=0.25 and 19/27 at X-IIoTID alpha=0.25, and in ≤7/27 at the other
|
||||
conditions. All comparisons are conditional on both modes sharing it.
|
||||
- **Rare-set composition differs by dataset** and is not comparable across datasets; absolute
|
||||
metric levels are not comparable across conditions either.
|
||||
- Train/validation/test adaptation (70/10/20 stratified) differs from the paper's literal 80/20;
|
||||
the validation eval partition is a project decision.
|
||||
- No `equal_available` control exists; `beta=0` is only arithmetically the equal-`S_j` mean.
|
||||
- **No cross-dataset causality or superiority**; **no monotone beta law**; the experiment grid is
|
||||
complete as planned (two datasets × three alphas × three seeds × five betas + baseline).
|
||||
@@ -0,0 +1,284 @@
|
||||
# Session 11 Post-Experiment Audit (5G-NIDD alpha=0.25)
|
||||
|
||||
Status: **PASS.** Independent recomputation from machine-readable artifacts and the raw dataset
|
||||
confirms every Session-11 Stage-B claim for 5G-NIDD `alpha=0.25`, and re-confirms the prior
|
||||
condition labels used in the cross-alpha and cross-dataset reports. No discovered issue changes
|
||||
`beta_star` or the scientific interpretation. **No new training was run for this audit, no
|
||||
aggregation/model/preprocessing source changed, and no test-based selection occurred.**
|
||||
|
||||
Audit performed: 2026-09-30. Audited HEAD: `bbd35085063194db58e4472e70b204d4be4a3607`.
|
||||
Machine-readable audit record:
|
||||
`artifacts/analysis/5gnidd__alpha0.25__multiseed__audit_recomputation.json`.
|
||||
|
||||
Companion documents: `docs/research/5GNIDD_ALPHA025_BETA_VALIDATION.md`,
|
||||
`docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md`, `docs/research/CROSS_DATASET_BETA_ANALYSIS.md`,
|
||||
`docs/research/5GNIDD_ALPHA050_BETA_VALIDATION.md`,
|
||||
`docs/research/5GNIDD_ALPHA075_BETA_VALIDATION.md`, `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md`,
|
||||
`docs/research/EXPERIMENT_PROTOCOL.md`, `docs/handoff/SESSION_HANDOFF.md`.
|
||||
|
||||
---
|
||||
|
||||
## A. Scope
|
||||
|
||||
Independently verify the Session-11 claim for:
|
||||
|
||||
- dataset `5gnidd`, alpha `0.25`, seeds `{0,1,2}`, beta `{0, 0.25, 0.5, 0.75, 1.0}` plus
|
||||
`original_protean` (18 runs);
|
||||
- PRIMARY pooled-validation selection `beta_star = 0.00` using validation per-client rare-class
|
||||
macro-F1 averaged within seed then across seeds;
|
||||
- `beta=0` equivalence class;
|
||||
- the 5G-NIDD three-alpha labels (0.25 = 0.00, 0.50 = 1.00, 0.75 = 0.50) and the six-condition
|
||||
cross-dataset labels.
|
||||
|
||||
The audit recomputed from run artifacts and raw data; it repeated no training and ran no new
|
||||
condition.
|
||||
|
||||
---
|
||||
|
||||
## B. Run inventory
|
||||
|
||||
18 full-data runs enumerated from `results/5gnidd__alpha0.25__*/metadata.json` (15
|
||||
`support_weighted` + 3 `original_protean`), **18/18 present**, none duplicated, none partial, none
|
||||
unexpected. Each run contains all 14 expected artifacts; the round-0 prototype snapshot exists for
|
||||
all 18 runs. Metadata verified per run: dataset=5gnidd, alpha=0.25, seed, mode, clients=10,
|
||||
rounds=10, local_epochs=3, batch=256, lr=1e-3, mu=0.1, lambda=1.0, `manifest_reload_ok=true`,
|
||||
`smoke=false`. All 18 exit status 0. No other 5G-NIDD alpha run was created or modified
|
||||
(alpha=0.50 and alpha=0.75 each still contain exactly 18 runs); `equal_available` is absent from
|
||||
`src/`, `scripts/`, `tests/`.
|
||||
|
||||
---
|
||||
|
||||
## C. Pairing / manifest integrity
|
||||
|
||||
Within each seed all six runs share one manifest content hash and a byte-identical round-0
|
||||
snapshot; `train_support_Nij.csv`, `characterization.csv` and `rare_unseen.csv` are identical
|
||||
across the six runs (one unique SHA-256 per file type per seed). `manifest_reload_ok = true` and
|
||||
`smoke = false` everywhere.
|
||||
|
||||
| seed | manifest content hash | round-0 snapshot SHA-256 | support/char/rare unique counts |
|
||||
|---|---|---|---|
|
||||
| 0 | `ab6d8a8b63d478c4…` | `e2243ac4ed4391ff…` | 1 / 1 / 1 |
|
||||
| 1 | `74df102e4edb4509…` | `26a162991b7cbde9…` | 1 / 1 / 1 |
|
||||
| 2 | `061c094d1913f942…` | `e287bff5ca71f8d2…` | 1 / 1 / 1 |
|
||||
|
||||
Each alpha=0.25 manifest was independently reproduced from the raw dataset
|
||||
(`prepare_dataset` → `reproduce_train_assignments`): exact `N_ij` match, partition-hash match and
|
||||
split-hash match for all three seeds; split sizes 851,123 / 121,589 / 243,177; 10 clients.
|
||||
Pairing is proven by content hashes, not filenames.
|
||||
|
||||
---
|
||||
|
||||
## D. Partition characterization audit
|
||||
|
||||
Independently recomputed from the manifests; matches the report exactly:
|
||||
|
||||
| statistic | seed 0 / 1 / 2 | reality (sum / mean) |
|
||||
|---|---|---|
|
||||
| `|S_j| < M` cells | 6 / 5 / 5 | **16 / 27** |
|
||||
| missing client-class cells | 10 / 10 / 10 | **30** |
|
||||
| client-size ratio | 9.93 / 27.83 / 15.05 | mean **17.60** |
|
||||
| union rare classes | 6 / 6 / 8 | 2 rare per client (all) |
|
||||
| union unseen classes | 6 / 5 / 5 | max 3 on a client |
|
||||
|
||||
Mean class-support CV = **1.302**; maximum per-class `max/min` ratio across seeds = 185,504.
|
||||
|
||||
---
|
||||
|
||||
## E. Label / rare / unseen integrity
|
||||
|
||||
- Cleaned 5G-NIDD label semantics reproduced: 1,215,890 raw rows, 1 duplicate dropped, 0 label
|
||||
rows dropped, 9 classes (`Benign` + 8 attacks), benign index 0, 46 raw feature columns
|
||||
(41 numeric, 5 categorical; 81 one-hot expanded).
|
||||
- Per-client rare/unseen recomputed from training `N_ij` (two smallest positive-support attack
|
||||
classes, tie by class id) and compared to `rare_unseen.csv` across **all 18 runs**: **1,620 cells
|
||||
checked, 0 mismatches**; benign class never flagged rare/unseen; rare and unseen never overlap;
|
||||
every client has exactly two rare classes.
|
||||
- `evaluated_rare_classes` equals the training-derived union rare set (per seed); validation/test
|
||||
labels did not influence rare identity.
|
||||
|
||||
---
|
||||
|
||||
## F. Validation metric recomputation
|
||||
|
||||
Recomputed from raw artifacts (no report numbers copied):
|
||||
|
||||
- seed-level primary score = mean over eligible clients of `val_per_client.csv`
|
||||
`rare_class_macro_f1`; equals `val_selection.json` per-client-mean to **max diff 0.0** for all
|
||||
15 runs; all 10 clients eligible in every seed.
|
||||
- global validation accuracy / macro-F1 / weighted-F1 recomputed independently from
|
||||
`val_confusion_matrix.json` with the production formulas: **max diff 0.0** versus
|
||||
`val_global_metrics.json`.
|
||||
|
||||
---
|
||||
|
||||
## G. Pooled-selection recomputation
|
||||
|
||||
Computed from the independently recomputed seed-level scores (not from the report). Pooled = mean
|
||||
of the three seed means; std sample (`ddof=1`).
|
||||
|
||||
| beta | pooled mean | sample std | recomputed selected |
|
||||
|---|---|---|---|
|
||||
| **0.00** | **0.761453** | 0.021903 | **yes** |
|
||||
| 0.25 | 0.749286 | 0.017107 | |
|
||||
| 0.50 | 0.725715 | 0.011017 | |
|
||||
| 0.75 | 0.722235 | 0.007967 | |
|
||||
| 1.00 | 0.737636 | 0.002756 | |
|
||||
|
||||
- Recompute **matches the report** to formatting precision.
|
||||
- `argmax` = **0.00** (no tie; top-two gap ≈ 1.2e-2). `tie_break_used=false`.
|
||||
- Per-seed diagnostics reproduce **0.25 / 0.00 / 0.00**.
|
||||
|
||||
---
|
||||
|
||||
## H. Validation / test leakage audit
|
||||
|
||||
- `run_baseline.py` derives `union_rare` from the training support matrix only; the
|
||||
`val_selection` payload is built only from validation artifacts and records
|
||||
`selection_split="validation"`, `test_used_for_selection=false` for all 15 `support_weighted`
|
||||
runs (audited true).
|
||||
- `analyze_multiseed_beta.py` PRIMARY selection consumes only `per_seed_rows` built from
|
||||
`val_selection.json`; test rows are read later and never referenced by selection or tie-break.
|
||||
No code path sorts/filters candidates by a test field.
|
||||
- The audit's own `beta_star = 0.00` was reproduced from validation artifacts alone.
|
||||
|
||||
Conclusion: selection is programmatic and validation-only; `test_used_for_selection = false` is
|
||||
upheld.
|
||||
|
||||
---
|
||||
|
||||
## I. Final test recomputation
|
||||
|
||||
`beta_star = 0.00` fixed from pooled validation; only then test metrics read. Recomputed from
|
||||
`test_global_metrics.json` per seed (and cross-checked against `test_descriptive.csv`, max diff
|
||||
0.0), averaged across seeds with sample std:
|
||||
|
||||
| metric | selected beta 0.00 (mean ± std) | original_protean (mean ± std) |
|
||||
|---|---|---|
|
||||
| rare-class macro-F1 | 0.841236 ± 0.024202 | 0.842274 ± 0.025370 |
|
||||
| rare-class recall | 0.961216 ± 0.031347 | 0.960556 ± 0.032423 |
|
||||
| macro-F1 | 0.816640 ± 0.013357 | 0.819405 ± 0.013919 |
|
||||
| accuracy | 0.730225 ± 0.019525 | 0.730968 ± 0.021431 |
|
||||
| weighted-F1 | 0.704463 ± 0.018175 | 0.705435 ± 0.020367 |
|
||||
|
||||
Matches the report exactly. No superiority claim is made. The selected-beta and
|
||||
`original_protean` test means differ only by floating-point divergence (see §J), as expected when
|
||||
the selected beta is 0.
|
||||
|
||||
---
|
||||
|
||||
## J. `beta = 0` equivalence audit
|
||||
|
||||
For seeds 0, 1, 2, `original_protean` vs `support_weighted beta=0`:
|
||||
|
||||
| property | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| same manifest / split / hashes | yes | yes | yes |
|
||||
| round-0 snapshot byte-identical | yes | yes | yes |
|
||||
| support/characterization/rare_unseen identical | yes | yes | yes |
|
||||
| independent aggregation diff | 4.4e-16 | 4.4e-16 | 6.7e-16 |
|
||||
| history rare-macro-F1 round-0 diff | 0.0 | 0.0 | 0.0 |
|
||||
| history rare-macro-F1 max diff (rounds ≥ 1) | 2.17e-2 | 6.85e-2 | 2.74e-2 |
|
||||
| test metric max diff | 5.36e-3 | 5.39e-3 | 2.85e-3 |
|
||||
| classification | **B** | **B** | **B** |
|
||||
|
||||
Class **B** (semantically equivalent + floating-point divergence); **no semantic discrepancy**.
|
||||
Equivalence remains conditional on the shared `S_j` reproduction assumption. Because the PRIMARY
|
||||
selection at this condition chose `beta=0`, the selected run *is* the `beta=0` run, so its test
|
||||
metrics differ from `original_protean` only by this divergence.
|
||||
|
||||
---
|
||||
|
||||
## K. Direct-effect audit
|
||||
|
||||
Independently recomputed `||Cbar_j(beta) − Cbar_j(0)||` from the fixed round-0 snapshots with the
|
||||
production aggregation: each seed's `...__prototype_effect.csv` is produced by
|
||||
`analyze_prototype_effect.py` from the snapshot and the cross-seed pooled file aggregates it. Mean
|
||||
displacement at `beta=1.0` is **2.919** (monotone: 1.295 / 2.055 / 2.559 / 2.919 for
|
||||
beta 0.25/0.50/0.75/1.00). Displacement is a direct-aggregation measure only; no detection
|
||||
causality is inferred.
|
||||
|
||||
---
|
||||
|
||||
## L. Client-level audit
|
||||
|
||||
At the selected `beta=0.00` the delta vs `beta=0` is trivially 0/0/10 for all seeds. At the common
|
||||
non-trivial `beta=0.75`, deltas recomputed from `val_per_client.csv` vs
|
||||
`client_validation_detail.csv` give improve/degrade/unchanged **3/4/3** (seed 0, mean −0.023),
|
||||
**2/5/3** (seed 1, mean −0.043), **1/4/5** (seed 2, mean −0.052). `N_ij` vs ΔF1 is non-monotonic.
|
||||
Client ids are per-seed realisations, not pooled.
|
||||
|
||||
---
|
||||
|
||||
## M. Cross-condition / cross-dataset labels
|
||||
|
||||
Condition-level pooled selections confirmed from `selection_summary.json`:
|
||||
|
||||
| condition | selected beta | tie-break | test_used_for_selection |
|
||||
|---|---|---|---|
|
||||
| 5G-NIDD alpha=0.25 (this session) | 0.00 | no | false |
|
||||
| 5G-NIDD alpha=0.50 (Session 9) | 1.00 | no | false |
|
||||
| 5G-NIDD alpha=0.75 (Session 10) | 0.50 | no | false |
|
||||
| X-IIoTID alpha=0.25 (Session 7) | 0.00 | no | false |
|
||||
| X-IIoTID alpha=0.50 (Session 6) | 0.75 | no | false |
|
||||
| X-IIoTID alpha=0.75 (Session 8) | 0.50 | no | false |
|
||||
|
||||
The cross-alpha and cross-dataset reports make no causal, monotone, universal-beta or
|
||||
cross-dataset-superiority claim and explicitly state that selections differ descriptively.
|
||||
|
||||
---
|
||||
|
||||
## N. Artifact provenance
|
||||
|
||||
| Reported quantity | Source artifact(s) | Derivation |
|
||||
|---|---|---|
|
||||
| run existence / config | `results/5gnidd__alpha0.25__*/metadata.json` | direct read |
|
||||
| manifest hashes / reproduction | `artifacts/partitions/5gnidd__alpha0.25__seed{0,1,2}.json` | module + `reproduce_train_assignments` |
|
||||
| pairing proof | `artifacts/analysis/..._prototype_snapshot.json` | sha256 across 6 runs/seed |
|
||||
| seed-level validation score | `results/.../val_per_client.csv` + `val_selection.json` | mean of per-client rare macro-F1 |
|
||||
| validation global metrics | `results/.../val_confusion_matrix.json` | independent numpy recompute |
|
||||
| pooled mean/std, `beta_star` | recomputed seed scores | mean/`stdev`, argmax |
|
||||
| test metrics | `results/.../test_global_metrics.json` + `..._test_descriptive.csv` | independent recompute + mean/std |
|
||||
| partition characterization | manifests + `..._partition_heterogeneity_*.csv` + `..._partition_characterization.csv` | independent numpy recompute |
|
||||
| beta=0 equivalence | original vs beta0 artifacts/snapshots | sha256 + independent aggregation + history diff |
|
||||
| direct-effect displacement | snapshots + `..._prototype_effect_pooled.csv` | production aggregation |
|
||||
| client improve/degrade | `results/.../val_per_client.csv` + `..._client_validation_detail.csv` | deltas vs beta=0 |
|
||||
| rare-class identity | `results/.../rare_unseen.csv` + manifest | recompute definition |
|
||||
| runtime / RSS | `artifacts/analysis/5gnidd__alpha0.25__multiseed__runtime.csv` (source `/tmp/opencode/5gnidd_alpha025_runtime.csv`) | parse `/usr/bin/time -v`; sum 68,798 s = 19.11 h |
|
||||
| cross-alpha table | `5gnidd__cross_alpha__*` (built from alpha0.25 + alpha0.5 + alpha0.75 multiseed artifacts) | read per-condition artifacts |
|
||||
| cross-dataset table | `cross_dataset__matched_alpha_summary.*` (built from both datasets' cross-alpha artifacts) | read per-dataset artifacts |
|
||||
|
||||
---
|
||||
|
||||
## O. Deviations / corrections found
|
||||
|
||||
**None** affecting science. All Session-11 checks PASS. The only corrections made were to draft
|
||||
narrative text in the cross-dataset report (a client-count sentence was recomputed from the
|
||||
`cross_alpha_client_summary.csv` before the report was finalised); no number, artifact, or
|
||||
selection was affected. No training/aggregation/preprocessing source file was modified.
|
||||
|
||||
---
|
||||
|
||||
## P. Final audit verdict
|
||||
|
||||
**PASS.**
|
||||
|
||||
- Run inventory: **PASS** (18/18; no duplicates/partials; no legacy or extra 5G-NIDD run).
|
||||
- Pairing / manifest reproduction: **PASS** (hash-identical; raw-data reproduction exact incl.
|
||||
partition hash).
|
||||
- Partition characterization: **PASS** (16/27, 30 missing, ratios 9.93/27.83/15.05, unseen 6/5/5).
|
||||
- Rare/unseen integrity: **PASS** (0 mismatches over 1,620 cells).
|
||||
- Validation recomputation (seed-level and global): **PASS** (exact).
|
||||
- Pooled selection: **PASS**; independent `beta_star = 0.00` using validation only.
|
||||
- Validation/test leakage: **PASS**.
|
||||
- Test summaries: **PASS** (exact).
|
||||
- beta=0 equivalence: **PASS**, class **B** for all seeds.
|
||||
- Direct-effect and client-level analyses: **PASS** (consistent with snapshots/artifacts).
|
||||
- Cross-condition labels: **PASS** (5G-NIDD 0.25/0.50/0.75 = 0.00/1.00/0.50; X-IIoTID
|
||||
0.25/0.50/0.75 = 0.00/0.75/0.50; no causal or universal-beta claim).
|
||||
- Source/data integrity: **PASS**; `planningawal/` unchanged; both raw dataset SHA-256 unchanged;
|
||||
`equal_available` absent; pytest **75 passed, 0 failed, 0 skipped**.
|
||||
|
||||
The planned two-dataset × three-alpha experiment grid is **complete**. Session 11 is **safe to
|
||||
checkpoint** with the documented limitations (three project-defined seeds per condition, weak
|
||||
effects relative to seed variance, `S_j` reproduction assumption, `beta=0` selection at both
|
||||
alpha=0.25 conditions, no global-optimum claim).
|
||||
Reference in new issue
Block a user