experiment: validate beta selection across three seeds
This commit is contained in:
willtanoe committed
1 parent
50f2da7216
commit
6b97a0d38b
14 files changed
+1963
-150
No files matched your search
@@ -121,19 +121,40 @@ Persisted metadata (minimum one row per `(dataset, alpha, seed, client, class)`)
|
||||
|
||||
## 8. Validation-based beta selection
|
||||
|
||||
Selection hierarchy (strictly ordered). Test data is never used in PRIMARY or
|
||||
SECONDARY.
|
||||
|
||||
**PRIMARY — pooled validation across clients and random seeds for one non-IID
|
||||
condition.** For each beta, average the per-client rare-class macro-F1 over eligible
|
||||
clients within each seed, then average those seed-level scores over the condition's
|
||||
seeds. Select `beta_star = argmax` of this pooled score using validation only. This
|
||||
is the beta reported for the condition. Tie-break: higher pooled per-client-mean
|
||||
rare-class macro-F1 -> higher pooled validation rare-class recall -> lower
|
||||
across-seed std -> smaller beta.
|
||||
|
||||
**SECONDARY — per-seed beta selection, for stability analysis only.** For each seed
|
||||
independently, choose the validation-best beta with the same metric. Per-seed choices
|
||||
are diagnostics of seed sensitivity and are **not** the condition's selection rule.
|
||||
|
||||
**TEST — used only after `beta_star` is fixed from pooled validation.** Evaluate the
|
||||
fixed `beta_star` on test, and report all candidates' test metrics for transparency.
|
||||
No test metric may influence `beta_star`.
|
||||
|
||||
Procedure, strictly ordered:
|
||||
|
||||
1. **Candidate evaluation (validation only).** For each beta candidate, train with the paired
|
||||
manifest and compute validation metrics. Primary criterion: rare-class macro-F1, computed
|
||||
per client that has rare classes, then averaged over those clients.
|
||||
2. **Beta selection.** Choose the beta with the best mean validation rare-class macro-F1 (ties
|
||||
broken by smaller beta). The result is "best among tested candidates under this
|
||||
procedure", never a global optimum.
|
||||
manifest and compute validation metrics. Primary metric: per-client rare-class macro-F1,
|
||||
computed per client that has rare classes, then averaged over those clients.
|
||||
2. **PRIMARY beta selection.** Choose the beta with the best pooled-across-seed mean
|
||||
validation per-client rare-class macro-F1 (tie-break per the hierarchy above). The result
|
||||
is "best among tested candidates under this procedure", never a global optimum.
|
||||
3. **Final test evaluation.** Evaluate the selected beta on the test split. Report overall
|
||||
macro-F1, rare-class recall, rare-class macro-F1, and per-client results, plus the test
|
||||
results of all candidates for transparency.
|
||||
|
||||
Test data is never used in steps 1–2.
|
||||
Session-5 (single-seed) used the one-seed special case of step 2. Session 6 applies the
|
||||
full pooled rule across seeds `{0,1,2}` for X-IIoTID alpha=0.50; see
|
||||
`docs/research/MULTISEED_BETA_VALIDATION.md`.
|
||||
|
||||
**Session-5 clarification (implementation detail, not a design change).** Running
|
||||
the protocol surfaced two concrete points that the original text left implicit.
|
||||
@@ -151,7 +172,9 @@ They do not alter the frozen design; they pin how step 1 is computed.
|
||||
evaluation, applied to the validation split (`split_name="validation"`); this
|
||||
is a PROJECT DECISION, paired across all beta candidates and seeds.
|
||||
2. **Tie-break order.** Higher validation rare-class macro-F1, then higher
|
||||
validation rare-class recall, then smaller beta.
|
||||
validation rare-class recall, then lower across-seed std (pooled PRIMARY only),
|
||||
then smaller beta. The pooled PRIMARY tie-break is authoritative; the per-seed
|
||||
SECONDARY uses the same order without the std step.
|
||||
|
||||
`scripts/run_baseline.py` writes `val_global_metrics.json`, `val_per_class.csv`,
|
||||
`val_per_client.csv` and `val_selection.json` to make this explicit; the test
|
||||
|
||||
@@ -0,0 +1,399 @@
|
||||
# Multi-Seed Beta Validation (Session 6)
|
||||
|
||||
Status: **three-seed paired protocol completed for one non-IID condition** with
|
||||
pooled-across-seed validation-only beta selection, followed by a single final test
|
||||
evaluation of the pooled-selected beta. This is a **stability / protocol-scaling
|
||||
experiment**, not a sweep and not evidence of a global optimum.
|
||||
|
||||
Companion documents:
|
||||
|
||||
- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen protocol and selection hierarchy.
|
||||
- `docs/research/PAIRED_BETA_VALIDATION.md` — Session-5 single-seed condition.
|
||||
- `docs/research/BETA_AGGREGATION_IMPLEMENTATION.md` — beta mode implementation.
|
||||
- `docs/baseline/BASELINE_VALIDATION.md` — frozen baseline evidence.
|
||||
- `docs/handoff/SESSION_HANDOFF.md` — continuation point.
|
||||
|
||||
---
|
||||
|
||||
## A. Scope
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| dataset | `xiiotid` (X-IIoTID) |
|
||||
| alpha | 0.50 |
|
||||
| seeds | 0, 1, 2 |
|
||||
| aggregation | `original_protean` (per seed) + `support_weighted` for `beta ∈ {0, 0.25, 0.5, 0.75, 1}` |
|
||||
| runs | 15 `support_weighted` + 3 `original_protean` |
|
||||
| reused | seed 0 (all five betas + `original_protean`) from Session 5, not re-run |
|
||||
| new this session | seeds 1 and 2 (five betas each + `original_protean` each) |
|
||||
|
||||
No other alpha, dataset, seed, mode or beta was run. `equal_available` was not added.
|
||||
|
||||
---
|
||||
|
||||
## B. Frozen protocol
|
||||
|
||||
Training config is identical to Session 5 and to the frozen protocol
|
||||
(`EXPERIMENT_PROTOCOL.md` §14): 10 clients, 10 rounds, 3 local epochs, batch 256,
|
||||
Adam `lr=1e-3`, `mu=0.1`, `lambda=1.0`; model architecture, preprocessing,
|
||||
train/validation/test split recipe, partition recipe, loss terms and inference
|
||||
unchanged. No training/aggregation source was changed this session; the only code
|
||||
changes are the read-only analysis script `scripts/analyze_multiseed_beta.py` (and
|
||||
the Session-5 `scripts/run_baseline.py` validation-artifact emission already at HEAD).
|
||||
|
||||
### B.1 Manifest identity (reused, never regenerated)
|
||||
|
||||
| seed | manifest path | content hash | partition hash | split hash | rng seed | train/val/test sizes |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0 | `artifacts/partitions/xiiotid__alpha0.5__seed0.json` | `e81a6538…` | `2465485539…` | `d0acc0c1…` | 2009062324 | 569,909 / 81,414 / 162,831 |
|
||||
| 1 | `artifacts/partitions/xiiotid__alpha0.5__seed1.json` | `54ea1df1…` | `4061c5a1…` | `880a9f82…` | 621679668 | 569,909 / 81,414 / 162,831 |
|
||||
| 2 | `artifacts/partitions/xiiotid__alpha0.5__seed2.json` | `be881ac6…` | `1118e7fa…` | `8e0db0be…` | 1438483982 | 569,909 / 81,414 / 162,831 |
|
||||
|
||||
`manifest_reload_ok = true` and the per-seed manifest content hash is identical
|
||||
across all six runs (five betas + `original_protean`) of that seed, so pairing holds.
|
||||
|
||||
### B.2 Build-level pairing proof (round-0 snapshot byte-identity)
|
||||
|
||||
The round-0 prototype snapshot is byte-identical across the five beta runs **and**
|
||||
`original_protean` within each seed:
|
||||
|
||||
| seed | SHA-256 of round-0 snapshot (all six runs) |
|
||||
|---|---|
|
||||
| 0 | `e27f41b3a62eb83c…` |
|
||||
| 1 | `b7f61c995e97474c…` |
|
||||
| 2 | `5b749f4185e11fc7…` |
|
||||
|
||||
`train_support_Nij.csv`, `characterization.csv` and `rare_unseen.csv` are also
|
||||
byte-identical across the runs of each seed. Beta only changes server aggregation.
|
||||
|
||||
### B.3 Rare/unseen derivation is per-seed
|
||||
|
||||
Rare classes are derived from the manifest training split (`EXPERIMENT_PROTOCOL.md`
|
||||
§7). The Dirichlet partitions differ by seed, so the union of rare classes differs:
|
||||
|
||||
| seed | per-client rare count | union of rare classes |
|
||||
|---|---|---|
|
||||
| 0 | 2 per client (10/10 clients) | C&C, Exfiltration, Exploitation, Lateral_movement, Tampering, Weaponization |
|
||||
| 1 | 2 per client (10/10 clients) | + RDOS |
|
||||
| 2 | 2 per client (10/10 clients) | + RDOS |
|
||||
|
||||
Integrity checks (all pass): rare classes always have `N_ij > 0`; unseen attack
|
||||
classes always have `N_ij = 0`; the benign class (`Normal`) is never rare or unseen;
|
||||
rare and unseen are never merged; every client has exactly two rare classes. The
|
||||
different union rare set is why global test rare support is 25,454 (seed 0) versus
|
||||
53,706 (seeds 1, 2); the primary per-client-mean metric uses each client's own rare
|
||||
classes and so is comparable across seeds.
|
||||
|
||||
### B.4 Selection rule
|
||||
|
||||
- **PRIMARY:** pooled-across-seed validation per-client-mean rare-class macro-F1
|
||||
(mean of each seed's per-client-mean score over seeds `{0,1,2}`), selected with
|
||||
validation only.
|
||||
- **SECONDARY (diagnostic):** per-seed validation selection, for stability analysis.
|
||||
- **TEST:** computed for all candidates by the frozen pipeline but used only after
|
||||
the pooled beta is fixed. No test metric influenced selection.
|
||||
|
||||
---
|
||||
|
||||
## C. Per-seed candidate validation results
|
||||
|
||||
`val rareMacroF1 (pc mean)` is the primary metric: each client scored on its
|
||||
deterministic Dirichlet **validation** eval partition with that client's rare class
|
||||
ids, then averaged over the 10 eligible clients. All numbers are validation only.
|
||||
|
||||
### Seed 0 (reused from Session 5)
|
||||
|
||||
| beta | eligible | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | 10 | 0.557340 | 0.611641 | 0.884184 | 0.708929 | 0.891038 |
|
||||
| 0.25 | 10 | 0.571787 | 0.629475 | 0.888488 | 0.723431 | 0.902953 |
|
||||
| **0.50** | 10 | **0.585762** | **0.641709** | 0.887173 | **0.731166** | 0.897831 |
|
||||
| 0.75 | 10 | 0.578003 | 0.637107 | 0.873910 | 0.726603 | 0.887894 |
|
||||
| 1.00 | 10 | 0.551443 | 0.615307 | 0.847043 | 0.715951 | 0.883263 |
|
||||
|
||||
### Seed 1 (new)
|
||||
|
||||
| beta | eligible | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | 10 | 0.628624 | 0.656935 | 0.890520 | 0.709324 | 0.899612 |
|
||||
| 0.25 | 10 | 0.636205 | 0.666240 | 0.884109 | 0.722367 | 0.919203 |
|
||||
| 0.50 | 10 | 0.645189 | 0.667383 | 0.863833 | 0.724212 | 0.922593 |
|
||||
| 0.75 | 10 | 0.655160 | 0.670472 | 0.855590 | 0.727134 | 0.923760 |
|
||||
| **1.00** | 10 | **0.683117** | **0.685707** | 0.856876 | **0.739006** | **0.924497** |
|
||||
|
||||
### Seed 2 (new)
|
||||
|
||||
| beta | eligible | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 | val acc |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | 10 | 0.447118 | 0.658720 | 0.880662 | 0.708464 | 0.907891 |
|
||||
| 0.25 | 10 | 0.458310 | 0.669242 | 0.885401 | 0.716162 | 0.907018 |
|
||||
| 0.50 | 10 | 0.471015 | 0.677892 | 0.887958 | 0.730956 | 0.920259 |
|
||||
| **0.75** | 10 | **0.504937** | **0.686124** | 0.887725 | **0.735912** | 0.902351 |
|
||||
| 1.00 | 10 | 0.485877 | 0.676314 | 0.877857 | 0.726922 | 0.883534 |
|
||||
|
||||
Machine-readable: `artifacts/analysis/xiiotid__alpha0.5__multiseed__validation_per_seed.csv`.
|
||||
|
||||
---
|
||||
|
||||
## D. Per-seed diagnostic beta selections (SECONDARY, stability only)
|
||||
|
||||
| seed | selected beta | val rareMacroF1 (pc mean) | tie-break used | test_used_for_selection |
|
||||
|---|---|---|---|---|
|
||||
| 0 | 0.50 | 0.585762 | no | false |
|
||||
| 1 | 1.00 | 0.683117 | no | false |
|
||||
| 2 | 0.75 | 0.504937 | no | false |
|
||||
|
||||
**These per-seed choices are diagnostic only.** They are not the primary selection
|
||||
rule for this condition; they measure seed sensitivity. The preferred beta is
|
||||
seed-dependent: 0.50, 1.00, 0.75. `val_selection.json` records
|
||||
`"test_used_for_selection": false` and `"selection_split": "validation"` for all 15
|
||||
`support_weighted` runs.
|
||||
|
||||
---
|
||||
|
||||
## E. PRIMARY pooled-across-seed validation selection
|
||||
|
||||
For each beta, `score_seed(beta)` is the seed-level validation per-client-mean
|
||||
rare-class macro-F1; `score_pooled(beta)` is its mean over seeds `{0,1,2}`.
|
||||
|
||||
| beta | mean (pooled) | std (sample, ddof=1) | std (pop) | min | max | pooled val rareRecall (pc mean) | selected |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 0.00 | 0.544361 | 0.091447 | 0.074666 | 0.447118 | 0.628624 | 0.839088 | |
|
||||
| 0.25 | 0.555434 | 0.090068 | 0.073540 | 0.458310 | 0.636205 | **0.841278** | |
|
||||
| 0.50 | 0.567322 | 0.088539 | 0.072292 | 0.471015 | 0.645189 | 0.838872 | |
|
||||
| **0.75** | **0.579367** | **0.075121** | 0.061336 | 0.504937 | 0.655160 | 0.829088 | **yes** |
|
||||
| 1.00 | 0.573479 | 0.100450 | 0.082017 | 0.485877 | 0.683117 | 0.820763 | |
|
||||
|
||||
- **PRIMARY selected beta (`beta_star`) = 0.75.**
|
||||
- **Criterion:** maximum pooled validation per-client-mean rare-class macro-F1.
|
||||
- **Tie-break:** not used (no exact tie). The secondary pooled recall is *highest* at
|
||||
`beta = 0.25`, but recall is only a tie-break and the primary metric governs.
|
||||
- **`test_used_for_selection = false`.** Test metrics were not inspected until
|
||||
`beta = 0.75` was fixed.
|
||||
- **Interpretation:** `beta = 0.75` is the *best among the tested candidates under
|
||||
pooled validation across the three project-defined seeds for X-IIoTID alpha=0.50.*
|
||||
It is **not** a global optimum.
|
||||
|
||||
Machine-readable: `artifacts/analysis/xiiotid__alpha0.5__multiseed__validation_pooled.csv`
|
||||
and `...__selection_summary.json`.
|
||||
|
||||
---
|
||||
|
||||
## F. Final test evaluation after the pooled selection
|
||||
|
||||
`beta_star = 0.75` was fixed from pooled validation; only then were test metrics
|
||||
read. Per-seed and mean ± std (sample) across the three seeds:
|
||||
|
||||
| metric | seed 0 | seed 1 | seed 2 | mean ± std |
|
||||
|---|---|---|---|---|
|
||||
| test rare-class macro-F1 | 0.636189 | 0.671937 | 0.688381 | 0.665503 ± 0.026685 |
|
||||
| test rare-class recall | 0.857543 | 0.851883 | 0.878321 | 0.862583 ± 0.013921 |
|
||||
| test macro-F1 | 0.726298 | 0.729015 | 0.738194 | 0.731169 ± 0.006234 |
|
||||
| test accuracy | 0.888670 | 0.925493 | 0.905012 | 0.906392 ± 0.018450 |
|
||||
| test weighted-F1 | 0.922278 | 0.944734 | 0.933884 | 0.933632 ± 0.011230 |
|
||||
|
||||
`original_protean` reference for the same three seeds and condition:
|
||||
|
||||
| metric | seed 0 | seed 1 | seed 2 | mean ± std |
|
||||
|---|---|---|---|---|
|
||||
| test rare-class macro-F1 | 0.609620 | 0.655665 | 0.662275 | 0.642520 ± 0.028683 |
|
||||
| test rare-class recall | 0.871487 | 0.889365 | 0.870018 | 0.876957 ± 0.010771 |
|
||||
| test macro-F1 | 0.707701 | 0.709176 | 0.710017 | 0.708965 ± 0.001172 |
|
||||
| test accuracy | 0.890432 | 0.900885 | 0.907045 | 0.899454 ± 0.008398 |
|
||||
| test weighted-F1 | 0.917585 | 0.929132 | 0.927481 | 0.924733 ± 0.006245 |
|
||||
|
||||
Descriptive difference (`beta_star` − `original_protean`) for test rare-class
|
||||
macro-F1 is +0.026569 (seed 0), +0.016272 (seed 1), +0.026106 (seed 2); mean
|
||||
+0.022982. **No superiority or generalisation is claimed:** this is a single
|
||||
dataset, a single alpha, three project-defined seeds, and the between-seed spread
|
||||
is comparable to the between-method difference.
|
||||
|
||||
Test columns are descriptive only. They are also recorded in
|
||||
`artifacts/analysis/xiiotid__alpha0.5__multiseed__test_descriptive.csv`, clearly
|
||||
separated from the validation tables.
|
||||
|
||||
---
|
||||
|
||||
## G. Direct prototype-effect analysis (fixed round-0 snapshots, no retraining)
|
||||
|
||||
Computed with the production aggregation from each seed's fixed `{C_ij, N_ij}`
|
||||
round-0 snapshot. `||Cbar_j(beta) - Cbar_j(0)||` at `beta = 1.0`:
|
||||
|
||||
| class | mean disp | std across seeds | min | max | mean rel disp | typical `|S_j|` |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Lateral_movement | 2.724 | 1.035 | 2.069 | 3.917 | 0.403 | 10 |
|
||||
| RDOS | 2.631 | 1.165 | 1.286 | 3.322 | 0.352 | 10 |
|
||||
| Weaponization | 2.561 | 1.228 | 1.189 | 3.559 | 0.433 | 10 |
|
||||
| Tampering | 2.354 | 1.165 | 1.627 | 3.698 | 0.483 | 9–10 |
|
||||
| Exploitation | 2.344 | 0.412 | 1.934 | 2.759 | 0.423 | 8–10 |
|
||||
| Exfiltration | 2.020 | 0.502 | 1.547 | 2.547 | 0.297 | 9–10 |
|
||||
| C&C | 1.694 | 0.505 | 1.155 | 2.156 | 0.319 | 8–10 |
|
||||
| Normal | 1.557 | 0.548 | 0.990 | 2.084 | 0.292 | 10 |
|
||||
| Reconnaissance | 1.002 | 0.261 | 0.703 | 1.184 | 0.169 | 10 |
|
||||
|
||||
Findings (descriptive):
|
||||
|
||||
- **Every class moves** for `beta > 0` in every seed, because every class has an
|
||||
unequal support distribution across contributors (no class has equal `N_ij`).
|
||||
- The **rank order of displacement is seed-sensitive**: seed 0's largest is
|
||||
`Weaponization`, seed 1's is `RDOS`, seed 2's is `Lateral_movement`. `Exploitation`
|
||||
and `Reconnaissance` are consistently smaller.
|
||||
- **High-support-imbalance classes tend to move more** (e.g. `Weaponization`,
|
||||
`RDOS`, `Tampering`, `Lateral_movement` have large max/min support ratios), but the
|
||||
relationship is not exact and is seed-dependent.
|
||||
- **Displacement is a direct-aggregation measure only** and is not interpreted as
|
||||
improved or degraded detection. The end-to-end effect (§F) is reported separately.
|
||||
|
||||
Outputs: `artifacts/analysis/xiiotid__alpha0.5__seed{0,1,2}__support_weighted__beta0__prototype_snapshot__prototype_effect.csv`
|
||||
and `...__prototype_weights.csv`; cross-seed aggregate
|
||||
`artifacts/analysis/xiiotid__alpha0.5__multiseed__prototype_effect_pooled.csv`.
|
||||
|
||||
---
|
||||
|
||||
## H. Client-level effect (validation, relative to beta = 0)
|
||||
|
||||
Full per-client table: `artifacts/analysis/xiiotid__alpha0.5__multiseed__client_validation_detail.csv`.
|
||||
Summary: `...__client_validation_summary.csv`.
|
||||
|
||||
| seed | beta | improve | degrade | unchanged | mean Δ | min Δ | max Δ |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 0 | 0.25 | 9 | 1 | 0 | +0.0144 | −0.0048 | +0.0593 |
|
||||
| 0 | 0.50 | 8 | 2 | 0 | +0.0284 | −0.0286 | +0.0926 |
|
||||
| 0 | 0.75 | 7 | 3 | 0 | +0.0207 | −0.0358 | +0.0926 |
|
||||
| 0 | 1.00 | 4 | 6 | 0 | −0.0059 | −0.0760 | +0.0414 |
|
||||
| 1 | 0.25 | 6 | 2 | 2 | +0.0076 | −0.0045 | +0.0240 |
|
||||
| 1 | 0.50 | 7 | 2 | 1 | +0.0166 | −0.0054 | +0.0424 |
|
||||
| 1 | 0.75 | 8 | 1 | 1 | +0.0265 | −0.0030 | +0.0872 |
|
||||
| 1 | 1.00 | 9 | 0 | 1 | +0.0545 | 0.0000 | +0.1927 |
|
||||
| 2 | 0.25 | 7 | 1 | 2 | +0.0112 | −0.0007 | +0.0295 |
|
||||
| 2 | 0.50 | 8 | 0 | 2 | +0.0239 | 0.0000 | +0.0697 |
|
||||
| 2 | 0.75 | 8 | 1 | 1 | +0.0578 | −0.0019 | +0.1771 |
|
||||
| 2 | 1.00 | 6 | 3 | 1 | +0.0388 | −0.4028 | +0.2676 |
|
||||
|
||||
Descriptive observations:
|
||||
|
||||
- At `beta = 0.75` the majority of eligible clients improve in every seed
|
||||
(7/10, 8/10, 8/10), and the mean client delta is positive in all three seeds.
|
||||
- The improvement fraction is **seed-dependent**: at `beta = 1.00` seed 0 degrades
|
||||
(4 improve / 6 degrade) while seed 1 improves for 9/10 clients.
|
||||
- **`min positive attack N_ij` vs ΔF1 is non-monotonic.** At `beta = 0.75`, clients
|
||||
with `N_ij = 1` show deltas of −0.021, +0.029, +0.041; clients with `N_ij ≥ 130`
|
||||
show +0.004 to +0.096; the largest gain (+0.177) is a client with `N_ij = 10`. Low
|
||||
support neither systematically benefits nor systematically degrades.
|
||||
- One extreme loss appears at `beta = 1.00`, seed 2 (client with `N_ij = 1`,
|
||||
Δ = −0.403), showing that high beta can be unstable on specific clients.
|
||||
|
||||
**Client ids are different realisations across seeds** and are not treated as the
|
||||
same statistical population. This analysis is descriptive and non-causal.
|
||||
|
||||
---
|
||||
|
||||
## I. Seed stability
|
||||
|
||||
- **Did the seed-specific preferred beta vary?** Yes: 0.50 (seed 0), 1.00 (seed 1),
|
||||
0.75 (seed 2).
|
||||
- **Was the pooled selection clear?** Arithmetically yes — `beta = 0.75` has the
|
||||
highest pooled mean and the lowest across-seed std, with no tie. Statistically it
|
||||
is **weak**: pooled means span only 0.544→0.579 (range ≈ 0.035), while the
|
||||
within-beta across-seed std is 0.075–0.100. Between-seed variance dominates the
|
||||
between-beta differences, so the pooled ordering is not strongly separated.
|
||||
- **Are effect directions stable?** Partly. The pooled validation rare-class
|
||||
macro-F1 rises from `beta = 0` to `0.75`, and the mean per-client delta is positive
|
||||
for `0.25–0.75` in all seeds. But the *shape* and the best beta differ by seed
|
||||
(seed 0 peaks at 0.50 and declines by 1.00; seed 1 increases monotonically; seed 2
|
||||
peaks at 0.75). Pooled validation rare-class recall actually decreases with beta.
|
||||
- **Is variance large relative to mean differences?** Yes. With only three seeds the
|
||||
standard error of the pooled mean is ≈ `std/√3` ≈ 0.043–0.058, of the same order as
|
||||
the total spread between beta candidates. The apparent "preference" for 0.75 is
|
||||
therefore not robustly distinguishable from noise in this condition.
|
||||
|
||||
---
|
||||
|
||||
## J. Runtime and resources
|
||||
|
||||
Measured with `/usr/bin/time -v`, CPU only, no GPU, no swap, exit status 0 for every
|
||||
run.
|
||||
|
||||
| seed | run | wall-clock | CPU % | peak RSS |
|
||||
|---|---|---|---|---|
|
||||
| 0 | `original_protean` (reused) | 42:06.78 | 740 % | 6.21 GiB |
|
||||
| 0 | beta 0.00 | 41:17 | 740 % | 6.21 GiB |
|
||||
| 0 | beta 0.25 | 42:11 | 738 % | 6.21 GiB |
|
||||
| 0 | beta 0.50 | 42:57 | 737 % | 6.21 GiB |
|
||||
| 0 | beta 0.75 | 43:45 | 733 % | 6.21 GiB |
|
||||
| 0 | beta 1.00 | 42:08 | 737 % | 6.21 GiB |
|
||||
| 1 | `original_protean` | 41:21.16 | 738 % | 7.48 GiB |
|
||||
| 1 | beta 0.00 | 40:38.89 | 744 % | 7.48 GiB |
|
||||
| 1 | beta 0.25 | 40:32.74 | 744 % | 7.48 GiB |
|
||||
| 1 | beta 0.50 | 40:43.99 | 743 % | 7.48 GiB |
|
||||
| 1 | beta 0.75 | 40:41.36 | 745 % | 7.48 GiB |
|
||||
| 1 | beta 1.00 | 41:08.86 | 744 % | 7.48 GiB |
|
||||
| 2 | `original_protean` | 41:09.70 | 743 % | 6.45 GiB |
|
||||
| 2 | beta 0.00 | 40:40.14 | 742 % | 6.45 GiB |
|
||||
| 2 | beta 0.25 | 41:25.22 | 740 % | 6.45 GiB |
|
||||
| 2 | beta 0.50 | 40:23.42 | 742 % | 6.45 GiB |
|
||||
| 2 | beta 0.75 | 41:05.82 | 740 % | 6.45 GiB |
|
||||
| 2 | beta 1.00 | 42:20.66 | 741 % | 6.45 GiB |
|
||||
|
||||
- Seed-0 runs were reused from Session 5 and were not re-run.
|
||||
- New compute this session: 12 runs, ≈ 8 h 11 m wall-clock (seed-1 chain ≈ 4 h 04 m,
|
||||
seed-2 chain ≈ 4 h 07 m), single-run sequential.
|
||||
- RSS rises to ≈ 7.48 GiB for seed 1; this is dataset/partition dependent, not
|
||||
beta dependent, and well within the 29 GiB host.
|
||||
- Preprocessing/load remains negligible next to training. No warnings, no client
|
||||
failures, 9/9 global prototypes in every round, no retries.
|
||||
|
||||
---
|
||||
|
||||
## K. Limitations
|
||||
|
||||
- **Only X-IIoTID**, **only alpha = 0.50**, **only three project-defined seeds**
|
||||
(`{0,1,2}`). No cross-alpha and no cross-dataset conclusion.
|
||||
- The `S_j` participant/denominator rule
|
||||
(`S_j = {i | N_ij > 0 and valid C_ij}`, equal weights in the baseline) remains a
|
||||
**REPRODUCTION ASSUMPTION**, material in 53/162 class-condition cells (32.7 %).
|
||||
The `beta = 0` comparison against `original_protean` is conditional on both modes
|
||||
sharing this rule and does **not** prove `beta = 0` reproduces original PROTEAN.
|
||||
- Train/validation/test adaptation (70/10/20 stratified) differs from the paper's
|
||||
literal 80/20; the validation eval partition is a project decision.
|
||||
- **No global optimum claim.** `beta = 0.75` is only the best among the five tested
|
||||
candidates under pooled validation for this one condition, and the between-seed
|
||||
variance is comparable to the between-beta spread.
|
||||
- The selected-beta test numbers are single-condition and must not be read as
|
||||
generalisation; no test metric was used for selection.
|
||||
- The union of rare classes (and hence global rare support) differs by seed because
|
||||
partitions differ; only the per-client-mean metric is directly comparable across
|
||||
seeds.
|
||||
- No 5G-NIDD beta sweep was run.
|
||||
|
||||
---
|
||||
|
||||
## L. `beta = 0` equivalence vs `original_protean` (seeds 1 and 2)
|
||||
|
||||
| check | seed 1 | seed 2 |
|
||||
|---|---|---|
|
||||
| same manifest / split / hashes | yes | yes |
|
||||
| round-0 snapshot byte-identical | yes (`b7f61c99…`) | yes (`5b749f41…`) |
|
||||
| `train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv` identical | yes | yes |
|
||||
| aggregation arithmetic | mathematically equivalent (`vec/count` vs `vec·(1/count)`) | same |
|
||||
| test / history / per-client / per-class | differ slightly (float ordering fed back through round≥1) | same |
|
||||
|
||||
Classified as **mathematically equivalent + floating-point numerical divergence**;
|
||||
**no semantic discrepancy** was found in either seed. Mechanism is identical to
|
||||
Session 5: round 0 is byte-identical, the ~1 ULP ordering difference enters the
|
||||
round-1 prototype-alignment training and trajectories diverge slowly.
|
||||
|
||||
---
|
||||
|
||||
## M. Analysis tooling
|
||||
|
||||
Read-only, no retraining:
|
||||
|
||||
- `scripts/analyze_multiseed_beta.py` — pooled validation tables, per-seed
|
||||
diagnostic selection, client validation summary/detail, cross-seed prototype-effect
|
||||
aggregation, test descriptive table, and `...__selection_summary.json`.
|
||||
- `scripts/analyze_prototype_effect.py` — per-seed direct prototype-effect CSVs from
|
||||
the fixed round-0 snapshot.
|
||||
|
||||
Outputs under `artifacts/analysis/xiiotid__alpha0.5__multiseed__*` and the per-seed
|
||||
`...__prototype_snapshot__prototype_effect.csv` / `...__prototype_weights.csv`.
|
||||
@@ -0,0 +1,319 @@
|
||||
# Session 6 Post-Experiment Audit
|
||||
|
||||
Status: **PASS WITH MINOR CORRECTIONS.** Independent recomputation from machine-readable
|
||||
artifacts confirms the Session 6 claims. No discovered issue changes `beta_star` or the
|
||||
scientific interpretation. One provenance gap was closed (runtime CSV) and one handoff
|
||||
wording nuance was clarified. **No training was run, no aggregation semantics changed, no
|
||||
test-based selection occurred.**
|
||||
|
||||
Audit performed: 2026-09-25. Audited HEAD: `50f2da72166e4f609077d65b4916f745da5e8e3b`.
|
||||
|
||||
Companion documents: `docs/research/MULTISEED_BETA_VALIDATION.md`,
|
||||
`docs/research/EXPERIMENT_PROTOCOL.md`, `docs/research/PAIRED_BETA_VALIDATION.md`,
|
||||
`docs/baseline/BASELINE_VALIDATION.md`, `docs/handoff/SESSION_HANDOFF.md`.
|
||||
|
||||
---
|
||||
|
||||
## A. Audit scope
|
||||
|
||||
Independently verify the Session 6 claim for:
|
||||
|
||||
- dataset `xiiotid`, alpha `0.50`, seeds `{0,1,2}`, beta `{0, 0.25, 0.5, 0.75, 1.0}`;
|
||||
- PRIMARY pooled-validation selection `beta_star = 0.75`, using validation per-client
|
||||
rare-class macro-F1 averaged within seed then across seeds.
|
||||
|
||||
The audit recomputed metrics from run artifacts and re-derived the selection; it did not
|
||||
repeat the report. No new condition was run.
|
||||
|
||||
---
|
||||
|
||||
## B. Run inventory
|
||||
|
||||
18 full-data runs were enumerated from `results/*/metadata.json` (15 `support_weighted` +
|
||||
3 `original_protean`). Every expected run is present, none duplicated, none partial.
|
||||
|
||||
| seed | runs present | metadata fields | manifest reload | smoke | missing artifacts |
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | original + beta {0,.25,.5,.75,1} | all correct | true | false | none |
|
||||
| 1 | original + beta {0,.25,.5,.75,1} | all correct | true | false | none |
|
||||
| 2 | original + beta {0,.25,.5,.75,1} | all correct | true | false | none |
|
||||
|
||||
Verified per run: dataset=xiiotid, alpha=0.5, seed, aggregation mode, beta, clients=10,
|
||||
rounds=10, local_epochs=3, batch=256, lr=1e-3, mu=0.1, lambda=1.0, `manifest_reload_ok=true`,
|
||||
`smoke=false`. No run exists for any alpha ≠ 0.50 and no `5gnidd...support_weighted` run
|
||||
exists.
|
||||
|
||||
Informational: the seed-0 `original_protean` `metadata.json` lacks the `beta` key because
|
||||
it predates the Session-5 validation-artifact change; the other 17 runs carry `beta`
|
||||
(`null` for `original_protean`). No scientific impact.
|
||||
|
||||
---
|
||||
|
||||
## C. Pairing / manifest integrity
|
||||
|
||||
Within each seed, all six runs share one manifest content hash and the deterministic
|
||||
artifacts are byte-identical:
|
||||
|
||||
| seed | manifest content hash | round-0 snapshot SHA-256 (all 6 runs) | support/characterization/rare_unseen identical |
|
||||
|---|---|---|---|
|
||||
| 0 | `e81a6538…` | `e27f41b3a62eb83c…` | yes |
|
||||
| 1 | `54ea1df1…` | `b7f61c995e97474c…` | yes |
|
||||
| 2 | `be881ac6…` | `5b749f4185e11fc7…` | yes |
|
||||
|
||||
`train_support_Nij.csv` equals the manifest support matrix for each seed. `history.json`
|
||||
and `test_global_metrics.json` differ across runs of a seed, exactly as expected because
|
||||
beta/original changes training. Pairing is proven by hashes, not filenames.
|
||||
|
||||
Reproducing each manifest from the raw dataset (`prepare_dataset` → `reproduce_train_assignments`):
|
||||
all three return `reproduce_ok=True`, `support_match=True`, sizes 569,909 / 81,414 / 162,831,
|
||||
10 clients; train/validation/test are **pairwise disjoint** and their union is the whole
|
||||
cleaned dataset (814,154 rows).
|
||||
|
||||
---
|
||||
|
||||
## D. Validation metric recomputation
|
||||
|
||||
Recomputed from raw artifacts (no report numbers copied):
|
||||
|
||||
- seed-level primary score = mean over eligible clients of `val_per_client.csv`
|
||||
`rare_class_macro_f1`; **equals `val_selection.json` per-client-mean exactly (diff 0.0)**;
|
||||
- global validation accuracy / macro-F1 / weighted-F1 / rare macro-F1 / rare recall
|
||||
recomputed **independently from `val_confusion_matrix.json`**; **equal to
|
||||
`val_global_metrics.json` exactly (diff 0.0)**.
|
||||
|
||||
| seed | beta | eligible | recomputed seed mean | report claim |
|
||||
|---|---|---|---|---|
|
||||
| 0 | 0.00 / 0.25 / 0.50 / 0.75 / 1.00 | 10 each | 0.557340 / 0.571787 / 0.585762 / 0.578003 / 0.551443 | same |
|
||||
| 1 | 0.00 / 0.25 / 0.50 / 0.75 / 1.00 | 10 each | 0.628624 / 0.636205 / 0.645189 / 0.655160 / 0.683117 | same |
|
||||
| 2 | 0.00 / 0.25 / 0.50 / 0.75 / 1.00 | 10 each | 0.447118 / 0.458310 / 0.471015 / 0.504937 / 0.485877 | same |
|
||||
|
||||
All recomputed values agree with the report to ≤ 4e-7 (six-decimal formatting precision).
|
||||
|
||||
Limitation (unavoidable, documented): the *per-client predictions* cannot be recomputed
|
||||
because the trained model is not persisted and retraining is forbidden by the audit. The
|
||||
per-client metric is therefore verified via (a) its defining source code
|
||||
(`evaluation/metrics.py::rare_class_metrics` over each client's training-derived rare ids),
|
||||
(b) exact equality of its mean with `val_selection.json`, and (c) exact equality of the
|
||||
global union-rare metric recomputed from the confusion matrix. This does not affect
|
||||
`beta_star`, which depends only on the seed-level means.
|
||||
|
||||
---
|
||||
|
||||
## E. Rare-class integrity
|
||||
|
||||
Per-client rare classes recomputed from training `N_ij` (two smallest positive-support
|
||||
attack classes, tie by class id) and compared to `rare_unseen.csv` flags:
|
||||
|
||||
- **0 mismatches** across all 30 client-seed cells.
|
||||
- Benign class (`Normal`) never flagged rare/unseen; rare and unseen never overlap.
|
||||
- Unseen = attack class with `N_ij = 0`; union of rare classes derived from training equals
|
||||
both `val_global_metrics.evaluated_rare_classes` and
|
||||
`test_global_metrics.evaluated_rare_classes` for every seed — i.e. validation/test labels
|
||||
did **not** influence rare-class identity.
|
||||
- Eligible clients = **10 in every seed** (no client excluded); every client has exactly 2
|
||||
rare classes; no client has only 1 attack class.
|
||||
- Union rare classes: seed 0 = `{C&C, Exfiltration, Exploitation, Lateral_movement,
|
||||
Tampering, Weaponization}`; seeds 1 and 2 additionally include `RDOS`. Hence global test
|
||||
rare support 25,454 (seed 0) vs 53,706 (seeds 1,2).
|
||||
|
||||
---
|
||||
|
||||
## F. Pooled-selection recomputation
|
||||
|
||||
Computed from the independently recomputed seed-level scores (not from the report).
|
||||
|
||||
| beta | pooled mean | sample std (ddof=1) | pop std (ddof=0) | min | max |
|
||||
|---|---|---|---|---|---|
|
||||
| 0.00 | 0.544361 | 0.091447 | 0.074666 | 0.447118 | 0.628624 |
|
||||
| 0.25 | 0.555434 | 0.090068 | 0.073540 | 0.458310 | 0.636205 |
|
||||
| 0.50 | 0.567322 | 0.088539 | 0.072292 | 0.471015 | 0.645189 |
|
||||
| **0.75** | **0.579367** | **0.075121** | 0.061336 | 0.504937 | 0.655160 |
|
||||
| 1.00 | 0.573479 | 0.100450 | 0.082017 | 0.485877 | 0.683117 |
|
||||
|
||||
- Recompute **matches the report** (mean and sample std) to ≤ 4e-7.
|
||||
- `argmax` of the pooled validation score = **0.75** (no tie).
|
||||
- Because every seed has the same eligible count (10), the required
|
||||
"average within seed, then across seeds" equals a flat average over all clients; the
|
||||
flat-pool sensitivity check is numerically identical here. The required hierarchy was
|
||||
still the implemented procedure.
|
||||
- **Std convention:** the report uses **sample** std (ddof=1), explicitly labelled
|
||||
`std (sample)` in the report table. Other std uses in the repository
|
||||
(`characterization.py`, `preprocessing.py`) are population std (ddof=0) for support
|
||||
statistics. No prior multi-seed std convention existed, and the label is present, so no
|
||||
convention was silently changed. Recommendation: keep the "(sample)" label wherever the
|
||||
pooled std is quoted.
|
||||
|
||||
---
|
||||
|
||||
## G. Validation/test leakage audit
|
||||
|
||||
- `run_baseline.py`: the `val_selection` payload (lines 297–313) is built **only** from
|
||||
`val_rare`, `val_metrics`, `val_per_client_rows`. Test variables appear in a separate
|
||||
block and feed only test artifacts. `test_used_for_selection: false` and
|
||||
`selection_split: "validation"` are recorded for all 15 `support_weighted` runs.
|
||||
- `analyze_multiseed_beta.py`: the PRIMARY selection block (lines 176–208) consumes only
|
||||
`per_seed_rows`, which are built from `val_selection.json`. `test_rows` are read in a
|
||||
later block (line 348+) and are never referenced by the selection logic or tie-break.
|
||||
The pooled argmax is reproduced by rerunning this read-only script.
|
||||
- No code path sorts/filters candidates by a test field.
|
||||
|
||||
Conclusion: selection is **programmatic and validation-only**; the boolean flag is a
|
||||
literal but is corroborated by the independent recomputation of `beta_star` from validation
|
||||
artifacts alone. `test_used_for_selection = false` is upheld.
|
||||
|
||||
---
|
||||
|
||||
## H. Test metric recomputation
|
||||
|
||||
`beta_star = 0.75` test metrics recomputed independently from `test_confusion_matrix.json`
|
||||
(max diff vs the JSON = 0.0), then averaged across seeds with sample std:
|
||||
|
||||
| metric | recomputed mean ± std | report claim |
|
||||
|---|---|---|
|
||||
| rare-class macro-F1 | 0.665503 ± 0.026685 | 0.665503 ± 0.026685 |
|
||||
| rare-class recall | 0.862583 ± 0.013921 | 0.862583 ± 0.013921 |
|
||||
| macro-F1 | 0.731169 ± 0.006234 | 0.731169 ± 0.006234 |
|
||||
| accuracy | 0.906392 ± 0.018450 | 0.906392 ± 0.018450 |
|
||||
| weighted-F1 | 0.933632 ± 0.011230 | 0.933632 ± 0.011230 |
|
||||
|
||||
`original_protean` recomputed identically: rare macro-F1 0.642520 ± 0.028683; rare recall
|
||||
0.876957 ± 0.010771; macro-F1 0.708965 ± 0.001172; accuracy 0.899454 ± 0.008398;
|
||||
weighted-F1 0.924733 ± 0.006245 — all match the report. No superiority claim is made.
|
||||
|
||||
---
|
||||
|
||||
## I. beta = 0 equivalence audit
|
||||
|
||||
For seeds 0, 1 and 2, `original_protean` vs `support_weighted beta=0`:
|
||||
|
||||
| property | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| same manifest/split | yes | yes | yes |
|
||||
| round-0 snapshot byte-identical | yes | yes | yes |
|
||||
| support/characterization/rare_unseen identical | yes | yes | yes |
|
||||
| independent aggregation diff (`ΣC/count` vs `Σ(1/count)C`) | ≤2.2e-16 | ≤2.2e-16 | ≤4.4e-16 |
|
||||
| test/history/per-client | differ slightly | differ slightly | differ slightly |
|
||||
| classification | **B** | **B** | **B** |
|
||||
|
||||
History divergence begins after round 0 (first-round rare macro-F1 abs diff 0.0 at round 0,
|
||||
growing to ~2e-4–3.6e-3 by round 9, max 6e-3–1.8e-2): floating-point ordering feedback, not
|
||||
a semantic change. Class **B** (semantically equivalent + floating-point divergence).
|
||||
Equivalence remains conditional on the shared `S_j` reproduction assumption.
|
||||
|
||||
---
|
||||
|
||||
## J. Direct-effect audit
|
||||
|
||||
Recomputed independently from the fixed round-0 snapshots (own `tensordot` implementation):
|
||||
|
||||
- per `(class, beta)` displacement matches `...__prototype_effect.csv` **exactly**
|
||||
(all classes, all betas, all seeds); rank order per seed matches;
|
||||
- normalised weights sum to 1 for every beta; `support_min/max` match the CSV;
|
||||
- the same snapshot is reused across beta candidates (byte-identical hashes, §C).
|
||||
|
||||
Cross-seed mean `||Cbar(1) − Cbar(0)||`: Lateral_movement 2.724 (std 1.035), RDOS 2.631
|
||||
(1.165), Weaponization 2.561 (1.228), Tampering 2.354 (1.165), Exploitation 2.344 (0.412),
|
||||
Exfiltration 2.020 (0.502), C&C 1.694 (0.505), Normal 1.557 (0.548), Reconnaissance 1.002
|
||||
(0.261). All match the report. Rank order is seed-sensitive (seed 0 top = Weaponization,
|
||||
seed 1 = RDOS, seed 2 = Lateral_movement). Displacement is a direct-aggregation measure
|
||||
only; no detection causality is inferred.
|
||||
|
||||
---
|
||||
|
||||
## K. Client-level audit
|
||||
|
||||
Recomputed from `val_per_client.csv` deltas vs `beta = 0`:
|
||||
|
||||
| seed | beta | improve | degrade | unchanged (ties) | total | report table |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0 | 0.75 | 7 | 3 | 0 | 10 | 7/3/0 |
|
||||
| 1 | 0.75 | 8 | 1 | 1 (client 7) | 10 | 8/1/1 |
|
||||
| 2 | 0.75 | 8 | 1 | 1 (client 4) | 10 | 8/1/1 |
|
||||
| 0 | 1.00 | 4 | 6 | 0 | 10 | 4/6/0 |
|
||||
| 1 | 1.00 | 9 | 0 | 1 (client 7) | 10 | 9/0/1 |
|
||||
|
||||
The report's §H table already carries an explicit `unchanged` column that accounts for the
|
||||
ties; only the handoff §9 shorthand ("8/1") omitted them (clarified, §N). Mean Δ / min / max
|
||||
recomputed match report §H for all six shown cells. `client_validation_detail.csv` deltas
|
||||
match recomputation (0 mismatches). `min positive N_ij` vs ΔF1 is confirmed **non-monotonic**
|
||||
(e.g. seed 0 N=1 → −0.021, N=2 → +0.093, N=10 → −0.036; seed 2 N=7 → +0.163, N=10 → +0.177,
|
||||
N=135 → +0.096). Client ids are treated as per-seed realisations, not one population.
|
||||
|
||||
---
|
||||
|
||||
## L. Artifact provenance
|
||||
|
||||
Every reported quantity traces to a machine-readable artifact or a deterministic
|
||||
recomputation.
|
||||
|
||||
| Reported quantity | Source artifact(s) | Derivation | Verified |
|
||||
|---|---|---|---|
|
||||
| run existence / config | `results/<run>/metadata.json` | direct read | yes |
|
||||
| manifest content/partition/split hashes | `artifacts/partitions/xiiotid__alpha0.5__seed{0,1,2}.json` | module hash + `reproduce_train_assignments` | yes |
|
||||
| pairing proof | `artifacts/analysis/...__prototype_snapshot.json` | sha256 across 6 runs/seed | yes |
|
||||
| seed-level validation score | `results/.../val_per_client.csv` + `val_selection.json` | mean of per-client rare macro-F1 | yes |
|
||||
| validation global metrics | `results/.../val_confusion_matrix.json` | independent numpy recompute | yes |
|
||||
| pooled mean/std, `beta_star` | recomputed seed scores | mean/`stdev` over seeds, argmax | yes |
|
||||
| test metrics (beta 0.75, original) | `results/.../test_confusion_matrix.json` | independent recompute + mean/std | yes |
|
||||
| beta=0 equivalence | original vs beta0 artifacts/snapshots | sha256 + independent aggregation | yes |
|
||||
| direct-effect displacement | snapshots + `...__prototype_effect.csv` | independent `tensordot` recompute | yes |
|
||||
| client improve/degrade | `results/.../val_per_client.csv` | deltas vs beta0 | yes |
|
||||
| rare-class identity | `artifacts/analysis/...__rare_unseen.csv` + manifest | recompute definition | yes |
|
||||
| runtime / RSS | `artifacts/analysis/xiiotid__alpha0.5__multiseed__runtime.csv` (added by this audit; seed 0 from Session-5 docs, seeds 1–2 from `/usr/bin/time -v` logs) | parse logs | yes |
|
||||
|
||||
---
|
||||
|
||||
## M. Source / data integrity
|
||||
|
||||
- HEAD `50f2da72166e4f609077d65b4916f745da5e8e3b`; working-tree source blobs for
|
||||
`prototype_aggregation.py` (`c298967a…`) and `scripts/run_baseline.py` (`2b469e35…`)
|
||||
equal their HEAD blobs. Only docs/analysis files are modified/untracked.
|
||||
- `planningawal/` unchanged (`git status`/`git diff` empty).
|
||||
- Raw dataset SHA-256 unchanged: X-IIoTID `7b929005…`, 5G-NIDD `fa36f808…`.
|
||||
- `artifacts/partitions/` unmodified (tracked, no diff); manifests reproduced from raw data.
|
||||
- `equal_available` mode: absent. Beta candidates: `(0.0, 0.25, 0.50, 0.75, 1.00)` unchanged.
|
||||
- No alpha ≠ 0.50 result directory; no 5G-NIDD `support_weighted` run.
|
||||
|
||||
---
|
||||
|
||||
## N. Deviations / corrections found
|
||||
|
||||
1. **Provenance gap (corrected).** Runtime/RSS numbers existed only in Markdown and
|
||||
ephemeral `/tmp` logs. Added machine-readable
|
||||
`artifacts/analysis/xiiotid__alpha0.5__multiseed__runtime.csv` (seed 0 from Session-5
|
||||
docs; seeds 1–2 parsed from `/usr/bin/time -v` logs). No scientific value changed.
|
||||
2. **Handoff wording (clarified).** Handoff §9 wrote "8/1 (seed 1), 8/1 (seed 2)" omitting
|
||||
the one unchanged client each; corrected to "7/3/0, 8/1/1, 8/1/1" to match the report
|
||||
§H table, which already showed the `unchanged` column.
|
||||
3. **Std convention (documented, not changed).** Pooled std is sample (ddof=1), labelled
|
||||
"(sample)" in the report; other repo std uses are population. No prior multi-seed
|
||||
convention existed. Kept sample, labelled.
|
||||
4. **Metadata schema (informational).** Seed-0 `original_protean` lacks the `beta` key
|
||||
(older code); no impact.
|
||||
5. **Audit-side only:** two first-pass audit scripts had class-list/order bugs
|
||||
(client-0 class union; support-ordered vs id-ordered comparison); corrected within the
|
||||
audit. These were audit-script defects, not repository defects.
|
||||
|
||||
No arithmetic error, mislabeled metric, or incorrect derived CSV was found in the Session 6
|
||||
report or analysis artifacts.
|
||||
|
||||
---
|
||||
|
||||
## O. Final audit verdict
|
||||
|
||||
**PASS WITH MINOR CORRECTIONS.**
|
||||
|
||||
- Run inventory, pairing, manifest/split/data integrity: **PASS**.
|
||||
- Validation metric recomputation (seed-level and global): **PASS** (exact).
|
||||
- Rare-class integrity: **PASS** (0 mismatches; training-derived; no val/test influence).
|
||||
- Pooled selection: **PASS**; independent `beta_star = 0.75` using validation only.
|
||||
- Validation/test leakage: **PASS**; selection reads validation only and is reproducible.
|
||||
- Test summaries (selected beta and `original_protean`): **PASS**(exact).
|
||||
- beta=0 equivalence: **PASS**, class **B** (semantically equivalent + float divergence).
|
||||
- Direct-effect and client-level analyses: **PASS** (independent recompute exact).
|
||||
- Source/data integrity, pytest (75 passed, 0 failed, 0 skipped, 0 warnings): **PASS**.
|
||||
|
||||
The corrections were provenance/wording only and do not affect `beta_star` or the
|
||||
scientific interpretation. Session 6 is **safe to checkpoint** with the documented
|
||||
limitations (single dataset, single alpha, three project-defined seeds, weak effect
|
||||
relative to seed variance, `S_j` reproduction assumption, no global-optimum claim).
|
||||
Reference in new issue
Block a user