experiment: validate beta selection on one paired full condition
This commit is contained in:
willtanoe committed
1 parent
ad9aa53ba7
commit
50f2da7216
8 files changed
+12434
-150
No files matched your search
@@ -135,6 +135,29 @@ Procedure, strictly ordered:
|
||||
|
||||
Test data is never used in steps 1–2.
|
||||
|
||||
**Session-5 clarification (implementation detail, not a design change).** Running
|
||||
the protocol surfaced two concrete points that the original text left implicit.
|
||||
They do not alter the frozen design; they pin how step 1 is computed.
|
||||
|
||||
1. **Candidate validation metric.** For each beta, validation metrics are taken
|
||||
from the final trained model after the fixed round count (no best-round
|
||||
cherry-picking). Two validation readings are recorded: (a) the global
|
||||
union-rare validation rare-class macro-F1 (also stored in `history.json`), and
|
||||
(b) the primary selection metric, the **per-client mean** rare-class macro-F1,
|
||||
where each client is scored on its validation eval partition using that
|
||||
client's rare class ids and the result is averaged over clients that have rare
|
||||
classes. The per-client validation eval partition reuses the same deterministic
|
||||
Dirichlet eval-partition procedure already used for the per-client test
|
||||
evaluation, applied to the validation split (`split_name="validation"`); this
|
||||
is a PROJECT DECISION, paired across all beta candidates and seeds.
|
||||
2. **Tie-break order.** Higher validation rare-class macro-F1, then higher
|
||||
validation rare-class recall, then smaller beta.
|
||||
|
||||
`scripts/run_baseline.py` writes `val_global_metrics.json`, `val_per_class.csv`,
|
||||
`val_per_client.csv` and `val_selection.json` to make this explicit; the test
|
||||
path and the frozen training path are unchanged. See
|
||||
`docs/research/PAIRED_BETA_VALIDATION.md`.
|
||||
|
||||
## 9. Required metrics
|
||||
|
||||
Per dataset/alpha/seed/beta and per client, record:
|
||||
|
||||
@@ -0,0 +1,321 @@
|
||||
# Paired Beta Validation (Session 5)
|
||||
|
||||
Status: **ONE paired full condition run end to end** with validation-only beta
|
||||
selection, followed by a single final test evaluation of the selected beta. This
|
||||
is a **protocol-validation experiment**, not a sweep and not evidence of global
|
||||
behaviour.
|
||||
|
||||
Companion documents:
|
||||
|
||||
- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen protocol.
|
||||
- `docs/research/BETA_AGGREGATION_IMPLEMENTATION.md` — beta mode implementation.
|
||||
- `docs/baseline/BASELINE_VALIDATION.md` — frozen baseline evidence.
|
||||
- `docs/handoff/SESSION_HANDOFF.md` — continuation point.
|
||||
|
||||
---
|
||||
|
||||
## A. Condition
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| dataset | `xiiotid` (X-IIoTID) |
|
||||
| alpha | 0.50 |
|
||||
| seed | 0 |
|
||||
| aggregation modes | `original_protean` (reference), `support_weighted` for `beta ∈ {0, .25, .5, .75, 1}` |
|
||||
| clients / rounds / local epochs | 10 / 10 / 3 |
|
||||
| batch size / lr / optimizer | 256 / 1e-3 / Adam |
|
||||
| mu / lambda | 0.1 / 1.0 |
|
||||
| device | CPU only (`torch 2.14.0+cpu`, CUDA unavailable) |
|
||||
| code revision | HEAD `ad9aa53ba76f60671ace991fcee85f003781f196`; only `scripts/run_baseline.py` was modified this session (`git hash-object` `2b469e35ecbbc6ea3dbd4b0632419993e4d5130b`) to add validation-only evaluation artifacts. Training, partition, aggregation, loss and inference source are unchanged at `ad9aa53`. |
|
||||
|
||||
Frozen manifest reused (never regenerated):
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| path | `artifacts/partitions/xiiotid__alpha0.5__seed0.json` |
|
||||
| manifest content hash | `e81a6538ecefcda202aa7602310ff8d8c9529031ab9dcbec4ebbabc0e63b636c` |
|
||||
| partition hash | `2465485539a3e923548dcc0c73d8acc5076d456536b78682d512897ea586e779` |
|
||||
| Dirichlet rng seed | `2009062324` |
|
||||
| split hash | `d0acc0c14e3ecbeba02dfa9e6227c983da12dec2d6111df0b714002e6e03c29d` |
|
||||
| split sizes (train/val/test) | 569,909 / 81,414 / 162,831 |
|
||||
| preprocessing hash | `8580cee41c698ce43cf3269e4a2b3a01c92d6df22820f4004c0267478bd2579a` |
|
||||
| `manifest_reload_ok` | true for every beta run |
|
||||
|
||||
Pairing evidence: the **round-0 prototype snapshot is byte-identical**
|
||||
(SHA-256 prefix `e27f41b3a62eb83c`) for all five beta runs **and** the frozen
|
||||
`original_protean` run, so every candidate shares the same partition, split,
|
||||
model initialisation and round-0 local training. `train_support_Nij.csv`,
|
||||
`characterization.csv` and `rare_unseen.csv` are byte-identical across all runs.
|
||||
|
||||
### A.1 Validation-evaluation clarification (new this session)
|
||||
|
||||
The frozen runner previously wrote only test artifacts; `history.json` held a
|
||||
global (union-rare) validation trajectory. To evaluate the protocol's
|
||||
validation-only selection, `scripts/run_baseline.py` now additionally writes, at
|
||||
the end of the fixed 10 rounds:
|
||||
|
||||
- `val_global_metrics.json`, `val_confusion_matrix.json`, `val_per_class.csv`
|
||||
(global validation metrics over the union of per-client rare classes);
|
||||
- `val_per_client.csv` (per-client validation metrics, using the same
|
||||
deterministic Dirichlet eval-partition procedure already used for test,
|
||||
applied to the validation split with `split_name="validation"`);
|
||||
- `val_selection.json` (the selection summary).
|
||||
|
||||
The test path is untouched: re-running a smoke condition before/after the change
|
||||
produced byte-identical test artifacts (`history.json`, `test_global_metrics.json`,
|
||||
`test_confusion_matrix.json`, `test_per_class.csv`, `per_client.csv`,
|
||||
`train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv`,
|
||||
`metadata.json`). The candidate's validation metric is taken from the **final
|
||||
trained model after the fixed 10 rounds** (no best-round cherry-picking).
|
||||
|
||||
---
|
||||
|
||||
## B. Candidate results
|
||||
|
||||
`val rareMacroF1 (pc mean)` is the primary selection metric:
|
||||
per-client rare-class macro-F1 (each client scored on its validation eval
|
||||
partition using that client's rare class ids), averaged over clients with rare
|
||||
classes. The global column is the union-rare validation metric also recorded in
|
||||
`history.json`.
|
||||
|
||||
### B.1 USED FOR SELECTION — validation only
|
||||
|
||||
| beta | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 (global) | val accuracy | val weighted-F1 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | 0.557340 | 0.611641 | 0.884184 | 0.708929 | 0.891038 | 0.917741 |
|
||||
| 0.25 | 0.571787 | 0.629475 | 0.888488 | 0.723431 | 0.902953 | 0.925933 |
|
||||
| **0.50** | **0.585762** | **0.641709** | 0.887173 | **0.731166** | 0.897831 | **0.926005** |
|
||||
| 0.75 | 0.578003 | 0.637107 | 0.873910 | 0.726603 | 0.887894 | 0.921696 |
|
||||
| 1.00 | 0.551443 | 0.615307 | 0.847043 | 0.715951 | 0.883263 | 0.921182 |
|
||||
|
||||
Validation rare support: 12,726. Validation evaluated rare classes: `[0,1,2,3,7,8]`.
|
||||
Ten of ten clients had validation eval data and rare classes.
|
||||
|
||||
### B.2 NOT USED FOR SELECTION — test (produced automatically by the frozen pipeline)
|
||||
|
||||
| beta | test accuracy | test macro-F1 | test weighted-F1 | test rareMacroF1 | test rare recall | wall-clock |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | 0.891974 | 0.710481 | 0.918369 | 0.613575 | 0.873508 | 41:17 |
|
||||
| 0.25 | 0.903520 | 0.725299 | 0.926452 | 0.632154 | 0.877350 | 42:11 |
|
||||
| 0.50 | 0.898158 | 0.730743 | 0.926333 | 0.640829 | 0.871583 | 42:57 |
|
||||
| 0.75 | 0.888670 | 0.726298 | 0.922278 | 0.636189 | 0.857543 | 43:45 |
|
||||
| 1.00 | 0.884635 | 0.716383 | 0.921910 | 0.615509 | 0.832407 | 42:08 |
|
||||
|
||||
Test rare support: 25,454. The test columns are reported for transparency only and
|
||||
played no part in selecting beta.
|
||||
|
||||
---
|
||||
|
||||
## C. Beta selection
|
||||
|
||||
- **Selected beta: `0.50`** — best among the tested candidates under this
|
||||
validation condition.
|
||||
- **Criterion:** maximum validation per-client-mean rare-class macro-F1
|
||||
(`0.585762`). The secondary global union-rare validation metric independently
|
||||
also peaks at `beta = 0.50` (`0.641709`), so the two readings agree here.
|
||||
- **Tie-break (not used):** higher validation rare-class macro-F1, then higher
|
||||
validation rare-class recall, then smaller beta.
|
||||
- **Test was not used for selection.** `val_selection.json` records
|
||||
`"test_used_for_selection": false` for every run. Test metrics were not
|
||||
inspected until after `beta = 0.50` was fixed from validation.
|
||||
|
||||
---
|
||||
|
||||
## D. Final test evaluation (selected beta vs baseline)
|
||||
|
||||
Selected `beta = 0.50` (test) versus the frozen `original_protean` reference for
|
||||
the identical condition:
|
||||
|
||||
| metric | original_protean | support_weighted beta=0.50 | difference |
|
||||
|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.609620 | 0.640829 | +0.031209 |
|
||||
| rare-class recall | 0.871487 | 0.871583 | +0.000096 |
|
||||
| overall macro-F1 | 0.707701 | 0.730743 | +0.023042 |
|
||||
| accuracy | 0.890432 | 0.898158 | +0.007726 |
|
||||
| weighted-F1 | 0.917585 | 0.926333 | +0.008748 |
|
||||
|
||||
Per-class test results, selected beta (support in parentheses):
|
||||
|
||||
| class | precision | recall | F1 | support |
|
||||
|---|---|---|---|---|
|
||||
| C&C | 0.098 | 0.960 | 0.178 | 573 |
|
||||
| Exfiltration | 1.000 | 0.944 | 0.971 | 4,427 |
|
||||
| Exploitation | 0.017 | 0.581 | 0.033 | 227 |
|
||||
| Lateral_movement | 0.965 | 0.781 | 0.864 | 6,319 |
|
||||
| Normal | 0.971 | 0.935 | 0.953 | 83,651 |
|
||||
| RDOS | 0.992 | 0.997 | 0.995 | 28,252 |
|
||||
| Reconnaissance | 0.993 | 0.648 | 0.784 | 25,474 |
|
||||
| Tampering | 0.732 | 0.989 | 0.841 | 1,024 |
|
||||
| Weaponization | 0.941 | 0.973 | 0.957 | 12,884 |
|
||||
|
||||
Per-client test results, selected beta:
|
||||
|
||||
| client | n | accuracy | macro-F1 | rare macro-F1 | rare recall |
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | 22,614 | 0.8996 | 0.6158 | 0.5123 | 0.9886 |
|
||||
| 1 | 33,759 | 0.9265 | 0.6662 | 0.3998 | 0.6941 |
|
||||
| 2 | 5,625 | 0.9195 | 0.7840 | 0.9038 | 0.9760 |
|
||||
| 3 | 13,539 | 0.9846 | 0.8284 | 0.5481 | 0.5758 |
|
||||
| 4 | 11,881 | 0.7090 | 0.6334 | 0.1411 | 0.7973 |
|
||||
| 5 | 12,785 | 0.9200 | 0.7898 | 0.8331 | 0.9920 |
|
||||
| 6 | 6,131 | 0.9335 | 0.7243 | 0.4422 | 1.0000 |
|
||||
| 7 | 34,434 | 0.9145 | 0.6202 | 0.0477 | 0.8492 |
|
||||
| 8 | 5,090 | 0.9589 | 0.8038 | 0.6343 | 0.9532 |
|
||||
| 9 | 16,973 | 0.8158 | 0.5706 | 0.4051 | 0.6541 |
|
||||
|
||||
**No superiority is claimed.** These are single-dataset, single-alpha,
|
||||
single-seed results from one protocol-validation run.
|
||||
|
||||
---
|
||||
|
||||
## E. Direct prototype analysis (no retraining)
|
||||
|
||||
Computed from the fixed round-0 snapshot
|
||||
(`artifacts/analysis/xiiotid__alpha0.5__seed0__support_weighted__beta0__prototype_snapshot.json`,
|
||||
byte-identical across all beta runs) using the production aggregation; see
|
||||
`...__prototype_effect.csv` and `...__prototype_weights.csv` under
|
||||
`artifacts/analysis/`.
|
||||
|
||||
Per-class `||Cbar_j(beta) - Cbar_j(0)||` at `beta = 1.0` (relative displacement
|
||||
in parentheses), sorted by displacement:
|
||||
|
||||
| class | contributors | support min–max (cv) | disp@beta=1 | rel disp |
|
||||
|---|---|---|---|---|
|
||||
| Weaponization | 10 | 18–32,409 (2.10) | 3.559 | 0.577 |
|
||||
| C&C | 10 | 1–966 (1.50) | 2.156 | 0.370 |
|
||||
| Lateral_movement | 10 | 49–8,385 (1.22) | 2.069 | 0.371 |
|
||||
| Exploitation | 8 | 10–353 (1.09) | 1.934 | 0.342 |
|
||||
| Tampering | 9 | 2–1,266 (1.00) | 1.737 | 0.341 |
|
||||
| Exfiltration | 9 | 45–6,305 (1.04) | 1.547 | 0.214 |
|
||||
| RDOS | 10 | 168–49,283 (1.38) | 1.286 | 0.148 |
|
||||
| Reconnaissance | 10 | 1,137–29,234 (1.16) | 1.184 | 0.194 |
|
||||
| Normal | 10 | 604–68,657 (0.76) | 0.990 | 0.164 |
|
||||
|
||||
- **Largest direct displacement:** `Weaponization` (also the most support-imbalanced
|
||||
class, cv 2.10, max/min 1,800). For that class at `beta = 1.0` the single
|
||||
largest-support client (N=32,409) holds weight `0.719` versus `0.100` at
|
||||
`beta = 0`, while a client with N=18 drops to `0.0004`.
|
||||
- Classes with more balanced support (`Normal`, cv 0.76; `RDOS`, cv 1.38 with one
|
||||
dominant client) move less.
|
||||
- **No class has zero direct effect:** every class has `|S_j| ≥ 8` and unequal
|
||||
supports, so all nine prototypes shift for `beta > 0`. At `beta = 1.0`, no
|
||||
direct-effect displacement exceeded `0.58` of the class prototype norm.
|
||||
- **Displacement is a direct aggregation measure only** and is not interpreted as
|
||||
improved or degraded detection. The end-to-end effect (§D) is reported
|
||||
separately.
|
||||
|
||||
---
|
||||
|
||||
## F. Client-level analysis
|
||||
|
||||
Per-client test rarity metrics relative to `beta = 0`
|
||||
(`artifacts/analysis/xiiotid__alpha0.5__seed0__paired_client_beta_metrics.csv`
|
||||
contains the full paired table including validation columns and `N_ij`).
|
||||
|
||||
Delta test rare-class F1 vs `beta = 0` (negative = worse):
|
||||
|
||||
| client | min positive N_ij | beta=0 rare F1 | Δβ=0.25 | Δβ=0.50 | Δβ=0.75 | Δβ=1.00 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0 | 18 | 0.4996 | +0.0115 | +0.0127 | −0.0010 | −0.0177 |
|
||||
| 1 | 69 | 0.3844 | +0.0124 | +0.0154 | +0.0107 | +0.0061 |
|
||||
| 2 | 5 | 0.8823 | +0.0109 | +0.0215 | +0.0126 | −0.0364 |
|
||||
| 3 | 140 | 0.6162 | −0.0066 | −0.0681 | −0.2566 | −0.3422 |
|
||||
| 4 | 1 | 0.1375 | −0.0005 | +0.0036 | +0.0126 | −0.0143 |
|
||||
| 5 | 2 | 0.7126 | +0.0500 | +0.1204 | +0.1213 | +0.0622 |
|
||||
| 6 | 44 | 0.3788 | +0.0304 | +0.0634 | +0.0690 | +0.0325 |
|
||||
| 7 | 10 | 0.0470 | +0.0109 | +0.0006 | −0.0073 | −0.0140 |
|
||||
| 8 | 4 | 0.6316 | +0.0298 | +0.0027 | −0.0108 | −0.0274 |
|
||||
| 9 | 12 | 0.3910 | +0.0069 | +0.0141 | +0.0093 | +0.0027 |
|
||||
|
||||
Observations (descriptive, **not causal**):
|
||||
|
||||
- At `beta = 0.50`, six of ten clients improve and the only materially negative
|
||||
client is client 3 (−0.068); the per-client mean therefore peaks at
|
||||
`beta = 0.50` (§B.1).
|
||||
- At `beta = 0.75`/`1.00`, several clients lose ground, and the largest single
|
||||
loss is client 3 (−0.257 / −0.342). Client 3 has `min positive N_ij = 140`,
|
||||
so it is **not** the lowest-support client; conversely client 5
|
||||
(`min positive N_ij = 2`) is the largest gainer at `beta ≥ 0.5`.
|
||||
- The relationship between `min positive N_ij` and ΔF1 is **not monotonic** in
|
||||
this single condition. The data are recorded to enable the later, multi-seed
|
||||
analysis of whether high beta disproportionately harms low-support clients;
|
||||
no such claim is made here.
|
||||
|
||||
---
|
||||
|
||||
## G. Runtime and resources
|
||||
|
||||
Measured with `/usr/bin/time -v`, CPU only, no GPU, no swap.
|
||||
|
||||
| beta | wall-clock | CPU % | peak RSS |
|
||||
|---|---|---|---|
|
||||
| 0.00 | 41:17 | 740 % | 6.21 GiB |
|
||||
| 0.25 | 42:11 | 738 % | 6.21 GiB |
|
||||
| 0.50 | 42:57 | 737 % | 6.21 GiB |
|
||||
| 0.75 | 43:45 | 733 % | 6.21 GiB |
|
||||
| 1.00 | 42:08 | 737 % | 6.21 GiB |
|
||||
|
||||
Total: ~3 h 30 min for the five paired candidates, plus ~41 min for the initial
|
||||
beta=0 measurement run (same configuration; not counted twice in the table
|
||||
above beyond beta=0.00). Preprocessing/load is negligible next to training
|
||||
(Session 3 measured ~0.9 s read and ~1.3 s prepare). No warnings, no client
|
||||
failures, 9/9 global prototypes in every round, exit status 0.
|
||||
|
||||
---
|
||||
|
||||
## H. Limitations
|
||||
|
||||
- **Single dataset** (X-IIoTID), **single alpha** (0.50), **single seed** (0).
|
||||
- The `S_j` participant/denominator rule
|
||||
(`S_j = {i | N_ij > 0 and valid C_ij}`, equal weights in the baseline) is a
|
||||
**REPRODUCTION ASSUMPTION**, not verified original PROTEAN behaviour. It is
|
||||
material in 53/162 class-condition cells (32.7 %), 15/18 conditions
|
||||
(`docs/baseline/BASELINE_VALIDATION.md` §D). `beta = 0` equality with the
|
||||
baseline is **conditional** on both modes using this shared rule; the
|
||||
`beta = 0` vs `original_protean` comparison below does **not** prove the rule
|
||||
matches original PROTEAN.
|
||||
- **No claim of a global beta optimum.** `beta = 0.50` is only the best among the
|
||||
five tested candidates under this one validation procedure.
|
||||
- **No full experiment grid has been run** (no other alpha, seed, dataset).
|
||||
- The selected-beta test numbers are single-condition and may not generalise.
|
||||
- The validation per-client evaluation partition used for selection is a
|
||||
Session-5 clarification (mirrors the existing test per-client Dirichlet
|
||||
procedure); it is a project decision, not source-confirmed.
|
||||
- `beta = 0` under `support_weighted` is mathematically the equal mean but is
|
||||
**not bitwise identical** to `original_protean` (see §I); the difference is
|
||||
floating-point ordering, not a methodological change.
|
||||
|
||||
---
|
||||
|
||||
## I. `beta = 0` equivalence check (vs frozen `original_protean`)
|
||||
|
||||
| Property | Result |
|
||||
|---|---|
|
||||
| Same frozen manifest | **Yes** — identical manifest content/partition/split hashes |
|
||||
| Same initialisation / round-0 setup | **Yes** — round-0 prototype snapshot byte-identical (`e27f41b3a62eb83c`) |
|
||||
| Same training behaviour | **Mathematically yes, not bitwise.** `original_protean` computes `Σ(C_ij/count)`; `support_weighted beta=0` computes `Σ((1/count)·C_ij)`. Per-element difference is at most `2.2e-16`–`4.4e-16` (~1 ULP, relative ~1e-16) on the round-0 snapshot. |
|
||||
| Same validation metrics | **No** — tiny divergence (e.g. global rare macro-F1 0.611641 vs 0.608087) |
|
||||
| Same test metrics | **No** — accuracy 0.891974 vs 0.890432; rare macro-F1 0.613575 vs 0.609620 |
|
||||
| Same deterministic artifacts | Round-0 snapshot, `train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv` identical; `history.json` and per-client/per-class metrics differ |
|
||||
|
||||
Mechanism: round 0 is byte-identical, then the ~1 ULP aggregation difference
|
||||
feeds back into round-1 prototype-alignment training and the trajectories
|
||||
diverge slowly (differences grow from ~1e-4 at round 1 to ~1e-2 at round 9). This
|
||||
is floating-point arithmetic ordering (`vec/count` vs `vec*(1/count)`), amplified
|
||||
by iterative training, **not** a semantic difference in the aggregation formula.
|
||||
No code was changed to force bitwise equality; the frozen baseline arithmetic was
|
||||
left untouched.
|
||||
|
||||
**Caveat:** this equivalence is conditional on the shared `S_j` reproduction
|
||||
assumption and does not establish that `beta = 0` reproduces original PROTEAN.
|
||||
|
||||
---
|
||||
|
||||
## J. Verdict
|
||||
|
||||
The validation/test separation protocol ran cleanly end to end for this
|
||||
condition: candidates trained on the paired manifest, beta was selected using
|
||||
validation metrics only, and the selected beta was evaluated once on test. The
|
||||
protocol appears ready, with the documented clarifications (validation per-client
|
||||
evaluation partition; float-level `beta = 0` non-identity), to be scaled to
|
||||
multi-seed/multi-alpha experiments. No full grid was launched.
|
||||
Reference in new issue
Block a user