experiment: validate beta selection on one paired full condition
This commit is contained in:
willtanoe committed
1 parent
ad9aa53ba7
commit
50f2da7216
8 files changed
+12434
-150
No files matched your search
+176
-150
@@ -3,11 +3,11 @@
|
||||
This is the canonical continuation point for the next agent session. It is written to be
|
||||
readable without prior chat history, terminal scrollback, or git history.
|
||||
|
||||
Last updated: 2026-09-24 (Session 4, support-weighted beta aggregation implementation).
|
||||
Session 4 type: implementation + validation. The proposed `support_weighted`
|
||||
(`w_ij = N_ij ** beta`) mode was added **without changing the frozen `original_protean`
|
||||
baseline**. Only small smoke tests were run; **no beta sweep and no beta selection were
|
||||
performed**.
|
||||
Last updated: 2026-09-25 (Session 5, one paired full beta condition + validation-only
|
||||
selection). Session 5 type: **protocol-validation experiment**. Exactly one
|
||||
`(dataset, alpha, seed)` condition was run across all five beta candidates; beta was
|
||||
selected on validation only and the selected beta evaluated once on test. **No full grid
|
||||
was launched.**
|
||||
|
||||
---
|
||||
|
||||
@@ -24,204 +24,225 @@ S_j = { i | N_ij > 0 and a valid local prototype C_ij exists }
|
||||
beta in {0.0, 0.25, 0.5, 0.75, 1.0}
|
||||
```
|
||||
|
||||
Both the baseline and the proposed mode are now implemented. The baseline remains the
|
||||
reference; `beta = 0` is arithmetic-identical to it only because both use the same
|
||||
documented `S_j` participant rule.
|
||||
Both the frozen `original_protean` baseline and the proposed `support_weighted` mode are
|
||||
implemented. `beta = 0` is mathematically the equal mean over the shared `S_j` but is not
|
||||
bitwise identical to the baseline (see §7).
|
||||
|
||||
## 2. Authoritative source and immutability
|
||||
|
||||
- `planningawal/PROTEAN_Revisi_Lurus.md` — authoritative research specification.
|
||||
- `planningawal/` is immutable. Re-verified unchanged this session: `git status --
|
||||
planningawal` and `git diff -- planningawal` are both empty.
|
||||
- Raw datasets re-verified unchanged (SHA-256):
|
||||
`~/datasets/X-IIoTID/X-IIoTID-dataset.csv`
|
||||
`7b9290057ee42e784da3c0d84b781815502c9205c74175c96374e71a5ffd98a0`;
|
||||
`~/datasets/5G-NIDD/5G-NIDD.csv`
|
||||
`fa36f80859585f474504ca69eb951a079b35145537976c86b00aae3aab46ee59`.
|
||||
|
||||
## 3. Dataset locations (read-only external source data)
|
||||
## 3. Session 5 objective and outcome
|
||||
|
||||
- `~/datasets/X-IIoTID/X-IIoTID-dataset.csv` — SHA-256
|
||||
`7b9290057ee42e784da3c0d84b781815502c9205c74175c96374e71a5ffd98a0` (re-verified unchanged).
|
||||
- `~/datasets/5G-NIDD/5G-NIDD.csv` — SHA-256
|
||||
`fa36f80859585f474504ca69eb951a079b35145537976c86b00aae3aab46ee59` (re-verified unchanged).
|
||||
Objective: run ONE paired full condition (X-IIoTID, alpha=0.5, seed=0) across
|
||||
`beta ∈ {0, 0.25, 0.5, 0.75, 1}`, reusing the frozen manifest and training settings;
|
||||
select beta on validation only; evaluate the selected beta on test; add the direct
|
||||
prototype-effect and client-level analyses; update docs. **All acceptance items were met.**
|
||||
No commit or push was made.
|
||||
|
||||
## 4. Session 4 objective and outcome
|
||||
## 4. Condition and pairing
|
||||
|
||||
Objective: implement the proposed support-weighted prototype aggregation while preserving
|
||||
the frozen `original_protean` baseline, add the deferred beta-invariant tests, add direct
|
||||
prototype-effect instrumentation, run small smoke tests for beta 0 / 0.5 / 1, and update
|
||||
docs. **All acceptance items were met.** No commit was made.
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| dataset / alpha / seed | `xiiotid` / 0.50 / 0 |
|
||||
| manifest | `artifacts/partitions/xiiotid__alpha0.5__seed0.json` (reused) |
|
||||
| manifest content hash | `e81a6538ecefcda202aa7602310ff8d8c9529031ab9dcbec4ebbabc0e63b636c` |
|
||||
| partition hash | `2465485539a3e923548dcc0c73d8acc5076d456536b78682d512897ea586e779` |
|
||||
| split hash / sizes | `d0acc0c14e3ecbeba02dfa9e6227c983da12dec2d6111df0b714002e6e03c29d` / 569,909 train, 81,414 val, 162,831 test |
|
||||
| preprocessing hash | `8580cee41c698ce43cf3269e4a2b3a01c92d6df22820f4004c0267478bd2579a` |
|
||||
| training | 10 clients, 10 rounds, 3 local epochs, batch 256, lr 1e-3, mu 0.1, lambda 1.0, Adam |
|
||||
| code revision | HEAD `ad9aa53ba76f60671ace991fcee85f003781f196`; only `scripts/run_baseline.py` modified (hash-object `2b469e35ecbbc6ea3dbd4b0632419993e4d5130b`) |
|
||||
|
||||
## 5. Implementation summary
|
||||
Pairing proof: the round-0 prototype snapshot is **byte-identical** (SHA-256 prefix
|
||||
`e27f41b3a62eb83c`) across all five beta runs and the frozen `original_protean` run;
|
||||
`train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv` are identical across
|
||||
runs. All runs are in `results/xiiotid__alpha0.5__seed0__support_weighted__beta{0,0.25,0.5,0.75,1}/`.
|
||||
|
||||
Added a new aggregation mode and threaded it through the stack; the baseline code path is
|
||||
untouched.
|
||||
## 5. Validation results and beta selection
|
||||
|
||||
- `src/protean_research/aggregation/prototype_aggregation.py`: `AggregationMode.SUPPORT_
|
||||
WEIGHTED`, `BETA_CANDIDATES = (0.0, 0.25, 0.50, 0.75, 1.00)`, strict `validate_beta`
|
||||
(`[0, 1]`; rejects `None`/non-numeric/non-finite/out-of-range), `normalized_support_
|
||||
weights`; `aggregate_prototypes(..., supports=None, beta=None)` and `aggregate_class_
|
||||
prototypes(..., beta=None)`. `S_j` is built once and is identical to the baseline:
|
||||
`None`/`N_ij <= 0` entries dropped, no zero-vector padding. Passing `beta` to
|
||||
`original_protean` raises.
|
||||
- `src/protean_research/aggregation/__init__.py`: exports updated.
|
||||
- `src/protean_research/models/prototypes.py`, `federated/server.py`: forward `beta`.
|
||||
- `src/protean_research/federated/round.py`: `TrainingConfig.aggregation_mode` (default
|
||||
`original_protean`) and `.beta` (default `None`); the round loop dispatches to the mode.
|
||||
- `scripts/run_baseline.py`: new `--aggregation-mode {original_protean,support_weighted}`
|
||||
and `--beta`. Default behavior is the unchanged baseline. Run name/metadata record mode
|
||||
and beta (`...__original_protean` unchanged; `...__support_weighted__beta0.5` for beta).
|
||||
- `src/protean_research/analysis/direct_prototype.py` (new) +
|
||||
`scripts/analyze_prototype_effect.py` (new): recompute `Cbar_j(beta)` from a fixed round
|
||||
snapshot of `{C_ij, N_ij}` **without retraining**; emit `||Cbar_j(beta) - Cbar_j(0)||`,
|
||||
support stats, norms, and normalised client weights.
|
||||
Primary selection metric: **validation per-client-mean rare-class macro-F1** (per-client
|
||||
rare F1 on each client's validation eval partition, averaged over clients with rare
|
||||
classes). Global union-rare validation metrics also shown.
|
||||
|
||||
`equal_available` was **not** added (the baseline already is equal-over-`S_j`, so it equals
|
||||
`beta = 0`).
|
||||
| beta | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall | val macroF1 | val acc | val wF1 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | 0.557340 | 0.611641 | 0.884184 | 0.708929 | 0.891038 | 0.917741 |
|
||||
| 0.25 | 0.571787 | 0.629475 | 0.888488 | 0.723431 | 0.902953 | 0.925933 |
|
||||
| **0.50** | **0.585762** | **0.641709** | 0.887173 | **0.731166** | 0.897831 | **0.926005** |
|
||||
| 0.75 | 0.578003 | 0.637107 | 0.873910 | 0.726603 | 0.887894 | 0.921696 |
|
||||
| 1.00 | 0.551443 | 0.615307 | 0.847043 | 0.715951 | 0.883263 | 0.921182 |
|
||||
|
||||
## 6. Tests and baseline regression status
|
||||
**Selected beta = 0.50** (best among tested candidates under this validation condition;
|
||||
no tie; both validation readings agree). Test was **not** used for selection.
|
||||
|
||||
```
|
||||
.venv/bin/python -m pytest -> 75 passed
|
||||
```
|
||||
## 6. Final test evaluation of the selected beta
|
||||
|
||||
Session-3 suite was 46 tests. Added `tests/test_support_weighted_aggregation.py` (21) and
|
||||
`tests/test_prototype_effect.py` (7); one baseline-aggregation test was replaced by two
|
||||
explicit tests (support_weighted now available; `beta` rejected for `original_protean`).
|
||||
| metric | original_protean | support_weighted beta=0.50 |
|
||||
|---|---|---|
|
||||
| rare-class macro-F1 | 0.609620 | 0.640829 |
|
||||
| rare-class recall | 0.871487 | 0.871583 |
|
||||
| overall macro-F1 | 0.707701 | 0.730743 |
|
||||
| accuracy | 0.890432 | 0.898158 |
|
||||
| weighted-F1 | 0.917585 | 0.926333 |
|
||||
|
||||
Covered beta invariants: `beta=0` arithmetic mean; `beta=1` support-weighted mean;
|
||||
hand-computed intermediate-beta fixture; single-contributor invariance; equal-support
|
||||
invariance; missing-class exclusion; normalised weights sum to 1; invalid/non-numeric
|
||||
beta rejection; supports strictly positive and aligned. **Baseline regression is green**:
|
||||
`test_end_to_end_baseline_is_deterministic_and_learns_separable_fixture` and all baseline
|
||||
arithmetic/validation tests pass unchanged.
|
||||
All five candidates' test metrics (transparency only) are in
|
||||
`docs/research/PAIRED_BETA_VALIDATION.md` §B.2. No superiority is claimed: single
|
||||
dataset / alpha / seed.
|
||||
|
||||
## 7. Smoke results (NOT scientific results)
|
||||
## 7. `beta = 0` equivalence result (important)
|
||||
|
||||
Condition: `xiiotid`, `alpha=0.5`, `seed=0`, `--max-rows 3000 --sample-from 60000`,
|
||||
4 clients, 2 rounds, 1 local epoch (ignored dirs only).
|
||||
`support_weighted beta=0` vs frozen `original_protean`:
|
||||
|
||||
| mode / beta | test accuracy | test macro-F1 | test rare macro-F1 | test rare recall |
|
||||
|---|---|---|---|---|
|
||||
| `original_protean` | 0.5167 | 0.2998 | 0.3140 | 0.4572 |
|
||||
| `support_weighted` 0.0 | 0.5167 | 0.2998 | 0.3140 | 0.4572 |
|
||||
| `support_weighted` 0.5 | 0.2333 | 0.1397 | 0.1797 | 0.3193 |
|
||||
| `support_weighted` 1.0 | 0.0383 | 0.0252 | 0.0324 | 0.1995 |
|
||||
- Same frozen manifest and split: **yes**.
|
||||
- Same initialisation/round-0 setup: **yes** (round-0 snapshot byte-identical).
|
||||
- Training behaviour: **mathematically identical, not bitwise**. `original_protean`
|
||||
computes `Σ(C_ij/count)`; `support_weighted beta=0` computes `Σ((1/count)·C_ij)`.
|
||||
Per-element difference ≤ `2.2e-16`–`4.4e-16` (~1 ULP, relative ~1e-16).
|
||||
- Validation/test metrics: **not identical** (baseline slightly lower; e.g. test
|
||||
accuracy 0.890432 vs 0.891974). The ~1 ULP aggregation difference feeds back into
|
||||
round-1 prototype-alignment training and the trajectories diverge slowly from round 1.
|
||||
- Deterministic artifacts identical: round-0 snapshot, `train_support_Nij.csv`,
|
||||
`characterization.csv`, `rare_unseen.csv`; `history.json`/per-client/per-class differ.
|
||||
|
||||
Baseline-preservation evidence: the `original_protean` and `support_weighted beta=0` smoke
|
||||
runs produced **byte-identical** result artifacts (only `metadata.json` differed in
|
||||
mode/beta/run_name). The round-0 snapshots for `beta = 0, 0.5, 1` were **byte-identical**,
|
||||
confirming the paired comparison (same partition/split/init/round-0 training). The
|
||||
beta=0.5/1 metrics are toy-scale and must not be interpreted as method quality.
|
||||
This is floating-point arithmetic ordering, not a semantic change; no code was altered to
|
||||
force bitwise equality. **Conditional on the shared `S_j` reproduction assumption** and
|
||||
does not prove `beta = 0` reproduces original PROTEAN.
|
||||
|
||||
Direct-effect analysis was exercised on the `beta=0` smoke snapshot (9 classes,
|
||||
max displacement 7.65 at beta=1; relative displacement up to 1.51 for the most
|
||||
support-imbalanced class), written under the ignored smoke directory
|
||||
`artifacts/processed/smoke/analysis/`.
|
||||
## 8. Direct prototype-effect findings (fixed round-0 snapshot, no retraining)
|
||||
|
||||
## 8. Methodological decisions / assumptions carried forward
|
||||
At `beta = 1.0`, largest `||Cbar_j(1)-Cbar_j(0)||`:
|
||||
`Weaponization` 3.559 (rel 0.577; most support-imbalanced, cv 2.10, max/min 1,800),
|
||||
`C&C` 2.156, `Lateral_movement` 2.069, `Exploitation` 1.934, `Tampering` 1.737,
|
||||
`Exfiltration` 1.547; smallest: `Normal` 0.990 (cv 0.76), `RDOS` 1.286,
|
||||
`Reconnaissance` 1.184. Every class has `|S_j| ≥ 8` and unequal support, so no class has
|
||||
zero direct effect. Outputs:
|
||||
`artifacts/analysis/xiiotid__alpha0.5__seed0__support_weighted__beta0__prototype_snapshot__prototype_effect.csv`
|
||||
and `...__prototype_weights.csv`. Displacement is a direct-aggregation measure only, not
|
||||
a detection result.
|
||||
|
||||
## 9. Client-level findings (test, vs beta=0)
|
||||
|
||||
Full paired table: `artifacts/analysis/xiiotid__alpha0.5__seed0__paired_client_beta_metrics.csv`.
|
||||
At `beta=0.50` six of ten clients improve; only client 3 is materially negative (−0.068).
|
||||
At `beta=0.75/1.00` several clients lose, with client 3 worst (−0.257/−0.342). The
|
||||
`min positive N_ij` vs ΔF1 relationship is **not monotonic** in this single condition:
|
||||
client 3 (`min N_ij=140`) is the worst loser while client 5 (`min N_ij=2`) is the biggest
|
||||
gainer. Descriptive only; no causal claim.
|
||||
|
||||
## 10. Runtime / resources
|
||||
|
||||
CPU only (no GPU). Per run: 41–44 min wall-clock, ~733–740 % CPU, peak RSS ~6.21 GiB,
|
||||
no swap, exit 0. Total ~3.5 h for the five paired candidates. Preprocessing/load
|
||||
negligible.
|
||||
|
||||
## 11. Methodological decisions / assumptions
|
||||
|
||||
1. **`S_j` participant/denominator rule** (`S_j = {i | N_ij > 0 and valid C_ij}`, equal
|
||||
weights in the baseline) is a **REPRODUCTION ASSUMPTION**, not verified original
|
||||
behaviour. It is material in 53/162 class-condition cells (32.7%), 15/18 conditions
|
||||
(`docs/baseline/BASELINE_VALIDATION.md` §D). `beta = 0` equality with the baseline is
|
||||
conditional on this shared rule. Carry this caveat in all reporting.
|
||||
2. `N_ij` is **class-specific sample support**, not prototype quality.
|
||||
3. `beta` is one value for all classes/clients; candidates are frozen at
|
||||
`{0, 0.25, 0.5, 0.75, 1}`.
|
||||
4. `lambda = 1.0`, optimiser (Adam), lr (1e-3), batch (256) remain PROJECT DECISIONS.
|
||||
behaviour; material in 53/162 class-condition cells (32.7 %), 15/18 conditions
|
||||
(`docs/baseline/BASELINE_VALIDATION.md` §D). Carry this caveat in all reporting.
|
||||
2. `N_ij` is class-specific sample support, not prototype quality.
|
||||
3. Beta is one value for all classes/clients; candidates frozen at `{0,.25,.5,.75,1}`.
|
||||
4. `lambda=1.0`, Adam, lr 1e-3, batch 256 remain PROJECT DECISIONS.
|
||||
5. **New this session (protocol clarification, documented in
|
||||
`EXPERIMENT_PROTOCOL.md` §8):** candidate validation metric = final-model validation
|
||||
metrics; primary selection = per-client-mean rare-class macro-F1 using a deterministic
|
||||
Dirichlet validation eval partition (mirrors the existing test per-client procedure);
|
||||
tie-break = higher val rare macro-F1 → higher val rare recall → smaller beta.
|
||||
6. No `equal_available` mode was added.
|
||||
|
||||
## 9. Unresolved questions / blockers
|
||||
## 12. Unresolved questions / blockers
|
||||
|
||||
1. `S_j` reproduction ambiguity (above) — unresolved, no official PROTEAN code exists.
|
||||
2. Beta selection has **not** been run; no validation/test separation result exists yet.
|
||||
3. Smoke-scale metrics are unstable and not meaningful; a full paired condition is needed.
|
||||
4. Session-2 manifests store `preprocessing_hash = null` / `config_hash = null`; live
|
||||
preprocessing hash is recorded in run metadata only.
|
||||
1. `S_j` reproduction ambiguity — unresolved; no official PROTEAN code exists.
|
||||
2. `beta = 0` is not bitwise identical to `original_protean` (float ordering). Whether the
|
||||
thesis should report this explicitly, or whether an equivalence check should use a
|
||||
tolerance, is a documentation decision for later.
|
||||
3. Single condition only; no multi-seed/multi-alpha evidence yet.
|
||||
4. Session-2 manifests still store `preprocessing_hash = null` / `config_hash = null`.
|
||||
|
||||
## 10. Git status
|
||||
## 13. Git status
|
||||
|
||||
No commit or push was made. Working tree has 9 modified tracked files and 5 untracked
|
||||
paths from this session (listed by `git status`). `planningawal/` and both raw datasets
|
||||
are unchanged. `results/`, `artifacts/processed/` remain ignored. `artifacts/analysis/` is
|
||||
tracked (no new files were written there this session; smoke analysis went to
|
||||
`artifacts/processed/smoke/analysis/`).
|
||||
No commit or push was made. `git status --short`:
|
||||
|
||||
## 11. Files created / modified this session
|
||||
```
|
||||
M scripts/run_baseline.py
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__seed0__paired_client_beta_metrics.csv
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__seed0__support_weighted__beta0__prototype_snapshot.json
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__seed0__support_weighted__beta0__prototype_snapshot__prototype_effect.csv
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__seed0__support_weighted__beta0__prototype_snapshot__prototype_weights.csv
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__seed0__support_weighted__beta0.25__prototype_snapshot.json
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__seed0__support_weighted__beta0.5__prototype_snapshot.json
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__seed0__support_weighted__beta0.75__prototype_snapshot.json
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__seed0__support_weighted__beta1__prototype_snapshot.json
|
||||
```
|
||||
|
||||
`planningawal/` and both raw datasets are unchanged; `results/` and
|
||||
`artifacts/processed/` are ignored; `artifacts/analysis/` is tracked.
|
||||
|
||||
## 14. Files created / modified this session
|
||||
|
||||
Created:
|
||||
|
||||
- `src/protean_research/analysis/__init__.py`
|
||||
- `src/protean_research/analysis/direct_prototype.py`
|
||||
- `scripts/analyze_prototype_effect.py`
|
||||
- `tests/test_support_weighted_aggregation.py`
|
||||
- `tests/test_prototype_effect.py`
|
||||
- `docs/research/BETA_AGGREGATION_IMPLEMENTATION.md`
|
||||
- `docs/research/PAIRED_BETA_VALIDATION.md`
|
||||
- `artifacts/analysis/xiiotid__alpha0.5__seed0__paired_client_beta_metrics.csv`
|
||||
- five `artifacts/analysis/...__support_weighted__beta*__prototype_snapshot.json`
|
||||
- two `artifacts/analysis/...__prototype_snapshot__prototype_effect.csv` / `__prototype_weights.csv`
|
||||
|
||||
Modified:
|
||||
|
||||
- `src/protean_research/aggregation/prototype_aggregation.py`
|
||||
- `src/protean_research/aggregation/__init__.py`
|
||||
- `src/protean_research/models/prototypes.py`
|
||||
- `src/protean_research/federated/server.py`
|
||||
- `src/protean_research/federated/round.py`
|
||||
- `scripts/run_baseline.py`
|
||||
- `tests/test_baseline_aggregation.py`
|
||||
- `configs/experiments/experiment_protocol.yaml`
|
||||
- `docs/research/EXPERIMENT_PROTOCOL.md`
|
||||
- `scripts/run_baseline.py` (added validation-only evaluation artifacts; test/training paths untouched)
|
||||
- `docs/research/EXPERIMENT_PROTOCOL.md` (§8 Session-5 clarification)
|
||||
- `docs/handoff/SESSION_HANDOFF.md` (this file)
|
||||
|
||||
## 12. Exact next task
|
||||
## 15. Exact recommended next phase
|
||||
|
||||
**Recommended: run ONE paired full `(dataset, alpha, seed)` condition across
|
||||
`beta in {0, 0.25, 0.5, 0.75, 1}` under the validation-based selection protocol, then
|
||||
inspect.** Do not launch the full grid.
|
||||
**Recommendation: B — checkpoint Session 5, then proceed to a controlled
|
||||
multi-seed/multi-alpha experiment.** The protocol ran cleanly (validation/test separation
|
||||
held, artifacts reproducible, pairing proven by byte-identical round-0 snapshots, no test
|
||||
leakage into selection), so the next step is to scale to seeds `{0,1,2}` at a small alpha
|
||||
set, still paired. Do **not** run the entire `dataset × alpha × seed × beta` grid at once;
|
||||
grow it in controlled, inspectable increments.
|
||||
|
||||
Concretely:
|
||||
Before scaling: decide how to report the float-level `beta = 0` vs `original_protean`
|
||||
non-identity (§7), and whether to re-run `original_protean` per condition with the new
|
||||
validation artifacts (the existing baseline run predates the validation-artifact change).
|
||||
|
||||
1. Reuse the frozen manifest for one condition (e.g. X-IIoTID `alpha=0.5`, seed 0) and the
|
||||
frozen training settings (10 clients, 10 rounds, 3 local epochs, batch 256, lr 1e-3,
|
||||
mu 0.1, lambda 1.0). Measure the single-run cost first (Session 3 measured ~42 min /
|
||||
6.2 GiB on CPU for one baseline run; beta adds one run per candidate).
|
||||
2. Train each beta with `run_baseline.py --aggregation-mode support_weighted --beta ...`
|
||||
and record validation rare-class macro-F1; select beta on validation only.
|
||||
3. Evaluate the selected beta on test, and report all candidates transparently.
|
||||
4. Run `scripts/analyze_prototype_effect.py` on the round-0 snapshot for the direct
|
||||
aggregation effect, clearly separated from end-to-end results.
|
||||
5. Report the `S_j` reproduction caveat alongside every between-beta claim.
|
||||
|
||||
Do **not** add `equal_available`. Do **not** run the alpha x seed x dataset x beta grid.
|
||||
|
||||
## 13. Important commands
|
||||
## 16. Important commands
|
||||
|
||||
```bash
|
||||
# Tests (75 expected)
|
||||
.venv/bin/python -m pytest
|
||||
|
||||
# Smoke: baseline and beta candidates (ignored output dirs)
|
||||
# One full paired beta run (CPU, ~42 min, ~6.2 GiB)
|
||||
.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.5 --seed 0 \
|
||||
--max-rows 3000 --sample-from 60000 --clients 4 --rounds 2 --local-epochs 1
|
||||
.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.5 --seed 0 \
|
||||
--max-rows 3000 --sample-from 60000 --clients 4 --rounds 2 --local-epochs 1 \
|
||||
--clients 10 --rounds 10 --local-epochs 3 \
|
||||
--aggregation-mode support_weighted --beta 0.5
|
||||
|
||||
# Direct prototype-effect analysis (no retraining)
|
||||
.venv/bin/python scripts/analyze_prototype_effect.py \
|
||||
--snapshot artifacts/processed/smoke/analysis/\
|
||||
xiiotid__alpha0.5__seed0__support_weighted__beta0__prototype_snapshot.json \
|
||||
--output-dir artifacts/processed/smoke/analysis
|
||||
|
||||
# Full-data baseline for one condition (measure first; ~40-65 min, CPU)
|
||||
.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.5 --seed 0 \
|
||||
--clients 10 --rounds 10 --local-epochs 3
|
||||
--snapshot artifacts/analysis/xiiotid__alpha0.5__seed0__support_weighted__beta0__prototype_snapshot.json \
|
||||
--class-names "C&C,Exfiltration,Exploitation,Lateral_movement,Normal,RDOS,Reconnaissance,Tampering,Weaponization" \
|
||||
--output-dir artifacts/analysis
|
||||
```
|
||||
|
||||
## 14. Recommended next-session prompt
|
||||
## 17. Recommended next-session prompt
|
||||
|
||||
```
|
||||
You are continuing the perdalam-protean research repository (Session 5).
|
||||
You are continuing the perdalam-protean research repository (Session 6).
|
||||
|
||||
FIRST, read completely:
|
||||
1. docs/handoff/SESSION_HANDOFF.md
|
||||
2. docs/research/BETA_AGGREGATION_IMPLEMENTATION.md
|
||||
3. docs/baseline/BASELINE_VALIDATION.md
|
||||
4. docs/research/EXPERIMENT_PROTOCOL.md
|
||||
2. docs/research/PAIRED_BETA_VALIDATION.md
|
||||
3. docs/research/EXPERIMENT_PROTOCOL.md
|
||||
4. docs/research/BETA_AGGREGATION_IMPLEMENTATION.md
|
||||
5. docs/baseline/BASELINE_VALIDATION.md
|
||||
|
||||
Critical rules:
|
||||
- planningawal/ is immutable. Do not edit/create/move/rename/delete anything inside it.
|
||||
@@ -231,14 +252,19 @@ Critical rules:
|
||||
class-condition cells). Carry this caveat in all reporting.
|
||||
- Do NOT add an executable equal_available mode.
|
||||
- Do NOT launch the full alpha x seed x dataset x beta grid.
|
||||
- Beta selection uses validation metrics only; test never influences selection.
|
||||
|
||||
Task: the support_weighted mode (w_ij = N_ij^beta) is implemented and unit/smoke
|
||||
validated; original_protean is unchanged. Run ONE paired full (dataset, alpha, seed)
|
||||
condition across beta in {0, 0.25, 0.5, 0.75, 1}, reusing the frozen manifest and frozen
|
||||
training settings. Select beta on validation rare-class macro-F1 only, then evaluate the
|
||||
selected beta on test and report all candidates. Add the direct prototype-effect analysis
|
||||
from the round-0 snapshot. Measure runtime before scaling. Update docs and
|
||||
docs/handoff/SESSION_HANDOFF.md.
|
||||
Context: Session 5 ran one paired full condition (xiiotid, alpha=0.5, seed=0) across
|
||||
beta in {0,0.25,0.5,0.75,1}. Selected beta = 0.50 on validation per-client-mean
|
||||
rare-class macro-F1; test rare-class macro-F1 0.640829 vs original_protean 0.609620.
|
||||
beta=0 is mathematically but not bitwise equal to original_protean (~1 ULP aggregation
|
||||
ordering). scripts/run_baseline.py now writes validation-only artifacts.
|
||||
|
||||
Task: scale the paired protocol to a controlled multi-seed/multi-alpha set (e.g. seeds
|
||||
{0,1,2} at alpha 0.50, reusing frozen manifests; grow only as compute allows), keeping
|
||||
validation-only beta selection and reporting all candidates. Decide/document how to
|
||||
report the beta=0 vs original_protean float non-identity. Update docs and
|
||||
docs/handoff/SESSION_HANDOFF.md. Measure runtime before scaling further.
|
||||
|
||||
Use .venv/bin/python for all Python commands. Do not commit or push unless asked.
|
||||
```
|
||||
@@ -135,6 +135,29 @@ Procedure, strictly ordered:
|
||||
|
||||
Test data is never used in steps 1–2.
|
||||
|
||||
**Session-5 clarification (implementation detail, not a design change).** Running
|
||||
the protocol surfaced two concrete points that the original text left implicit.
|
||||
They do not alter the frozen design; they pin how step 1 is computed.
|
||||
|
||||
1. **Candidate validation metric.** For each beta, validation metrics are taken
|
||||
from the final trained model after the fixed round count (no best-round
|
||||
cherry-picking). Two validation readings are recorded: (a) the global
|
||||
union-rare validation rare-class macro-F1 (also stored in `history.json`), and
|
||||
(b) the primary selection metric, the **per-client mean** rare-class macro-F1,
|
||||
where each client is scored on its validation eval partition using that
|
||||
client's rare class ids and the result is averaged over clients that have rare
|
||||
classes. The per-client validation eval partition reuses the same deterministic
|
||||
Dirichlet eval-partition procedure already used for the per-client test
|
||||
evaluation, applied to the validation split (`split_name="validation"`); this
|
||||
is a PROJECT DECISION, paired across all beta candidates and seeds.
|
||||
2. **Tie-break order.** Higher validation rare-class macro-F1, then higher
|
||||
validation rare-class recall, then smaller beta.
|
||||
|
||||
`scripts/run_baseline.py` writes `val_global_metrics.json`, `val_per_class.csv`,
|
||||
`val_per_client.csv` and `val_selection.json` to make this explicit; the test
|
||||
path and the frozen training path are unchanged. See
|
||||
`docs/research/PAIRED_BETA_VALIDATION.md`.
|
||||
|
||||
## 9. Required metrics
|
||||
|
||||
Per dataset/alpha/seed/beta and per client, record:
|
||||
|
||||
@@ -0,0 +1,321 @@
|
||||
# Paired Beta Validation (Session 5)
|
||||
|
||||
Status: **ONE paired full condition run end to end** with validation-only beta
|
||||
selection, followed by a single final test evaluation of the selected beta. This
|
||||
is a **protocol-validation experiment**, not a sweep and not evidence of global
|
||||
behaviour.
|
||||
|
||||
Companion documents:
|
||||
|
||||
- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen protocol.
|
||||
- `docs/research/BETA_AGGREGATION_IMPLEMENTATION.md` — beta mode implementation.
|
||||
- `docs/baseline/BASELINE_VALIDATION.md` — frozen baseline evidence.
|
||||
- `docs/handoff/SESSION_HANDOFF.md` — continuation point.
|
||||
|
||||
---
|
||||
|
||||
## A. Condition
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| dataset | `xiiotid` (X-IIoTID) |
|
||||
| alpha | 0.50 |
|
||||
| seed | 0 |
|
||||
| aggregation modes | `original_protean` (reference), `support_weighted` for `beta ∈ {0, .25, .5, .75, 1}` |
|
||||
| clients / rounds / local epochs | 10 / 10 / 3 |
|
||||
| batch size / lr / optimizer | 256 / 1e-3 / Adam |
|
||||
| mu / lambda | 0.1 / 1.0 |
|
||||
| device | CPU only (`torch 2.14.0+cpu`, CUDA unavailable) |
|
||||
| code revision | HEAD `ad9aa53ba76f60671ace991fcee85f003781f196`; only `scripts/run_baseline.py` was modified this session (`git hash-object` `2b469e35ecbbc6ea3dbd4b0632419993e4d5130b`) to add validation-only evaluation artifacts. Training, partition, aggregation, loss and inference source are unchanged at `ad9aa53`. |
|
||||
|
||||
Frozen manifest reused (never regenerated):
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| path | `artifacts/partitions/xiiotid__alpha0.5__seed0.json` |
|
||||
| manifest content hash | `e81a6538ecefcda202aa7602310ff8d8c9529031ab9dcbec4ebbabc0e63b636c` |
|
||||
| partition hash | `2465485539a3e923548dcc0c73d8acc5076d456536b78682d512897ea586e779` |
|
||||
| Dirichlet rng seed | `2009062324` |
|
||||
| split hash | `d0acc0c14e3ecbeba02dfa9e6227c983da12dec2d6111df0b714002e6e03c29d` |
|
||||
| split sizes (train/val/test) | 569,909 / 81,414 / 162,831 |
|
||||
| preprocessing hash | `8580cee41c698ce43cf3269e4a2b3a01c92d6df22820f4004c0267478bd2579a` |
|
||||
| `manifest_reload_ok` | true for every beta run |
|
||||
|
||||
Pairing evidence: the **round-0 prototype snapshot is byte-identical**
|
||||
(SHA-256 prefix `e27f41b3a62eb83c`) for all five beta runs **and** the frozen
|
||||
`original_protean` run, so every candidate shares the same partition, split,
|
||||
model initialisation and round-0 local training. `train_support_Nij.csv`,
|
||||
`characterization.csv` and `rare_unseen.csv` are byte-identical across all runs.
|
||||
|
||||
### A.1 Validation-evaluation clarification (new this session)
|
||||
|
||||
The frozen runner previously wrote only test artifacts; `history.json` held a
|
||||
global (union-rare) validation trajectory. To evaluate the protocol's
|
||||
validation-only selection, `scripts/run_baseline.py` now additionally writes, at
|
||||
the end of the fixed 10 rounds:
|
||||
|
||||
- `val_global_metrics.json`, `val_confusion_matrix.json`, `val_per_class.csv`
|
||||
(global validation metrics over the union of per-client rare classes);
|
||||
- `val_per_client.csv` (per-client validation metrics, using the same
|
||||
deterministic Dirichlet eval-partition procedure already used for test,
|
||||
applied to the validation split with `split_name="validation"`);
|
||||
- `val_selection.json` (the selection summary).
|
||||
|
||||
The test path is untouched: re-running a smoke condition before/after the change
|
||||
produced byte-identical test artifacts (`history.json`, `test_global_metrics.json`,
|
||||
`test_confusion_matrix.json`, `test_per_class.csv`, `per_client.csv`,
|
||||
`train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv`,
|
||||
`metadata.json`). The candidate's validation metric is taken from the **final
|
||||
trained model after the fixed 10 rounds** (no best-round cherry-picking).
|
||||
|
||||
---
|
||||
|
||||
## B. Candidate results
|
||||
|
||||
`val rareMacroF1 (pc mean)` is the primary selection metric:
|
||||
per-client rare-class macro-F1 (each client scored on its validation eval
|
||||
partition using that client's rare class ids), averaged over clients with rare
|
||||
classes. The global column is the union-rare validation metric also recorded in
|
||||
`history.json`.
|
||||
|
||||
### B.1 USED FOR SELECTION — validation only
|
||||
|
||||
| beta | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall (global) | val macroF1 (global) | val accuracy | val weighted-F1 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | 0.557340 | 0.611641 | 0.884184 | 0.708929 | 0.891038 | 0.917741 |
|
||||
| 0.25 | 0.571787 | 0.629475 | 0.888488 | 0.723431 | 0.902953 | 0.925933 |
|
||||
| **0.50** | **0.585762** | **0.641709** | 0.887173 | **0.731166** | 0.897831 | **0.926005** |
|
||||
| 0.75 | 0.578003 | 0.637107 | 0.873910 | 0.726603 | 0.887894 | 0.921696 |
|
||||
| 1.00 | 0.551443 | 0.615307 | 0.847043 | 0.715951 | 0.883263 | 0.921182 |
|
||||
|
||||
Validation rare support: 12,726. Validation evaluated rare classes: `[0,1,2,3,7,8]`.
|
||||
Ten of ten clients had validation eval data and rare classes.
|
||||
|
||||
### B.2 NOT USED FOR SELECTION — test (produced automatically by the frozen pipeline)
|
||||
|
||||
| beta | test accuracy | test macro-F1 | test weighted-F1 | test rareMacroF1 | test rare recall | wall-clock |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | 0.891974 | 0.710481 | 0.918369 | 0.613575 | 0.873508 | 41:17 |
|
||||
| 0.25 | 0.903520 | 0.725299 | 0.926452 | 0.632154 | 0.877350 | 42:11 |
|
||||
| 0.50 | 0.898158 | 0.730743 | 0.926333 | 0.640829 | 0.871583 | 42:57 |
|
||||
| 0.75 | 0.888670 | 0.726298 | 0.922278 | 0.636189 | 0.857543 | 43:45 |
|
||||
| 1.00 | 0.884635 | 0.716383 | 0.921910 | 0.615509 | 0.832407 | 42:08 |
|
||||
|
||||
Test rare support: 25,454. The test columns are reported for transparency only and
|
||||
played no part in selecting beta.
|
||||
|
||||
---
|
||||
|
||||
## C. Beta selection
|
||||
|
||||
- **Selected beta: `0.50`** — best among the tested candidates under this
|
||||
validation condition.
|
||||
- **Criterion:** maximum validation per-client-mean rare-class macro-F1
|
||||
(`0.585762`). The secondary global union-rare validation metric independently
|
||||
also peaks at `beta = 0.50` (`0.641709`), so the two readings agree here.
|
||||
- **Tie-break (not used):** higher validation rare-class macro-F1, then higher
|
||||
validation rare-class recall, then smaller beta.
|
||||
- **Test was not used for selection.** `val_selection.json` records
|
||||
`"test_used_for_selection": false` for every run. Test metrics were not
|
||||
inspected until after `beta = 0.50` was fixed from validation.
|
||||
|
||||
---
|
||||
|
||||
## D. Final test evaluation (selected beta vs baseline)
|
||||
|
||||
Selected `beta = 0.50` (test) versus the frozen `original_protean` reference for
|
||||
the identical condition:
|
||||
|
||||
| metric | original_protean | support_weighted beta=0.50 | difference |
|
||||
|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.609620 | 0.640829 | +0.031209 |
|
||||
| rare-class recall | 0.871487 | 0.871583 | +0.000096 |
|
||||
| overall macro-F1 | 0.707701 | 0.730743 | +0.023042 |
|
||||
| accuracy | 0.890432 | 0.898158 | +0.007726 |
|
||||
| weighted-F1 | 0.917585 | 0.926333 | +0.008748 |
|
||||
|
||||
Per-class test results, selected beta (support in parentheses):
|
||||
|
||||
| class | precision | recall | F1 | support |
|
||||
|---|---|---|---|---|
|
||||
| C&C | 0.098 | 0.960 | 0.178 | 573 |
|
||||
| Exfiltration | 1.000 | 0.944 | 0.971 | 4,427 |
|
||||
| Exploitation | 0.017 | 0.581 | 0.033 | 227 |
|
||||
| Lateral_movement | 0.965 | 0.781 | 0.864 | 6,319 |
|
||||
| Normal | 0.971 | 0.935 | 0.953 | 83,651 |
|
||||
| RDOS | 0.992 | 0.997 | 0.995 | 28,252 |
|
||||
| Reconnaissance | 0.993 | 0.648 | 0.784 | 25,474 |
|
||||
| Tampering | 0.732 | 0.989 | 0.841 | 1,024 |
|
||||
| Weaponization | 0.941 | 0.973 | 0.957 | 12,884 |
|
||||
|
||||
Per-client test results, selected beta:
|
||||
|
||||
| client | n | accuracy | macro-F1 | rare macro-F1 | rare recall |
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | 22,614 | 0.8996 | 0.6158 | 0.5123 | 0.9886 |
|
||||
| 1 | 33,759 | 0.9265 | 0.6662 | 0.3998 | 0.6941 |
|
||||
| 2 | 5,625 | 0.9195 | 0.7840 | 0.9038 | 0.9760 |
|
||||
| 3 | 13,539 | 0.9846 | 0.8284 | 0.5481 | 0.5758 |
|
||||
| 4 | 11,881 | 0.7090 | 0.6334 | 0.1411 | 0.7973 |
|
||||
| 5 | 12,785 | 0.9200 | 0.7898 | 0.8331 | 0.9920 |
|
||||
| 6 | 6,131 | 0.9335 | 0.7243 | 0.4422 | 1.0000 |
|
||||
| 7 | 34,434 | 0.9145 | 0.6202 | 0.0477 | 0.8492 |
|
||||
| 8 | 5,090 | 0.9589 | 0.8038 | 0.6343 | 0.9532 |
|
||||
| 9 | 16,973 | 0.8158 | 0.5706 | 0.4051 | 0.6541 |
|
||||
|
||||
**No superiority is claimed.** These are single-dataset, single-alpha,
|
||||
single-seed results from one protocol-validation run.
|
||||
|
||||
---
|
||||
|
||||
## E. Direct prototype analysis (no retraining)
|
||||
|
||||
Computed from the fixed round-0 snapshot
|
||||
(`artifacts/analysis/xiiotid__alpha0.5__seed0__support_weighted__beta0__prototype_snapshot.json`,
|
||||
byte-identical across all beta runs) using the production aggregation; see
|
||||
`...__prototype_effect.csv` and `...__prototype_weights.csv` under
|
||||
`artifacts/analysis/`.
|
||||
|
||||
Per-class `||Cbar_j(beta) - Cbar_j(0)||` at `beta = 1.0` (relative displacement
|
||||
in parentheses), sorted by displacement:
|
||||
|
||||
| class | contributors | support min–max (cv) | disp@beta=1 | rel disp |
|
||||
|---|---|---|---|---|
|
||||
| Weaponization | 10 | 18–32,409 (2.10) | 3.559 | 0.577 |
|
||||
| C&C | 10 | 1–966 (1.50) | 2.156 | 0.370 |
|
||||
| Lateral_movement | 10 | 49–8,385 (1.22) | 2.069 | 0.371 |
|
||||
| Exploitation | 8 | 10–353 (1.09) | 1.934 | 0.342 |
|
||||
| Tampering | 9 | 2–1,266 (1.00) | 1.737 | 0.341 |
|
||||
| Exfiltration | 9 | 45–6,305 (1.04) | 1.547 | 0.214 |
|
||||
| RDOS | 10 | 168–49,283 (1.38) | 1.286 | 0.148 |
|
||||
| Reconnaissance | 10 | 1,137–29,234 (1.16) | 1.184 | 0.194 |
|
||||
| Normal | 10 | 604–68,657 (0.76) | 0.990 | 0.164 |
|
||||
|
||||
- **Largest direct displacement:** `Weaponization` (also the most support-imbalanced
|
||||
class, cv 2.10, max/min 1,800). For that class at `beta = 1.0` the single
|
||||
largest-support client (N=32,409) holds weight `0.719` versus `0.100` at
|
||||
`beta = 0`, while a client with N=18 drops to `0.0004`.
|
||||
- Classes with more balanced support (`Normal`, cv 0.76; `RDOS`, cv 1.38 with one
|
||||
dominant client) move less.
|
||||
- **No class has zero direct effect:** every class has `|S_j| ≥ 8` and unequal
|
||||
supports, so all nine prototypes shift for `beta > 0`. At `beta = 1.0`, no
|
||||
direct-effect displacement exceeded `0.58` of the class prototype norm.
|
||||
- **Displacement is a direct aggregation measure only** and is not interpreted as
|
||||
improved or degraded detection. The end-to-end effect (§D) is reported
|
||||
separately.
|
||||
|
||||
---
|
||||
|
||||
## F. Client-level analysis
|
||||
|
||||
Per-client test rarity metrics relative to `beta = 0`
|
||||
(`artifacts/analysis/xiiotid__alpha0.5__seed0__paired_client_beta_metrics.csv`
|
||||
contains the full paired table including validation columns and `N_ij`).
|
||||
|
||||
Delta test rare-class F1 vs `beta = 0` (negative = worse):
|
||||
|
||||
| client | min positive N_ij | beta=0 rare F1 | Δβ=0.25 | Δβ=0.50 | Δβ=0.75 | Δβ=1.00 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0 | 18 | 0.4996 | +0.0115 | +0.0127 | −0.0010 | −0.0177 |
|
||||
| 1 | 69 | 0.3844 | +0.0124 | +0.0154 | +0.0107 | +0.0061 |
|
||||
| 2 | 5 | 0.8823 | +0.0109 | +0.0215 | +0.0126 | −0.0364 |
|
||||
| 3 | 140 | 0.6162 | −0.0066 | −0.0681 | −0.2566 | −0.3422 |
|
||||
| 4 | 1 | 0.1375 | −0.0005 | +0.0036 | +0.0126 | −0.0143 |
|
||||
| 5 | 2 | 0.7126 | +0.0500 | +0.1204 | +0.1213 | +0.0622 |
|
||||
| 6 | 44 | 0.3788 | +0.0304 | +0.0634 | +0.0690 | +0.0325 |
|
||||
| 7 | 10 | 0.0470 | +0.0109 | +0.0006 | −0.0073 | −0.0140 |
|
||||
| 8 | 4 | 0.6316 | +0.0298 | +0.0027 | −0.0108 | −0.0274 |
|
||||
| 9 | 12 | 0.3910 | +0.0069 | +0.0141 | +0.0093 | +0.0027 |
|
||||
|
||||
Observations (descriptive, **not causal**):
|
||||
|
||||
- At `beta = 0.50`, six of ten clients improve and the only materially negative
|
||||
client is client 3 (−0.068); the per-client mean therefore peaks at
|
||||
`beta = 0.50` (§B.1).
|
||||
- At `beta = 0.75`/`1.00`, several clients lose ground, and the largest single
|
||||
loss is client 3 (−0.257 / −0.342). Client 3 has `min positive N_ij = 140`,
|
||||
so it is **not** the lowest-support client; conversely client 5
|
||||
(`min positive N_ij = 2`) is the largest gainer at `beta ≥ 0.5`.
|
||||
- The relationship between `min positive N_ij` and ΔF1 is **not monotonic** in
|
||||
this single condition. The data are recorded to enable the later, multi-seed
|
||||
analysis of whether high beta disproportionately harms low-support clients;
|
||||
no such claim is made here.
|
||||
|
||||
---
|
||||
|
||||
## G. Runtime and resources
|
||||
|
||||
Measured with `/usr/bin/time -v`, CPU only, no GPU, no swap.
|
||||
|
||||
| beta | wall-clock | CPU % | peak RSS |
|
||||
|---|---|---|---|
|
||||
| 0.00 | 41:17 | 740 % | 6.21 GiB |
|
||||
| 0.25 | 42:11 | 738 % | 6.21 GiB |
|
||||
| 0.50 | 42:57 | 737 % | 6.21 GiB |
|
||||
| 0.75 | 43:45 | 733 % | 6.21 GiB |
|
||||
| 1.00 | 42:08 | 737 % | 6.21 GiB |
|
||||
|
||||
Total: ~3 h 30 min for the five paired candidates, plus ~41 min for the initial
|
||||
beta=0 measurement run (same configuration; not counted twice in the table
|
||||
above beyond beta=0.00). Preprocessing/load is negligible next to training
|
||||
(Session 3 measured ~0.9 s read and ~1.3 s prepare). No warnings, no client
|
||||
failures, 9/9 global prototypes in every round, exit status 0.
|
||||
|
||||
---
|
||||
|
||||
## H. Limitations
|
||||
|
||||
- **Single dataset** (X-IIoTID), **single alpha** (0.50), **single seed** (0).
|
||||
- The `S_j` participant/denominator rule
|
||||
(`S_j = {i | N_ij > 0 and valid C_ij}`, equal weights in the baseline) is a
|
||||
**REPRODUCTION ASSUMPTION**, not verified original PROTEAN behaviour. It is
|
||||
material in 53/162 class-condition cells (32.7 %), 15/18 conditions
|
||||
(`docs/baseline/BASELINE_VALIDATION.md` §D). `beta = 0` equality with the
|
||||
baseline is **conditional** on both modes using this shared rule; the
|
||||
`beta = 0` vs `original_protean` comparison below does **not** prove the rule
|
||||
matches original PROTEAN.
|
||||
- **No claim of a global beta optimum.** `beta = 0.50` is only the best among the
|
||||
five tested candidates under this one validation procedure.
|
||||
- **No full experiment grid has been run** (no other alpha, seed, dataset).
|
||||
- The selected-beta test numbers are single-condition and may not generalise.
|
||||
- The validation per-client evaluation partition used for selection is a
|
||||
Session-5 clarification (mirrors the existing test per-client Dirichlet
|
||||
procedure); it is a project decision, not source-confirmed.
|
||||
- `beta = 0` under `support_weighted` is mathematically the equal mean but is
|
||||
**not bitwise identical** to `original_protean` (see §I); the difference is
|
||||
floating-point ordering, not a methodological change.
|
||||
|
||||
---
|
||||
|
||||
## I. `beta = 0` equivalence check (vs frozen `original_protean`)
|
||||
|
||||
| Property | Result |
|
||||
|---|---|
|
||||
| Same frozen manifest | **Yes** — identical manifest content/partition/split hashes |
|
||||
| Same initialisation / round-0 setup | **Yes** — round-0 prototype snapshot byte-identical (`e27f41b3a62eb83c`) |
|
||||
| Same training behaviour | **Mathematically yes, not bitwise.** `original_protean` computes `Σ(C_ij/count)`; `support_weighted beta=0` computes `Σ((1/count)·C_ij)`. Per-element difference is at most `2.2e-16`–`4.4e-16` (~1 ULP, relative ~1e-16) on the round-0 snapshot. |
|
||||
| Same validation metrics | **No** — tiny divergence (e.g. global rare macro-F1 0.611641 vs 0.608087) |
|
||||
| Same test metrics | **No** — accuracy 0.891974 vs 0.890432; rare macro-F1 0.613575 vs 0.609620 |
|
||||
| Same deterministic artifacts | Round-0 snapshot, `train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv` identical; `history.json` and per-client/per-class metrics differ |
|
||||
|
||||
Mechanism: round 0 is byte-identical, then the ~1 ULP aggregation difference
|
||||
feeds back into round-1 prototype-alignment training and the trajectories
|
||||
diverge slowly (differences grow from ~1e-4 at round 1 to ~1e-2 at round 9). This
|
||||
is floating-point arithmetic ordering (`vec/count` vs `vec*(1/count)`), amplified
|
||||
by iterative training, **not** a semantic difference in the aggregation formula.
|
||||
No code was changed to force bitwise equality; the frozen baseline arithmetic was
|
||||
left untouched.
|
||||
|
||||
**Caveat:** this equivalence is conditional on the shared `S_j` reproduction
|
||||
assumption and does not establish that `beta = 0` reproduces original PROTEAN.
|
||||
|
||||
---
|
||||
|
||||
## J. Verdict
|
||||
|
||||
The validation/test separation protocol ran cleanly end to end for this
|
||||
condition: candidates trained on the paired manifest, beta was selected using
|
||||
validation metrics only, and the selected beta was evaluated once on test. The
|
||||
protocol appears ready, with the documented clarifications (validation per-client
|
||||
evaluation partition; float-level `beta = 0` non-identity), to be scaled to
|
||||
multi-seed/multi-alpha experiments. No full grid was launched.
|
||||
Reference in new issue
Block a user