experiment: validate beta selection on one paired full condition
This commit is contained in:
willtanoe committed
1 parent
ad9aa53ba7
commit
50f2da7216
8 files changed
+12434
-150
No files matched your search
+176
-150
@@ -3,11 +3,11 @@
|
||||
This is the canonical continuation point for the next agent session. It is written to be
|
||||
readable without prior chat history, terminal scrollback, or git history.
|
||||
|
||||
Last updated: 2026-09-24 (Session 4, support-weighted beta aggregation implementation).
|
||||
Session 4 type: implementation + validation. The proposed `support_weighted`
|
||||
(`w_ij = N_ij ** beta`) mode was added **without changing the frozen `original_protean`
|
||||
baseline**. Only small smoke tests were run; **no beta sweep and no beta selection were
|
||||
performed**.
|
||||
Last updated: 2026-09-25 (Session 5, one paired full beta condition + validation-only
|
||||
selection). Session 5 type: **protocol-validation experiment**. Exactly one
|
||||
`(dataset, alpha, seed)` condition was run across all five beta candidates; beta was
|
||||
selected on validation only and the selected beta evaluated once on test. **No full grid
|
||||
was launched.**
|
||||
|
||||
---
|
||||
|
||||
@@ -24,204 +24,225 @@ S_j = { i | N_ij > 0 and a valid local prototype C_ij exists }
|
||||
beta in {0.0, 0.25, 0.5, 0.75, 1.0}
|
||||
```
|
||||
|
||||
Both the baseline and the proposed mode are now implemented. The baseline remains the
|
||||
reference; `beta = 0` is arithmetic-identical to it only because both use the same
|
||||
documented `S_j` participant rule.
|
||||
Both the frozen `original_protean` baseline and the proposed `support_weighted` mode are
|
||||
implemented. `beta = 0` is mathematically the equal mean over the shared `S_j` but is not
|
||||
bitwise identical to the baseline (see §7).
|
||||
|
||||
## 2. Authoritative source and immutability
|
||||
|
||||
- `planningawal/PROTEAN_Revisi_Lurus.md` — authoritative research specification.
|
||||
- `planningawal/` is immutable. Re-verified unchanged this session: `git status --
|
||||
planningawal` and `git diff -- planningawal` are both empty.
|
||||
- Raw datasets re-verified unchanged (SHA-256):
|
||||
`~/datasets/X-IIoTID/X-IIoTID-dataset.csv`
|
||||
`7b9290057ee42e784da3c0d84b781815502c9205c74175c96374e71a5ffd98a0`;
|
||||
`~/datasets/5G-NIDD/5G-NIDD.csv`
|
||||
`fa36f80859585f474504ca69eb951a079b35145537976c86b00aae3aab46ee59`.
|
||||
|
||||
## 3. Dataset locations (read-only external source data)
|
||||
## 3. Session 5 objective and outcome
|
||||
|
||||
- `~/datasets/X-IIoTID/X-IIoTID-dataset.csv` — SHA-256
|
||||
`7b9290057ee42e784da3c0d84b781815502c9205c74175c96374e71a5ffd98a0` (re-verified unchanged).
|
||||
- `~/datasets/5G-NIDD/5G-NIDD.csv` — SHA-256
|
||||
`fa36f80859585f474504ca69eb951a079b35145537976c86b00aae3aab46ee59` (re-verified unchanged).
|
||||
Objective: run ONE paired full condition (X-IIoTID, alpha=0.5, seed=0) across
|
||||
`beta ∈ {0, 0.25, 0.5, 0.75, 1}`, reusing the frozen manifest and training settings;
|
||||
select beta on validation only; evaluate the selected beta on test; add the direct
|
||||
prototype-effect and client-level analyses; update docs. **All acceptance items were met.**
|
||||
No commit or push was made.
|
||||
|
||||
## 4. Session 4 objective and outcome
|
||||
## 4. Condition and pairing
|
||||
|
||||
Objective: implement the proposed support-weighted prototype aggregation while preserving
|
||||
the frozen `original_protean` baseline, add the deferred beta-invariant tests, add direct
|
||||
prototype-effect instrumentation, run small smoke tests for beta 0 / 0.5 / 1, and update
|
||||
docs. **All acceptance items were met.** No commit was made.
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| dataset / alpha / seed | `xiiotid` / 0.50 / 0 |
|
||||
| manifest | `artifacts/partitions/xiiotid__alpha0.5__seed0.json` (reused) |
|
||||
| manifest content hash | `e81a6538ecefcda202aa7602310ff8d8c9529031ab9dcbec4ebbabc0e63b636c` |
|
||||
| partition hash | `2465485539a3e923548dcc0c73d8acc5076d456536b78682d512897ea586e779` |
|
||||
| split hash / sizes | `d0acc0c14e3ecbeba02dfa9e6227c983da12dec2d6111df0b714002e6e03c29d` / 569,909 train, 81,414 val, 162,831 test |
|
||||
| preprocessing hash | `8580cee41c698ce43cf3269e4a2b3a01c92d6df22820f4004c0267478bd2579a` |
|
||||
| training | 10 clients, 10 rounds, 3 local epochs, batch 256, lr 1e-3, mu 0.1, lambda 1.0, Adam |
|
||||
| code revision | HEAD `ad9aa53ba76f60671ace991fcee85f003781f196`; only `scripts/run_baseline.py` modified (hash-object `2b469e35ecbbc6ea3dbd4b0632419993e4d5130b`) |
|
||||
|
||||
## 5. Implementation summary
|
||||
Pairing proof: the round-0 prototype snapshot is **byte-identical** (SHA-256 prefix
|
||||
`e27f41b3a62eb83c`) across all five beta runs and the frozen `original_protean` run;
|
||||
`train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv` are identical across
|
||||
runs. All runs are in `results/xiiotid__alpha0.5__seed0__support_weighted__beta{0,0.25,0.5,0.75,1}/`.
|
||||
|
||||
Added a new aggregation mode and threaded it through the stack; the baseline code path is
|
||||
untouched.
|
||||
## 5. Validation results and beta selection
|
||||
|
||||
- `src/protean_research/aggregation/prototype_aggregation.py`: `AggregationMode.SUPPORT_
|
||||
WEIGHTED`, `BETA_CANDIDATES = (0.0, 0.25, 0.50, 0.75, 1.00)`, strict `validate_beta`
|
||||
(`[0, 1]`; rejects `None`/non-numeric/non-finite/out-of-range), `normalized_support_
|
||||
weights`; `aggregate_prototypes(..., supports=None, beta=None)` and `aggregate_class_
|
||||
prototypes(..., beta=None)`. `S_j` is built once and is identical to the baseline:
|
||||
`None`/`N_ij <= 0` entries dropped, no zero-vector padding. Passing `beta` to
|
||||
`original_protean` raises.
|
||||
- `src/protean_research/aggregation/__init__.py`: exports updated.
|
||||
- `src/protean_research/models/prototypes.py`, `federated/server.py`: forward `beta`.
|
||||
- `src/protean_research/federated/round.py`: `TrainingConfig.aggregation_mode` (default
|
||||
`original_protean`) and `.beta` (default `None`); the round loop dispatches to the mode.
|
||||
- `scripts/run_baseline.py`: new `--aggregation-mode {original_protean,support_weighted}`
|
||||
and `--beta`. Default behavior is the unchanged baseline. Run name/metadata record mode
|
||||
and beta (`...__original_protean` unchanged; `...__support_weighted__beta0.5` for beta).
|
||||
- `src/protean_research/analysis/direct_prototype.py` (new) +
|
||||
`scripts/analyze_prototype_effect.py` (new): recompute `Cbar_j(beta)` from a fixed round
|
||||
snapshot of `{C_ij, N_ij}` **without retraining**; emit `||Cbar_j(beta) - Cbar_j(0)||`,
|
||||
support stats, norms, and normalised client weights.
|
||||
Primary selection metric: **validation per-client-mean rare-class macro-F1** (per-client
|
||||
rare F1 on each client's validation eval partition, averaged over clients with rare
|
||||
classes). Global union-rare validation metrics also shown.
|
||||
|
||||
`equal_available` was **not** added (the baseline already is equal-over-`S_j`, so it equals
|
||||
`beta = 0`).
|
||||
| beta | val rareMacroF1 (pc mean) | val rareMacroF1 (global) | val rareRecall | val macroF1 | val acc | val wF1 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | 0.557340 | 0.611641 | 0.884184 | 0.708929 | 0.891038 | 0.917741 |
|
||||
| 0.25 | 0.571787 | 0.629475 | 0.888488 | 0.723431 | 0.902953 | 0.925933 |
|
||||
| **0.50** | **0.585762** | **0.641709** | 0.887173 | **0.731166** | 0.897831 | **0.926005** |
|
||||
| 0.75 | 0.578003 | 0.637107 | 0.873910 | 0.726603 | 0.887894 | 0.921696 |
|
||||
| 1.00 | 0.551443 | 0.615307 | 0.847043 | 0.715951 | 0.883263 | 0.921182 |
|
||||
|
||||
## 6. Tests and baseline regression status
|
||||
**Selected beta = 0.50** (best among tested candidates under this validation condition;
|
||||
no tie; both validation readings agree). Test was **not** used for selection.
|
||||
|
||||
```
|
||||
.venv/bin/python -m pytest -> 75 passed
|
||||
```
|
||||
## 6. Final test evaluation of the selected beta
|
||||
|
||||
Session-3 suite was 46 tests. Added `tests/test_support_weighted_aggregation.py` (21) and
|
||||
`tests/test_prototype_effect.py` (7); one baseline-aggregation test was replaced by two
|
||||
explicit tests (support_weighted now available; `beta` rejected for `original_protean`).
|
||||
| metric | original_protean | support_weighted beta=0.50 |
|
||||
|---|---|---|
|
||||
| rare-class macro-F1 | 0.609620 | 0.640829 |
|
||||
| rare-class recall | 0.871487 | 0.871583 |
|
||||
| overall macro-F1 | 0.707701 | 0.730743 |
|
||||
| accuracy | 0.890432 | 0.898158 |
|
||||
| weighted-F1 | 0.917585 | 0.926333 |
|
||||
|
||||
Covered beta invariants: `beta=0` arithmetic mean; `beta=1` support-weighted mean;
|
||||
hand-computed intermediate-beta fixture; single-contributor invariance; equal-support
|
||||
invariance; missing-class exclusion; normalised weights sum to 1; invalid/non-numeric
|
||||
beta rejection; supports strictly positive and aligned. **Baseline regression is green**:
|
||||
`test_end_to_end_baseline_is_deterministic_and_learns_separable_fixture` and all baseline
|
||||
arithmetic/validation tests pass unchanged.
|
||||
All five candidates' test metrics (transparency only) are in
|
||||
`docs/research/PAIRED_BETA_VALIDATION.md` §B.2. No superiority is claimed: single
|
||||
dataset / alpha / seed.
|
||||
|
||||
## 7. Smoke results (NOT scientific results)
|
||||
## 7. `beta = 0` equivalence result (important)
|
||||
|
||||
Condition: `xiiotid`, `alpha=0.5`, `seed=0`, `--max-rows 3000 --sample-from 60000`,
|
||||
4 clients, 2 rounds, 1 local epoch (ignored dirs only).
|
||||
`support_weighted beta=0` vs frozen `original_protean`:
|
||||
|
||||
| mode / beta | test accuracy | test macro-F1 | test rare macro-F1 | test rare recall |
|
||||
|---|---|---|---|---|
|
||||
| `original_protean` | 0.5167 | 0.2998 | 0.3140 | 0.4572 |
|
||||
| `support_weighted` 0.0 | 0.5167 | 0.2998 | 0.3140 | 0.4572 |
|
||||
| `support_weighted` 0.5 | 0.2333 | 0.1397 | 0.1797 | 0.3193 |
|
||||
| `support_weighted` 1.0 | 0.0383 | 0.0252 | 0.0324 | 0.1995 |
|
||||
- Same frozen manifest and split: **yes**.
|
||||
- Same initialisation/round-0 setup: **yes** (round-0 snapshot byte-identical).
|
||||
- Training behaviour: **mathematically identical, not bitwise**. `original_protean`
|
||||
computes `Σ(C_ij/count)`; `support_weighted beta=0` computes `Σ((1/count)·C_ij)`.
|
||||
Per-element difference ≤ `2.2e-16`–`4.4e-16` (~1 ULP, relative ~1e-16).
|
||||
- Validation/test metrics: **not identical** (baseline slightly lower; e.g. test
|
||||
accuracy 0.890432 vs 0.891974). The ~1 ULP aggregation difference feeds back into
|
||||
round-1 prototype-alignment training and the trajectories diverge slowly from round 1.
|
||||
- Deterministic artifacts identical: round-0 snapshot, `train_support_Nij.csv`,
|
||||
`characterization.csv`, `rare_unseen.csv`; `history.json`/per-client/per-class differ.
|
||||
|
||||
Baseline-preservation evidence: the `original_protean` and `support_weighted beta=0` smoke
|
||||
runs produced **byte-identical** result artifacts (only `metadata.json` differed in
|
||||
mode/beta/run_name). The round-0 snapshots for `beta = 0, 0.5, 1` were **byte-identical**,
|
||||
confirming the paired comparison (same partition/split/init/round-0 training). The
|
||||
beta=0.5/1 metrics are toy-scale and must not be interpreted as method quality.
|
||||
This is floating-point arithmetic ordering, not a semantic change; no code was altered to
|
||||
force bitwise equality. **Conditional on the shared `S_j` reproduction assumption** and
|
||||
does not prove `beta = 0` reproduces original PROTEAN.
|
||||
|
||||
Direct-effect analysis was exercised on the `beta=0` smoke snapshot (9 classes,
|
||||
max displacement 7.65 at beta=1; relative displacement up to 1.51 for the most
|
||||
support-imbalanced class), written under the ignored smoke directory
|
||||
`artifacts/processed/smoke/analysis/`.
|
||||
## 8. Direct prototype-effect findings (fixed round-0 snapshot, no retraining)
|
||||
|
||||
## 8. Methodological decisions / assumptions carried forward
|
||||
At `beta = 1.0`, largest `||Cbar_j(1)-Cbar_j(0)||`:
|
||||
`Weaponization` 3.559 (rel 0.577; most support-imbalanced, cv 2.10, max/min 1,800),
|
||||
`C&C` 2.156, `Lateral_movement` 2.069, `Exploitation` 1.934, `Tampering` 1.737,
|
||||
`Exfiltration` 1.547; smallest: `Normal` 0.990 (cv 0.76), `RDOS` 1.286,
|
||||
`Reconnaissance` 1.184. Every class has `|S_j| ≥ 8` and unequal support, so no class has
|
||||
zero direct effect. Outputs:
|
||||
`artifacts/analysis/xiiotid__alpha0.5__seed0__support_weighted__beta0__prototype_snapshot__prototype_effect.csv`
|
||||
and `...__prototype_weights.csv`. Displacement is a direct-aggregation measure only, not
|
||||
a detection result.
|
||||
|
||||
## 9. Client-level findings (test, vs beta=0)
|
||||
|
||||
Full paired table: `artifacts/analysis/xiiotid__alpha0.5__seed0__paired_client_beta_metrics.csv`.
|
||||
At `beta=0.50` six of ten clients improve; only client 3 is materially negative (−0.068).
|
||||
At `beta=0.75/1.00` several clients lose, with client 3 worst (−0.257/−0.342). The
|
||||
`min positive N_ij` vs ΔF1 relationship is **not monotonic** in this single condition:
|
||||
client 3 (`min N_ij=140`) is the worst loser while client 5 (`min N_ij=2`) is the biggest
|
||||
gainer. Descriptive only; no causal claim.
|
||||
|
||||
## 10. Runtime / resources
|
||||
|
||||
CPU only (no GPU). Per run: 41–44 min wall-clock, ~733–740 % CPU, peak RSS ~6.21 GiB,
|
||||
no swap, exit 0. Total ~3.5 h for the five paired candidates. Preprocessing/load
|
||||
negligible.
|
||||
|
||||
## 11. Methodological decisions / assumptions
|
||||
|
||||
1. **`S_j` participant/denominator rule** (`S_j = {i | N_ij > 0 and valid C_ij}`, equal
|
||||
weights in the baseline) is a **REPRODUCTION ASSUMPTION**, not verified original
|
||||
behaviour. It is material in 53/162 class-condition cells (32.7%), 15/18 conditions
|
||||
(`docs/baseline/BASELINE_VALIDATION.md` §D). `beta = 0` equality with the baseline is
|
||||
conditional on this shared rule. Carry this caveat in all reporting.
|
||||
2. `N_ij` is **class-specific sample support**, not prototype quality.
|
||||
3. `beta` is one value for all classes/clients; candidates are frozen at
|
||||
`{0, 0.25, 0.5, 0.75, 1}`.
|
||||
4. `lambda = 1.0`, optimiser (Adam), lr (1e-3), batch (256) remain PROJECT DECISIONS.
|
||||
behaviour; material in 53/162 class-condition cells (32.7 %), 15/18 conditions
|
||||
(`docs/baseline/BASELINE_VALIDATION.md` §D). Carry this caveat in all reporting.
|
||||
2. `N_ij` is class-specific sample support, not prototype quality.
|
||||
3. Beta is one value for all classes/clients; candidates frozen at `{0,.25,.5,.75,1}`.
|
||||
4. `lambda=1.0`, Adam, lr 1e-3, batch 256 remain PROJECT DECISIONS.
|
||||
5. **New this session (protocol clarification, documented in
|
||||
`EXPERIMENT_PROTOCOL.md` §8):** candidate validation metric = final-model validation
|
||||
metrics; primary selection = per-client-mean rare-class macro-F1 using a deterministic
|
||||
Dirichlet validation eval partition (mirrors the existing test per-client procedure);
|
||||
tie-break = higher val rare macro-F1 → higher val rare recall → smaller beta.
|
||||
6. No `equal_available` mode was added.
|
||||
|
||||
## 9. Unresolved questions / blockers
|
||||
## 12. Unresolved questions / blockers
|
||||
|
||||
1. `S_j` reproduction ambiguity (above) — unresolved, no official PROTEAN code exists.
|
||||
2. Beta selection has **not** been run; no validation/test separation result exists yet.
|
||||
3. Smoke-scale metrics are unstable and not meaningful; a full paired condition is needed.
|
||||
4. Session-2 manifests store `preprocessing_hash = null` / `config_hash = null`; live
|
||||
preprocessing hash is recorded in run metadata only.
|
||||
1. `S_j` reproduction ambiguity — unresolved; no official PROTEAN code exists.
|
||||
2. `beta = 0` is not bitwise identical to `original_protean` (float ordering). Whether the
|
||||
thesis should report this explicitly, or whether an equivalence check should use a
|
||||
tolerance, is a documentation decision for later.
|
||||
3. Single condition only; no multi-seed/multi-alpha evidence yet.
|
||||
4. Session-2 manifests still store `preprocessing_hash = null` / `config_hash = null`.
|
||||
|
||||
## 10. Git status
|
||||
## 13. Git status
|
||||
|
||||
No commit or push was made. Working tree has 9 modified tracked files and 5 untracked
|
||||
paths from this session (listed by `git status`). `planningawal/` and both raw datasets
|
||||
are unchanged. `results/`, `artifacts/processed/` remain ignored. `artifacts/analysis/` is
|
||||
tracked (no new files were written there this session; smoke analysis went to
|
||||
`artifacts/processed/smoke/analysis/`).
|
||||
No commit or push was made. `git status --short`:
|
||||
|
||||
## 11. Files created / modified this session
|
||||
```
|
||||
M scripts/run_baseline.py
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__seed0__paired_client_beta_metrics.csv
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__seed0__support_weighted__beta0__prototype_snapshot.json
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__seed0__support_weighted__beta0__prototype_snapshot__prototype_effect.csv
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__seed0__support_weighted__beta0__prototype_snapshot__prototype_weights.csv
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__seed0__support_weighted__beta0.25__prototype_snapshot.json
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__seed0__support_weighted__beta0.5__prototype_snapshot.json
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__seed0__support_weighted__beta0.75__prototype_snapshot.json
|
||||
?? artifacts/analysis/xiiotid__alpha0.5__seed0__support_weighted__beta1__prototype_snapshot.json
|
||||
```
|
||||
|
||||
`planningawal/` and both raw datasets are unchanged; `results/` and
|
||||
`artifacts/processed/` are ignored; `artifacts/analysis/` is tracked.
|
||||
|
||||
## 14. Files created / modified this session
|
||||
|
||||
Created:
|
||||
|
||||
- `src/protean_research/analysis/__init__.py`
|
||||
- `src/protean_research/analysis/direct_prototype.py`
|
||||
- `scripts/analyze_prototype_effect.py`
|
||||
- `tests/test_support_weighted_aggregation.py`
|
||||
- `tests/test_prototype_effect.py`
|
||||
- `docs/research/BETA_AGGREGATION_IMPLEMENTATION.md`
|
||||
- `docs/research/PAIRED_BETA_VALIDATION.md`
|
||||
- `artifacts/analysis/xiiotid__alpha0.5__seed0__paired_client_beta_metrics.csv`
|
||||
- five `artifacts/analysis/...__support_weighted__beta*__prototype_snapshot.json`
|
||||
- two `artifacts/analysis/...__prototype_snapshot__prototype_effect.csv` / `__prototype_weights.csv`
|
||||
|
||||
Modified:
|
||||
|
||||
- `src/protean_research/aggregation/prototype_aggregation.py`
|
||||
- `src/protean_research/aggregation/__init__.py`
|
||||
- `src/protean_research/models/prototypes.py`
|
||||
- `src/protean_research/federated/server.py`
|
||||
- `src/protean_research/federated/round.py`
|
||||
- `scripts/run_baseline.py`
|
||||
- `tests/test_baseline_aggregation.py`
|
||||
- `configs/experiments/experiment_protocol.yaml`
|
||||
- `docs/research/EXPERIMENT_PROTOCOL.md`
|
||||
- `scripts/run_baseline.py` (added validation-only evaluation artifacts; test/training paths untouched)
|
||||
- `docs/research/EXPERIMENT_PROTOCOL.md` (§8 Session-5 clarification)
|
||||
- `docs/handoff/SESSION_HANDOFF.md` (this file)
|
||||
|
||||
## 12. Exact next task
|
||||
## 15. Exact recommended next phase
|
||||
|
||||
**Recommended: run ONE paired full `(dataset, alpha, seed)` condition across
|
||||
`beta in {0, 0.25, 0.5, 0.75, 1}` under the validation-based selection protocol, then
|
||||
inspect.** Do not launch the full grid.
|
||||
**Recommendation: B — checkpoint Session 5, then proceed to a controlled
|
||||
multi-seed/multi-alpha experiment.** The protocol ran cleanly (validation/test separation
|
||||
held, artifacts reproducible, pairing proven by byte-identical round-0 snapshots, no test
|
||||
leakage into selection), so the next step is to scale to seeds `{0,1,2}` at a small alpha
|
||||
set, still paired. Do **not** run the entire `dataset × alpha × seed × beta` grid at once;
|
||||
grow it in controlled, inspectable increments.
|
||||
|
||||
Concretely:
|
||||
Before scaling: decide how to report the float-level `beta = 0` vs `original_protean`
|
||||
non-identity (§7), and whether to re-run `original_protean` per condition with the new
|
||||
validation artifacts (the existing baseline run predates the validation-artifact change).
|
||||
|
||||
1. Reuse the frozen manifest for one condition (e.g. X-IIoTID `alpha=0.5`, seed 0) and the
|
||||
frozen training settings (10 clients, 10 rounds, 3 local epochs, batch 256, lr 1e-3,
|
||||
mu 0.1, lambda 1.0). Measure the single-run cost first (Session 3 measured ~42 min /
|
||||
6.2 GiB on CPU for one baseline run; beta adds one run per candidate).
|
||||
2. Train each beta with `run_baseline.py --aggregation-mode support_weighted --beta ...`
|
||||
and record validation rare-class macro-F1; select beta on validation only.
|
||||
3. Evaluate the selected beta on test, and report all candidates transparently.
|
||||
4. Run `scripts/analyze_prototype_effect.py` on the round-0 snapshot for the direct
|
||||
aggregation effect, clearly separated from end-to-end results.
|
||||
5. Report the `S_j` reproduction caveat alongside every between-beta claim.
|
||||
|
||||
Do **not** add `equal_available`. Do **not** run the alpha x seed x dataset x beta grid.
|
||||
|
||||
## 13. Important commands
|
||||
## 16. Important commands
|
||||
|
||||
```bash
|
||||
# Tests (75 expected)
|
||||
.venv/bin/python -m pytest
|
||||
|
||||
# Smoke: baseline and beta candidates (ignored output dirs)
|
||||
# One full paired beta run (CPU, ~42 min, ~6.2 GiB)
|
||||
.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.5 --seed 0 \
|
||||
--max-rows 3000 --sample-from 60000 --clients 4 --rounds 2 --local-epochs 1
|
||||
.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.5 --seed 0 \
|
||||
--max-rows 3000 --sample-from 60000 --clients 4 --rounds 2 --local-epochs 1 \
|
||||
--clients 10 --rounds 10 --local-epochs 3 \
|
||||
--aggregation-mode support_weighted --beta 0.5
|
||||
|
||||
# Direct prototype-effect analysis (no retraining)
|
||||
.venv/bin/python scripts/analyze_prototype_effect.py \
|
||||
--snapshot artifacts/processed/smoke/analysis/\
|
||||
xiiotid__alpha0.5__seed0__support_weighted__beta0__prototype_snapshot.json \
|
||||
--output-dir artifacts/processed/smoke/analysis
|
||||
|
||||
# Full-data baseline for one condition (measure first; ~40-65 min, CPU)
|
||||
.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.5 --seed 0 \
|
||||
--clients 10 --rounds 10 --local-epochs 3
|
||||
--snapshot artifacts/analysis/xiiotid__alpha0.5__seed0__support_weighted__beta0__prototype_snapshot.json \
|
||||
--class-names "C&C,Exfiltration,Exploitation,Lateral_movement,Normal,RDOS,Reconnaissance,Tampering,Weaponization" \
|
||||
--output-dir artifacts/analysis
|
||||
```
|
||||
|
||||
## 14. Recommended next-session prompt
|
||||
## 17. Recommended next-session prompt
|
||||
|
||||
```
|
||||
You are continuing the perdalam-protean research repository (Session 5).
|
||||
You are continuing the perdalam-protean research repository (Session 6).
|
||||
|
||||
FIRST, read completely:
|
||||
1. docs/handoff/SESSION_HANDOFF.md
|
||||
2. docs/research/BETA_AGGREGATION_IMPLEMENTATION.md
|
||||
3. docs/baseline/BASELINE_VALIDATION.md
|
||||
4. docs/research/EXPERIMENT_PROTOCOL.md
|
||||
2. docs/research/PAIRED_BETA_VALIDATION.md
|
||||
3. docs/research/EXPERIMENT_PROTOCOL.md
|
||||
4. docs/research/BETA_AGGREGATION_IMPLEMENTATION.md
|
||||
5. docs/baseline/BASELINE_VALIDATION.md
|
||||
|
||||
Critical rules:
|
||||
- planningawal/ is immutable. Do not edit/create/move/rename/delete anything inside it.
|
||||
@@ -231,14 +252,19 @@ Critical rules:
|
||||
class-condition cells). Carry this caveat in all reporting.
|
||||
- Do NOT add an executable equal_available mode.
|
||||
- Do NOT launch the full alpha x seed x dataset x beta grid.
|
||||
- Beta selection uses validation metrics only; test never influences selection.
|
||||
|
||||
Task: the support_weighted mode (w_ij = N_ij^beta) is implemented and unit/smoke
|
||||
validated; original_protean is unchanged. Run ONE paired full (dataset, alpha, seed)
|
||||
condition across beta in {0, 0.25, 0.5, 0.75, 1}, reusing the frozen manifest and frozen
|
||||
training settings. Select beta on validation rare-class macro-F1 only, then evaluate the
|
||||
selected beta on test and report all candidates. Add the direct prototype-effect analysis
|
||||
from the round-0 snapshot. Measure runtime before scaling. Update docs and
|
||||
docs/handoff/SESSION_HANDOFF.md.
|
||||
Context: Session 5 ran one paired full condition (xiiotid, alpha=0.5, seed=0) across
|
||||
beta in {0,0.25,0.5,0.75,1}. Selected beta = 0.50 on validation per-client-mean
|
||||
rare-class macro-F1; test rare-class macro-F1 0.640829 vs original_protean 0.609620.
|
||||
beta=0 is mathematically but not bitwise equal to original_protean (~1 ULP aggregation
|
||||
ordering). scripts/run_baseline.py now writes validation-only artifacts.
|
||||
|
||||
Task: scale the paired protocol to a controlled multi-seed/multi-alpha set (e.g. seeds
|
||||
{0,1,2} at alpha 0.50, reusing frozen manifests; grow only as compute allows), keeping
|
||||
validation-only beta selection and reporting all candidates. Decide/document how to
|
||||
report the beta=0 vs original_protean float non-identity. Update docs and
|
||||
docs/handoff/SESSION_HANDOFF.md. Measure runtime before scaling further.
|
||||
|
||||
Use .venv/bin/python for all Python commands. Do not commit or push unless asked.
|
||||
```
|
||||
Reference in new issue
Block a user