feat: implement support-weighted prototype aggregation
This commit is contained in:
willtanoe committed
1 parent
16b3add2cf
commit
ad9aa53ba7
16 files changed
+1220
-234
No files matched your search
+156
-179
@@ -3,11 +3,11 @@
|
||||
This is the canonical continuation point for the next agent session. It is written to be
|
||||
readable without prior chat history, terminal scrollback, or git history.
|
||||
|
||||
Last updated: 2026-09-24 (Session 3, baseline validation and reproduction verification).
|
||||
Session 3 type: validation-only. No beta / support-weighted / `equal_available` code was
|
||||
added. Two measured full-data baseline conditions were run; partition consistency, per-
|
||||
class/per-client behaviour, the `S_j` ambiguity, round-0, the loss, inference,
|
||||
reproducibility and a deterministic end-to-end regression test were all checked.
|
||||
Last updated: 2026-09-24 (Session 4, support-weighted beta aggregation implementation).
|
||||
Session 4 type: implementation + validation. The proposed `support_weighted`
|
||||
(`w_ij = N_ij ** beta`) mode was added **without changing the frozen `original_protean`
|
||||
baseline**. Only small smoke tests were run; **no beta sweep and no beta selection were
|
||||
performed**.
|
||||
|
||||
---
|
||||
|
||||
@@ -24,14 +24,15 @@ S_j = { i | N_ij > 0 and a valid local prototype C_ij exists }
|
||||
beta in {0.0, 0.25, 0.5, 0.75, 1.0}
|
||||
```
|
||||
|
||||
**The repo still contains the baseline only.** The beta-weighted aggregation and the
|
||||
`equal_available` control are still NOT implemented. Session 3 did not add them.
|
||||
Both the baseline and the proposed mode are now implemented. The baseline remains the
|
||||
reference; `beta = 0` is arithmetic-identical to it only because both use the same
|
||||
documented `S_j` participant rule.
|
||||
|
||||
## 2. Authoritative source and immutability
|
||||
|
||||
- `planningawal/PROTEAN_Revisi_Lurus.md` — authoritative research specification.
|
||||
- `planningawal/` is immutable. Re-verified unchanged this session:
|
||||
SHA-256 `9ba1d7beb94354db0c4b588199fb1e2823aec8a23be1625ec09a756507fbcb89`.
|
||||
- `planningawal/` is immutable. Re-verified unchanged this session: `git status --
|
||||
planningawal` and `git diff -- planningawal` are both empty.
|
||||
|
||||
## 3. Dataset locations (read-only external source data)
|
||||
|
||||
@@ -40,228 +41,204 @@ beta in {0.0, 0.25, 0.5, 0.75, 1.0}
|
||||
- `~/datasets/5G-NIDD/5G-NIDD.csv` — SHA-256
|
||||
`fa36f80859585f474504ca69eb951a079b35145537976c86b00aae3aab46ee59` (re-verified unchanged).
|
||||
|
||||
## 4. Session 3 objective and outcome
|
||||
## 4. Session 4 objective and outcome
|
||||
|
||||
Objective: determine whether the frozen `original_protean` baseline is coherent,
|
||||
reproducible and consistent with the best available PROTEAN evidence, and explicitly
|
||||
identify unresolved reproduction assumptions. **All acceptance items were met.**
|
||||
Objective: implement the proposed support-weighted prototype aggregation while preserving
|
||||
the frozen `original_protean` baseline, add the deferred beta-invariant tests, add direct
|
||||
prototype-effect instrumentation, run small smoke tests for beta 0 / 0.5 / 1, and update
|
||||
docs. **All acceptance items were met.** No commit was made.
|
||||
|
||||
## 5. Full-data runs completed
|
||||
## 5. Implementation summary
|
||||
|
||||
Both with the frozen protocol (`original_protean`, 10 clients, 10 rounds, 3 local epochs,
|
||||
batch 256, lr 1e-3, mu 0.1, lambda 1.0), CPU-only (`torch 2.14.0+cpu`, CUDA unavailable).
|
||||
Added a new aggregation mode and threaded it through the stack; the baseline code path is
|
||||
untouched.
|
||||
|
||||
| condition | wall | peak RSS | accuracy | macro-F1 | weighted-F1 | rare macro-F1 | rare recall | exit |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| X-IIoTID α=0.50 seed 0 | 42:07 | 6.21 GiB | 0.8904 | 0.7077 | 0.9176 | 0.6096 | 0.8715 | 0 |
|
||||
| 5G-NIDD α=0.50 seed 0 | 1:04:52 | 10.37 GiB | 0.7344 | 0.8369 | 0.7088 | 0.8720 | 0.9743 | 0 |
|
||||
- `src/protean_research/aggregation/prototype_aggregation.py`: `AggregationMode.SUPPORT_
|
||||
WEIGHTED`, `BETA_CANDIDATES = (0.0, 0.25, 0.50, 0.75, 1.00)`, strict `validate_beta`
|
||||
(`[0, 1]`; rejects `None`/non-numeric/non-finite/out-of-range), `normalized_support_
|
||||
weights`; `aggregate_prototypes(..., supports=None, beta=None)` and `aggregate_class_
|
||||
prototypes(..., beta=None)`. `S_j` is built once and is identical to the baseline:
|
||||
`None`/`N_ij <= 0` entries dropped, no zero-vector padding. Passing `beta` to
|
||||
`original_protean` raises.
|
||||
- `src/protean_research/aggregation/__init__.py`: exports updated.
|
||||
- `src/protean_research/models/prototypes.py`, `federated/server.py`: forward `beta`.
|
||||
- `src/protean_research/federated/round.py`: `TrainingConfig.aggregation_mode` (default
|
||||
`original_protean`) and `.beta` (default `None`); the round loop dispatches to the mode.
|
||||
- `scripts/run_baseline.py`: new `--aggregation-mode {original_protean,support_weighted}`
|
||||
and `--beta`. Default behavior is the unchanged baseline. Run name/metadata record mode
|
||||
and beta (`...__original_protean` unchanged; `...__support_weighted__beta0.5` for beta).
|
||||
- `src/protean_research/analysis/direct_prototype.py` (new) +
|
||||
`scripts/analyze_prototype_effect.py` (new): recompute `Cbar_j(beta)` from a fixed round
|
||||
snapshot of `{C_ij, N_ij}` **without retraining**; emit `||Cbar_j(beta) - Cbar_j(0)||`,
|
||||
support stats, norms, and normalised client weights.
|
||||
|
||||
Both re-used their frozen manifest (`manifest_reload_ok = true`), completed all 10 rounds
|
||||
with all 10 clients active and no failures, and produced 9/9 global prototypes. Artifact
|
||||
dirs: `results/xiiotid__alpha0.5__seed0__original_protean/`,
|
||||
`results/5gnidd__alpha0.5__seed0__original_protean/` (ignored by git). Full detail:
|
||||
`docs/baseline/BASELINE_VALIDATION.md`.
|
||||
`equal_available` was **not** added (the baseline already is equal-over-`S_j`, so it equals
|
||||
`beta = 0`).
|
||||
|
||||
Resource sizing for the future grid: ~42 min and ~6 GiB for X-IIoTID, ~65 min and ~10 GiB
|
||||
for 5G-NIDD, per `(dataset, alpha, seed)` condition. A full grid
|
||||
`2 datasets × 3 alphas × 3 seeds = 18` conditions is therefore on the order of
|
||||
15–20 CPU-hours plus beta multiplicity; measure before launching anything large.
|
||||
## 6. Tests and baseline regression status
|
||||
|
||||
## 6. Tests added / test status
|
||||
```
|
||||
.venv/bin/python -m pytest -> 75 passed
|
||||
```
|
||||
|
||||
- New file `tests/test_baseline_validation.py` (12 tests): round-0 handling (spy confirms
|
||||
round 0 receives no global prototypes and round 1 does), independent replication of the
|
||||
documented loss formula (`atol=1e-5`), prototype-alignment skip for classes without a
|
||||
global prototype, frozen-coefficient guard (`lambda=1.0`, `mu=0.1`, rounds=10, epochs=3,
|
||||
batch=256, lr=1e-3), equal-weight model averaging, non-contiguous class-id inference,
|
||||
brute-force squared-L2 cross-check, empty-prototype errors, local prototype class
|
||||
mapping, and a deterministic end-to-end regression test.
|
||||
- Total suite: `.venv/bin/python -m pytest -q` -> **46 passed** (34 pre-existing + 12 new).
|
||||
- Note: `BASELINE_IMPLEMENTATION.md` previously said "39 tests"; the accurate Session-2
|
||||
count was 34. Corrected this session.
|
||||
Session-3 suite was 46 tests. Added `tests/test_support_weighted_aggregation.py` (21) and
|
||||
`tests/test_prototype_effect.py` (7); one baseline-aggregation test was replaced by two
|
||||
explicit tests (support_weighted now available; `beta` rejected for `original_protean`).
|
||||
|
||||
## 7. `S_j` stress-test findings (central methodological result)
|
||||
Covered beta invariants: `beta=0` arithmetic mean; `beta=1` support-weighted mean;
|
||||
hand-computed intermediate-beta fixture; single-contributor invariance; equal-support
|
||||
invariance; missing-class exclusion; normalised weights sum to 1; invalid/non-numeric
|
||||
beta rejection; supports strictly positive and aligned. **Baseline regression is green**:
|
||||
`test_end_to_end_baseline_is_deterministic_and_learns_separable_fixture` and all baseline
|
||||
arithmetic/validation tests pass unchanged.
|
||||
|
||||
- `|S_j| < M` occurs in **53/162 class-condition cells (32.7 %)** across **15/18**
|
||||
`(dataset, alpha, seed)` conditions (`artifacts/analysis/sj_participant_rule_analysis.csv`).
|
||||
- Affected classes include high-frequency ones and even the benign class in two conditions;
|
||||
the effect is strongest at `alpha = 0.25`.
|
||||
- On the fixed X-IIoTID round-0 snapshot, the hypothetical literal `1/M` reading with
|
||||
missing prototypes replaced by the zero vector is displaced by **~10–20 % of the
|
||||
prototype norm** for the three affected classes (Exfiltration 0.724, Exploitation 1.131,
|
||||
Tampering 0.510). Hypothetical-only, never used for training:
|
||||
`artifacts/analysis/sj_vs_literal_1m_hypothetical_xiiotid__alpha0.5__seed0__original_protean.csv`,
|
||||
reproducible via `scripts/validate_sj_participant_rule.py`.
|
||||
- Conclusion: the ambiguity is **material, not rare**, but **cannot be resolved** without
|
||||
original code (none exists). The current `S_j` equal-weight rule remains a documented
|
||||
REPRODUCTION ASSUMPTION, defensible from the thesis sentence that prototypes are shared
|
||||
only for locally observed classes. It is **not** verified original behaviour. The
|
||||
`equal_available` control is unnecessary: the baseline already *is* equal-over-`S_j`, so
|
||||
it is arithmetically identical to the proposed `beta = 0` rule.
|
||||
## 7. Smoke results (NOT scientific results)
|
||||
|
||||
## 8. Round-0, loss and inference validation
|
||||
Condition: `xiiotid`, `alpha=0.5`, `seed=0`, `--max-rows 3000 --sample-from 60000`,
|
||||
4 clients, 2 rounds, 1 local epoch (ignored dirs only).
|
||||
|
||||
- **Round 0**: no global prototypes exist; the prototype-alignment term is skipped; the
|
||||
first global prototypes come from round-0 aggregation and are used from round 1.
|
||||
Provenance INFERRED / PROJECT DECISION. Covered by a focused test.
|
||||
- **Loss**: `nll_loss(mean) + lambda * mean_matched ||phi(x)-Cbar_y||^2 + (mu/2)*sum||w-w_global||^2`.
|
||||
Prototype term is per-sample and only for classes with a global prototype; missing
|
||||
prototypes add no term and do not enter the normaliser. Coefficients `lambda=1.0`
|
||||
(PROJECT DECISION), `mu=0.1` (PAPER). Independently replicated to `atol=1e-5`.
|
||||
- **Inference**: nearest global prototype by squared L2 (expanded norm), returns true
|
||||
class ids, restricts to classes with a global prototype, raises on empty prototype sets.
|
||||
Cross-checked against brute force.
|
||||
| mode / beta | test accuracy | test macro-F1 | test rare macro-F1 | test rare recall |
|
||||
|---|---|---|---|---|
|
||||
| `original_protean` | 0.5167 | 0.2998 | 0.3140 | 0.4572 |
|
||||
| `support_weighted` 0.0 | 0.5167 | 0.2998 | 0.3140 | 0.4572 |
|
||||
| `support_weighted` 0.5 | 0.2333 | 0.1397 | 0.1797 | 0.3193 |
|
||||
| `support_weighted` 1.0 | 0.0383 | 0.0252 | 0.0324 | 0.1995 |
|
||||
|
||||
## 9. Reproducibility findings
|
||||
Baseline-preservation evidence: the `original_protean` and `support_weighted beta=0` smoke
|
||||
runs produced **byte-identical** result artifacts (only `metadata.json` differed in
|
||||
mode/beta/run_name). The round-0 snapshots for `beta = 0, 0.5, 1` were **byte-identical**,
|
||||
confirming the paired comparison (same partition/split/init/round-0 training). The
|
||||
beta=0.5/1 metrics are toy-scale and must not be interpreted as method quality.
|
||||
|
||||
The same smoke condition was run twice into separate output roots. All result artifacts
|
||||
(`history.json`, `test_global_metrics.json`, `test_confusion_matrix.json`,
|
||||
`test_per_class.csv`, `per_client.csv`, `train_support_Nij.csv`, `characterization.csv`,
|
||||
`rare_unseen.csv`, prototype snapshot) were **byte-identical**; `manifest_content_hash` and
|
||||
`preprocessing_hash` were identical. Only differences were the run-specific manifest
|
||||
`created_utc` timestamp and the explicit `manifest_path`. The deterministic regression test
|
||||
also produced exactly equal parameters/prototypes/history across two in-process runs.
|
||||
No nondeterminism source was observed on this CPU backend; tolerance clause for future
|
||||
accelerated backends is `atol=1e-5 / rtol=1e-4`.
|
||||
Direct-effect analysis was exercised on the `beta=0` smoke snapshot (9 classes,
|
||||
max displacement 7.65 at beta=1; relative displacement up to 1.51 for the most
|
||||
support-imbalanced class), written under the ignored smoke directory
|
||||
`artifacts/processed/smoke/analysis/`.
|
||||
|
||||
## 10. Partition consistency (X-IIoTID α=0.50 seed 0)
|
||||
## 8. Methodological decisions / assumptions carried forward
|
||||
|
||||
All PASS: manifest reproduces (split hash + partition hash + `N_ij`); splits disjoint and
|
||||
cover the cleaned dataset; each train sample assigned exactly once with no cross-client
|
||||
duplication; `N_ij` matches actual assigned samples; client totals sum to `num_train`
|
||||
(569,909) and class totals match train label counts.
|
||||
1. **`S_j` participant/denominator rule** (`S_j = {i | N_ij > 0 and valid C_ij}`, equal
|
||||
weights in the baseline) is a **REPRODUCTION ASSUMPTION**, not verified original
|
||||
behaviour. It is material in 53/162 class-condition cells (32.7%), 15/18 conditions
|
||||
(`docs/baseline/BASELINE_VALIDATION.md` §D). `beta = 0` equality with the baseline is
|
||||
conditional on this shared rule. Carry this caveat in all reporting.
|
||||
2. `N_ij` is **class-specific sample support**, not prototype quality.
|
||||
3. `beta` is one value for all classes/clients; candidates are frozen at
|
||||
`{0, 0.25, 0.5, 0.75, 1}`.
|
||||
4. `lambda = 1.0`, optimiser (Adam), lr (1e-3), batch (256) remain PROJECT DECISIONS.
|
||||
|
||||
## 9. Unresolved questions / blockers
|
||||
|
||||
1. `S_j` reproduction ambiguity (above) — unresolved, no official PROTEAN code exists.
|
||||
2. Beta selection has **not** been run; no validation/test separation result exists yet.
|
||||
3. Smoke-scale metrics are unstable and not meaningful; a full paired condition is needed.
|
||||
4. Session-2 manifests store `preprocessing_hash = null` / `config_hash = null`; live
|
||||
preprocessing hash is recorded in run metadata only.
|
||||
|
||||
## 10. Git status
|
||||
|
||||
No commit or push was made. Working tree has 9 modified tracked files and 5 untracked
|
||||
paths from this session (listed by `git status`). `planningawal/` and both raw datasets
|
||||
are unchanged. `results/`, `artifacts/processed/` remain ignored. `artifacts/analysis/` is
|
||||
tracked (no new files were written there this session; smoke analysis went to
|
||||
`artifacts/processed/smoke/analysis/`).
|
||||
|
||||
## 11. Files created / modified this session
|
||||
|
||||
Created:
|
||||
|
||||
- `docs/baseline/BASELINE_VALIDATION.md` (new authoritative validation record).
|
||||
- `tests/test_baseline_validation.py` (12 tests).
|
||||
- `scripts/validate_sj_participant_rule.py` (read-only analysis, no experiment mode).
|
||||
- `artifacts/analysis/sj_participant_rule_analysis.csv` (all 18 conditions, |S_j| analysis).
|
||||
- `artifacts/analysis/sj_vs_literal_1m_hypothetical_xiiotid__alpha0.5__seed0__original_protean.csv`.
|
||||
- `artifacts/analysis/xiiotid__alpha0.5__seed0__original_protean__prototype_snapshot.json`.
|
||||
- `artifacts/analysis/5gnidd__alpha0.5__seed0__original_protean__prototype_snapshot.json`.
|
||||
- `src/protean_research/analysis/__init__.py`
|
||||
- `src/protean_research/analysis/direct_prototype.py`
|
||||
- `scripts/analyze_prototype_effect.py`
|
||||
- `tests/test_support_weighted_aggregation.py`
|
||||
- `tests/test_prototype_effect.py`
|
||||
- `docs/research/BETA_AGGREGATION_IMPLEMENTATION.md`
|
||||
|
||||
Modified:
|
||||
|
||||
- `docs/baseline/BASELINE_IMPLEMENTATION.md` (validation link, corrected test count,
|
||||
updated remaining-assumptions §14).
|
||||
- `docs/handoff/SESSION_HANDOFF.md` (this file).
|
||||
- `src/protean_research/aggregation/prototype_aggregation.py`
|
||||
- `src/protean_research/aggregation/__init__.py`
|
||||
- `src/protean_research/models/prototypes.py`
|
||||
- `src/protean_research/federated/server.py`
|
||||
- `src/protean_research/federated/round.py`
|
||||
- `scripts/run_baseline.py`
|
||||
- `tests/test_baseline_aggregation.py`
|
||||
- `configs/experiments/experiment_protocol.yaml`
|
||||
- `docs/research/EXPERIMENT_PROTOCOL.md`
|
||||
- `docs/handoff/SESSION_HANDOFF.md` (this file)
|
||||
|
||||
Ignored/generated (not tracked): `results/xiiotid__alpha0.5__seed0__original_protean/`,
|
||||
`results/5gnidd__alpha0.5__seed0__original_protean/`,
|
||||
`artifacts/processed/*` (preprocessor state).
|
||||
## 12. Exact next task
|
||||
|
||||
`planningawal/` and both raw datasets were not modified.
|
||||
**Recommended: run ONE paired full `(dataset, alpha, seed)` condition across
|
||||
`beta in {0, 0.25, 0.5, 0.75, 1}` under the validation-based selection protocol, then
|
||||
inspect.** Do not launch the full grid.
|
||||
|
||||
## 12. Git status
|
||||
Concretely:
|
||||
|
||||
No commit was requested or made. Session 2 baseline commit `0aa8e2b` and the docs commit
|
||||
`374651a` are on `main` at `origin/main`. Working tree has one modified tracked file
|
||||
(`docs/baseline/BASELINE_IMPLEMENTATION.md`) and the new untracked files listed above.
|
||||
`results/`, `artifacts/processed/`, `.venv/`, `__pycache__/` remain ignored. No secrets,
|
||||
no raw datasets. `AGENTS.md` is tracked (commit `374651a`).
|
||||
1. Reuse the frozen manifest for one condition (e.g. X-IIoTID `alpha=0.5`, seed 0) and the
|
||||
frozen training settings (10 clients, 10 rounds, 3 local epochs, batch 256, lr 1e-3,
|
||||
mu 0.1, lambda 1.0). Measure the single-run cost first (Session 3 measured ~42 min /
|
||||
6.2 GiB on CPU for one baseline run; beta adds one run per candidate).
|
||||
2. Train each beta with `run_baseline.py --aggregation-mode support_weighted --beta ...`
|
||||
and record validation rare-class macro-F1; select beta on validation only.
|
||||
3. Evaluate the selected beta on test, and report all candidates transparently.
|
||||
4. Run `scripts/analyze_prototype_effect.py` on the round-0 snapshot for the direct
|
||||
aggregation effect, clearly separated from end-to-end results.
|
||||
5. Report the `S_j` reproduction caveat alongside every between-beta claim.
|
||||
|
||||
## 13. Unresolved assumptions / blockers
|
||||
Do **not** add `equal_available`. Do **not** run the alpha x seed x dataset x beta grid.
|
||||
|
||||
1. **`S_j` participant/denominator rule** is a reproduction assumption, quantified but
|
||||
unresolved (see §7). All future between-method claims must carry this caveat.
|
||||
2. `lambda = 1.0`, optimiser (Adam), lr (1e-3), batch size (256) are PROJECT DECISIONS.
|
||||
3. Prototype-alignment form (per-sample) and round-0 initialisation are INFERRED.
|
||||
4. Model aggregation is equal `1/M` (PAPER); a FedAvg-style weighted wording in the thesis
|
||||
intro conflicts and is not implemented (documented conflict).
|
||||
5. Per-client evaluation partitioning is a PROJECT DECISION.
|
||||
6. Session-2 manifests store `preprocessing_hash = null` / `config_hash = null`; they do
|
||||
not bind the preprocessing recipe. Split/partition hashes and `N_ij` reproduce exactly,
|
||||
and run metadata records the live preprocessing hash. Binding the hash into future
|
||||
manifests would close this provenance gap.
|
||||
7. Only one full-data condition per dataset has been validated; no reproduction success is
|
||||
claimed from two runs.
|
||||
|
||||
## 14. Exact next task
|
||||
|
||||
**Recommended: (B) freeze the validated baseline and prepare the paired beta
|
||||
aggregation implementation — not a full grid.**
|
||||
|
||||
Justification from evidence:
|
||||
|
||||
- Implementation correctness, determinism, partition consistency and the full-data paths
|
||||
are validated on both datasets (Sessions 2–3).
|
||||
- Option (A), "resolve remaining baseline reproduction ambiguity", is **not achievable
|
||||
with available evidence**: no official PROTEAN code exists and the paper is internally
|
||||
inconsistent. The ambiguity has been quantified; further "resolution" would only be
|
||||
another documented decision.
|
||||
- Therefore freeze the baseline definition *including* the documented `S_j` equal-weight
|
||||
assumption and its quantified materiality, then implement the proposed beta weighting
|
||||
as a new aggregation mode **without altering the baseline** and run a **single paired
|
||||
`(dataset, alpha, seed)` condition first** (e.g. X-IIoTID α=0.5 seed 0, reusing the
|
||||
frozen manifest) across `beta in {0, 0.25, 0.5, 0.75, 1}` before any grid.
|
||||
|
||||
Next session must:
|
||||
|
||||
1. Add `support_weighted` (`w_ij = N_ij^beta`) as a new mode only; keep
|
||||
`original_protean` byte-identical and add the deferred beta-invariant tests
|
||||
(`beta=0` == equal mean over `S_j`; `beta=1` == support weighting; single-contributor
|
||||
and equal-support invariance; normalised weights sum to 1).
|
||||
2. Do **not** add an executable `equal_available` mode (baseline already is equal-over-`S_j`).
|
||||
3. Keep validation/test separation: beta selection on validation rare-class macro-F1,
|
||||
test only after selection.
|
||||
4. Run one paired condition end to end and inspect; measure before any grid.
|
||||
5. Report the `S_j` reproduction assumption explicitly alongside any beta result.
|
||||
|
||||
Do **not** launch the full grid until the single paired condition is verified.
|
||||
|
||||
## 15. Important commands
|
||||
## 13. Important commands
|
||||
|
||||
```bash
|
||||
# Tests (46 expected)
|
||||
.venv/bin/python -m pytest -q
|
||||
# Tests (75 expected)
|
||||
.venv/bin/python -m pytest
|
||||
|
||||
# Smoke: baseline and beta candidates (ignored output dirs)
|
||||
.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.5 --seed 0 \
|
||||
--max-rows 3000 --sample-from 60000 --clients 4 --rounds 2 --local-epochs 1
|
||||
.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.5 --seed 0 \
|
||||
--max-rows 3000 --sample-from 60000 --clients 4 --rounds 2 --local-epochs 1 \
|
||||
--aggregation-mode support_weighted --beta 0.5
|
||||
|
||||
# Direct prototype-effect analysis (no retraining)
|
||||
.venv/bin/python scripts/analyze_prototype_effect.py \
|
||||
--snapshot artifacts/processed/smoke/analysis/\
|
||||
xiiotid__alpha0.5__seed0__support_weighted__beta0__prototype_snapshot.json \
|
||||
--output-dir artifacts/processed/smoke/analysis
|
||||
|
||||
# Full-data baseline for one condition (measure first; ~40-65 min, CPU)
|
||||
.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.5 --seed 0 \
|
||||
--clients 10 --rounds 10 --local-epochs 3
|
||||
.venv/bin/python scripts/run_baseline.py --dataset 5gnidd --alpha 0.5 --seed 0 \
|
||||
--clients 10 --rounds 10 --local-epochs 3
|
||||
|
||||
# Smoke (writes only to ignored dirs)
|
||||
.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.5 --seed 0 \
|
||||
--max-rows 3000 --sample-from 60000 --clients 4 --rounds 2 --local-epochs 1
|
||||
|
||||
# Read-only S_j participant-rule validation (writes analysis artifacts only)
|
||||
.venv/bin/python scripts/validate_sj_participant_rule.py \
|
||||
--snapshot artifacts/analysis/xiiotid__alpha0.5__seed0__original_protean__prototype_snapshot.json
|
||||
|
||||
# Regenerate / verify all partition manifests (full data, ~30 s)
|
||||
.venv/bin/python scripts/generate_partitions.py
|
||||
```
|
||||
|
||||
## 16. Recommended next-session prompt
|
||||
## 14. Recommended next-session prompt
|
||||
|
||||
```
|
||||
You are continuing the perdalam-protean research repository (Session 4).
|
||||
You are continuing the perdalam-protean research repository (Session 5).
|
||||
|
||||
FIRST, read completely:
|
||||
1. docs/handoff/SESSION_HANDOFF.md
|
||||
2. docs/baseline/BASELINE_VALIDATION.md
|
||||
3. docs/baseline/BASELINE_IMPLEMENTATION.md
|
||||
2. docs/research/BETA_AGGREGATION_IMPLEMENTATION.md
|
||||
3. docs/baseline/BASELINE_VALIDATION.md
|
||||
4. docs/research/EXPERIMENT_PROTOCOL.md
|
||||
|
||||
Critical rules:
|
||||
- planningawal/ is immutable. Do not edit/create/move/rename/delete anything inside it.
|
||||
- ~/datasets/X-IIoTID/X-IIoTID-dataset.csv and ~/datasets/5G-NIDD/5G-NIDD.csv are
|
||||
read-only external source data. Do not modify or copy them into the repo.
|
||||
- The baseline is validated but the S_j participant rule is a REPRODUCTION ASSUMPTION
|
||||
(material in 32.7% of class-condition cells). Carry this caveat in all reporting.
|
||||
- The S_j participant rule is a REPRODUCTION ASSUMPTION (material in 32.7% of
|
||||
class-condition cells). Carry this caveat in all reporting.
|
||||
- Do NOT add an executable equal_available mode.
|
||||
- Do NOT launch the full alpha x seed x dataset x beta grid.
|
||||
|
||||
Task (Section 14 of the handoff): freeze the validated baseline and add the proposed
|
||||
support_weighted aggregation (w_ij = N_ij^beta) as a NEW mode without changing
|
||||
original_protean. Add the deferred beta-invariant tests. Then run ONE paired
|
||||
(dataset, alpha, seed) condition across beta in {0, 0.25, 0.5, 0.75, 1}, select beta on
|
||||
validation rare-class macro-F1, and evaluate the selected beta on test. Measure runtime
|
||||
before scaling. Update docs and docs/handoff/SESSION_HANDOFF.md.
|
||||
Task: the support_weighted mode (w_ij = N_ij^beta) is implemented and unit/smoke
|
||||
validated; original_protean is unchanged. Run ONE paired full (dataset, alpha, seed)
|
||||
condition across beta in {0, 0.25, 0.5, 0.75, 1}, reusing the frozen manifest and frozen
|
||||
training settings. Select beta on validation rare-class macro-F1 only, then evaluate the
|
||||
selected beta on test and report all candidates. Add the direct prototype-effect analysis
|
||||
from the round-0 snapshot. Measure runtime before scaling. Update docs and
|
||||
docs/handoff/SESSION_HANDOFF.md.
|
||||
|
||||
Use .venv/bin/python for all Python commands. Do not commit or push unless asked.
|
||||
```
|
||||
@@ -0,0 +1,176 @@
|
||||
# Beta-Weighted Prototype Aggregation — Implementation
|
||||
|
||||
Status: **IMPLEMENTED and unit/smoke-validated** (Session 4, 2026-09-24). The
|
||||
proposed class-specific sample-support weighting is now available as an
|
||||
independent aggregation mode. The frozen `original_protean` baseline path is
|
||||
unchanged. **No scientific beta sweep was run**; only small smoke tests.
|
||||
|
||||
Companion documents:
|
||||
|
||||
- `docs/baseline/BASELINE_IMPLEMENTATION.md` — baseline semantics and provenance.
|
||||
- `docs/baseline/BASELINE_VALIDATION.md` — Session-3 baseline validation evidence.
|
||||
- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen protocol.
|
||||
- `docs/handoff/SESSION_HANDOFF.md` — continuation point.
|
||||
|
||||
---
|
||||
|
||||
## 1. Scope
|
||||
|
||||
The only change relative to the validated baseline is the **server-side prototype
|
||||
aggregation**. The data pipeline, partitions, splits, model, local loss, model
|
||||
averaging (`1/M`) and inference are reused unchanged. The proposed rule is:
|
||||
|
||||
```
|
||||
Cbar_j(beta) = ( sum_{i in S_j} N_ij^beta * C_ij ) / ( sum_{i in S_j} N_ij^beta )
|
||||
S_j = { i | N_ij > 0 and a valid local prototype C_ij exists }
|
||||
beta ∈ {0.00, 0.25, 0.50, 0.75, 1.00}, 0 <= beta <= 1
|
||||
```
|
||||
|
||||
`N_ij` is **class-specific sample support**, not a quality score. Larger `N_ij`
|
||||
is not assumed to mean a better prototype (AGENTS.md §9).
|
||||
|
||||
## 2. Code changes
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `src/protean_research/aggregation/prototype_aggregation.py` | Added `AggregationMode.SUPPORT_WEIGHTED`, `validate_beta`, `normalized_support_weights`; extended `aggregate_prototypes` and `aggregate_class_prototypes` with optional `supports`/`beta`. Baseline branch kept arithmetic-identical. |
|
||||
| `src/protean_research/aggregation/__init__.py` | Exports the new mode, helpers and `BETA_CANDIDATES`. |
|
||||
| `src/protean_research/models/prototypes.py` | `collect_global_prototypes` forwards `beta`. |
|
||||
| `src/protean_research/federated/server.py` | `aggregate_prototypes` forwards `mode`/`beta`. |
|
||||
| `src/protean_research/federated/round.py` | `TrainingConfig.aggregation_mode` + `.beta`; the round loop dispatches to the active mode. |
|
||||
| `src/protean_research/analysis/direct_prototype.py` | New direct prototype-effect instrumentation (fixed `{C_ij, N_ij}`). |
|
||||
| `scripts/analyze_prototype_effect.py` | New CLI: snapshot → effect CSV + weights CSV, no retraining. |
|
||||
| `scripts/run_baseline.py` | New `--aggregation-mode`/`--beta`; run name and metadata now record mode + beta. Default remains `original_protean`. |
|
||||
|
||||
### 2.1 Interface
|
||||
|
||||
```python
|
||||
aggregate_prototypes(prototypes, mode="original_protean", supports=None, beta=None)
|
||||
aggregate_class_prototypes(contributions, mode="original_protean", beta=None)
|
||||
normalized_support_weights(supports, beta) # weights summing to 1
|
||||
validate_beta(beta) # strict [0, 1] validation
|
||||
```
|
||||
|
||||
- `original_protean` computes the equal mean over `S_j`. Passing `beta` with it
|
||||
raises `ValueError` (no silent mixing).
|
||||
- `support_weighted` requires strictly positive, aligned `supports` and a
|
||||
`beta ∈ [0, 1]`; `None`, non-numeric, non-finite and out-of-range betas raise
|
||||
`ValueError`.
|
||||
- `S_j` is built once, identically, for both modes: entries with `proto is None`
|
||||
or `N_ij <= 0` are dropped; a class with no valid contributor is omitted. No
|
||||
zero-vector padding or prototype fabrication.
|
||||
|
||||
## 3. Beta semantics and invariants
|
||||
|
||||
| Property | Expected | Test |
|
||||
|---|---|---|
|
||||
| `beta = 0` | equal (arithmetic) mean over `S_j` | `test_beta0_is_equal_arithmetic_mean_over_s_j`, `test_beta0_matches_original_protean_on_identical_s_j` |
|
||||
| `beta = 1` | support-weighted mean | `test_beta1_is_support_weighted_mean` |
|
||||
| intermediate beta | closed-form weighted mean | `test_intermediate_beta_fixture`, `test_intermediate_beta_matches_closed_form` |
|
||||
| single contributor | beta-invariant | `test_single_contributor_is_invariant_to_beta` |
|
||||
| equal support | beta-invariant | `test_equal_support_is_invariant_to_beta` |
|
||||
| missing class | excluded from `S_j`, never fabricated | `test_missing_class_is_excluded_from_s_j`, `test_class_with_only_missing_contributors_is_omitted` |
|
||||
| weights | normalised, sum to 1, positive | `test_normalized_weights_sum_to_one`, `test_beta0_weights_are_uniform` |
|
||||
| invalid beta | rejected | `test_invalid_beta_is_rejected`, `test_none_and_non_numeric_beta_are_rejected` |
|
||||
|
||||
`beta = 0` is **not** labelled original PROTEAN. It is arithmetically identical
|
||||
to the baseline *only because both use the same documented `S_j` participant
|
||||
rule*; the `S_j` rule itself remains a reproduction assumption (§5).
|
||||
|
||||
## 4. Direct prototype-effect instrumentation
|
||||
|
||||
`analysis/direct_prototype.py` reads a round snapshot of fixed `{C_ij, N_ij}`
|
||||
(produced by the federated loop at round 0) and recomputes `Cbar_j(beta)` for the
|
||||
candidate betas **without retraining**. It reuses the production aggregation, so
|
||||
only the training feedback is removed, not the aggregation arithmetic.
|
||||
|
||||
Outputs (via `scripts/analyze_prototype_effect.py`):
|
||||
|
||||
- `*__prototype_effect.csv`: per `(class, beta)` — `n_contributors`,
|
||||
`support_min/max/mean`, `prototype_norm`, `reference_norm_beta0`,
|
||||
`displacement_from_beta0 = ||Cbar_j(beta) - Cbar_j(0)||`,
|
||||
`relative_displacement`.
|
||||
- `*__prototype_weights.csv`: per `(class, beta, client)` normalised weights.
|
||||
|
||||
Displacement is a *direct aggregation* measure only; it is **not** interpreted as
|
||||
improved or degraded detection (AGENTS.md §17).
|
||||
|
||||
## 5. Validation performed
|
||||
|
||||
### 5.1 Test suite
|
||||
|
||||
```
|
||||
.venv/bin/python -m pytest -> 75 passed
|
||||
```
|
||||
|
||||
Session-3 suite was 46 tests. Added: `tests/test_support_weighted_aggregation.py`
|
||||
(21), `tests/test_prototype_effect.py` (7), and one extra baseline-aggregation
|
||||
test (the old "support_weighted not available" guard was replaced by explicit
|
||||
availability/validation guards). The deterministic end-to-end baseline regression
|
||||
(`test_end_to_end_baseline_is_deterministic_and_learns_separable_fixture`) and all
|
||||
baseline arithmetic tests remain green.
|
||||
|
||||
### 5.2 Baseline preservation (integration evidence)
|
||||
|
||||
Smoke condition: `xiiotid`, `alpha=0.5`, `seed=0`, `max-rows=3000`,
|
||||
`sample-from=60000`, 4 clients, 2 rounds, 1 local epoch.
|
||||
|
||||
- `original_protean` and `support_weighted beta=0` produced **byte-identical**
|
||||
result artifacts (`history.json`, `test_global_metrics.json`, confusion matrix,
|
||||
per-class/per-client CSVs, `N_ij`, characterization, rare/unseen, round-0
|
||||
snapshot). Only `metadata.json` differed (`aggregation_mode`, `beta`,
|
||||
`run_name`).
|
||||
- The three round-0 snapshots (`beta = 0, 0.5, 1`) were **byte-identical**,
|
||||
confirming that the paired comparison reuses the same partition, split, model
|
||||
initialisation and round-0 training; beta only changes server aggregation.
|
||||
|
||||
### 5.3 Smoke runs (NOT scientific results)
|
||||
|
||||
| mode / beta | test accuracy | test macro-F1 | test rare macro-F1 | test rare recall |
|
||||
|---|---|---|---|---|
|
||||
| `original_protean` | 0.5167 | 0.2998 | 0.3140 | 0.4572 |
|
||||
| `support_weighted` 0.0 | 0.5167 | 0.2998 | 0.3140 | 0.4572 |
|
||||
| `support_weighted` 0.5 | 0.2333 | 0.1397 | 0.1797 | 0.3193 |
|
||||
| `support_weighted` 1.0 | 0.0383 | 0.0252 | 0.0324 | 0.1995 |
|
||||
|
||||
These are 3,000-row toy smoke runs with a random 4-client partition and a
|
||||
stratified 70/10/20 split; the split's class support at this scale is tiny and the
|
||||
metrics are unstable. They demonstrate that the beta path runs end to end and
|
||||
that beta **does** change training feedback (global prototypes enter the local
|
||||
loss from round 1). They must not be read as evidence about method quality. A
|
||||
full-scale paired validation/test protocol is required (§7).
|
||||
|
||||
Direct-effect instrumentation was exercised on the `beta=0` smoke snapshot
|
||||
(9 classes, max `||Cbar_j(1) - Cbar_j(0)|| = 7.65`, relative displacement up to
|
||||
1.51 for the most support-imbalanced class), written under the ignored smoke
|
||||
directory `artifacts/processed/smoke/analysis/`.
|
||||
|
||||
## 6. Provenance and assumptions
|
||||
|
||||
- `support_weighted` arithmetic (`N_ij^beta`, normalised over `S_j`) — **SPEC /
|
||||
PROJECT IMPLEMENTATION**; the research specification defines the formula.
|
||||
- `S_j = {i | N_ij > 0 and valid C_ij}` for **both** modes — **REPRODUCTION
|
||||
ASSUMPTION**, not verified original PROTEAN behaviour, carried from the
|
||||
baseline. It is **material, not rare**: `|S_j| < M` in **53/162 class-condition
|
||||
cells (32.7%)** across **15/18** `(dataset, alpha, seed)` conditions
|
||||
(`docs/baseline/BASELINE_VALIDATION.md` §D). It is not source-confirmed.
|
||||
- `beta = 0` equality with the baseline is **conditional on that shared `S_j`
|
||||
rule**; it is not an independent claim that `beta = 0` reproduces original
|
||||
PROTEAN.
|
||||
- No `equal_available` executable mode was added (the baseline already *is*
|
||||
equal-over-`S_j`, so it is arithmetically identical to `beta = 0`).
|
||||
- `N_ij` is treated strictly as sample support, not prototype quality.
|
||||
|
||||
## 7. Not done in this session
|
||||
|
||||
- No validation-based beta selection, no test evaluation of a selected beta.
|
||||
- No full or partial `(dataset, alpha, seed, beta)` sweep.
|
||||
- No `equal_available` mode, no client rejection, no new distance/metric.
|
||||
|
||||
## 8. Exact next step
|
||||
|
||||
Run **one** paired full condition (e.g. X-IIoTID `alpha=0.5`, seed 0, reusing the
|
||||
frozen manifest and frozen training settings) across
|
||||
`beta ∈ {0, 0.25, 0.5, 0.75, 1}` under the validation/test separation of
|
||||
`EXPERIMENT_PROTOCOL.md` §8, then inspect per-client/per-class artifacts before
|
||||
scaling. Measure runtime before any grid.
|
||||
@@ -70,9 +70,9 @@ A beta run MUST NOT derive a new partition; it loads the manifest for `(dataset,
|
||||
|
||||
## 5. Aggregation modes
|
||||
|
||||
**Baseline checkpoint: only `original_protean` is implemented.** The proposed
|
||||
support-weighted aggregation (`support_weighted`) and the `equal_available` control are
|
||||
**DEFERRED** to the later beta/controls phase and are not present in executable code.
|
||||
**Both `original_protean` and the proposed `support_weighted` are implemented.** The
|
||||
`equal_available` control is **not** implemented and is unnecessary (the baseline is already
|
||||
equal-over-`S_j`). See `docs/research/BETA_AGGREGATION_IMPLEMENTATION.md`.
|
||||
|
||||
- `original_protean` — reproduced baseline. **REPRODUCTION ASSUMPTION (project decision)**:
|
||||
because Eq. (2)/(5.5) is internally inconsistent about missing classes, the baseline
|
||||
@@ -80,12 +80,14 @@ support-weighted aggregation (`support_weighted`) and the `equal_available` cont
|
||||
**equal weights `1/|S_j|`**, following the thesis statement that prototypes are shared
|
||||
only for locally observed classes. This is not verified against code and must be
|
||||
validated in the baseline-verification phase.
|
||||
- `support_weighted` (deferred) — `w_ij = N_ij^beta`, normalised over `S_j`.
|
||||
- `equal_available` (deferred) — equal weights over `S_j`; only introduced later if the
|
||||
baseline participant rule is shown to differ from `S_j`.
|
||||
- `support_weighted` — `w_ij = N_ij^beta`, normalised over the **same** `S_j` as the
|
||||
baseline. `beta` must lie in `[0, 1]`; invalid or missing beta is rejected explicitly.
|
||||
- `equal_available` — deliberately not implemented; arithmetically identical to
|
||||
`support_weighted` with `beta = 0` given the shared `S_j`.
|
||||
|
||||
FROZEN candidates for the later phase: `beta ∈ {0.00, 0.25, 0.50, 0.75, 1.00}`, one beta for
|
||||
all classes and clients. They are **not** part of the baseline checkpoint.
|
||||
FROZEN candidates for the beta phase: `beta ∈ {0.00, 0.25, 0.50, 0.75, 1.00}`, one beta for
|
||||
all classes and clients. Implementation status and smoke validation:
|
||||
`docs/research/BETA_AGGREGATION_IMPLEMENTATION.md`.
|
||||
|
||||
## 6. Partition persistence
|
||||
|
||||
@@ -157,6 +159,10 @@ For every manifest, record at least: `N_ij`, class total, number of clients cont
|
||||
|
||||
Separate the direct mathematical aggregation effect from the end-to-end training effect.
|
||||
|
||||
**Implemented** in `src/protean_research/analysis/direct_prototype.py` and exposed through
|
||||
`scripts/analyze_prototype_effect.py`; it reuses the production aggregation on a fixed
|
||||
snapshot and requires no retraining.
|
||||
|
||||
- Persist at least one round's fixed `C_ij` and `N_ij` snapshot.
|
||||
- Reusing those snapshots, compute `Cbar_j(beta)` for several beta without retraining.
|
||||
- Report `|| Cbar_j(beta) - Cbar_j(0) ||`; optionally normalised aggregation weights,
|
||||
@@ -239,4 +245,8 @@ No statistic from validation or test data ever enters preprocessing.
|
||||
| Model aggregation | equal `1/M` | paper Eq. 2/5.5 |
|
||||
| Prototype computation | end-of-round mean embedding | paper Eq. 1 |
|
||||
|
||||
Until the baseline is verified, only smoke tests are permitted; no sweep may be launched.
|
||||
The baseline is validated and frozen (`docs/baseline/BASELINE_VALIDATION.md`), and the
|
||||
proposed `support_weighted` mode is implemented (`docs/research/BETA_AGGREGATION_IMPLEMENTATION.md`).
|
||||
Only smoke tests have been run for beta. Before any grid, one paired full
|
||||
`(dataset, alpha, seed)` condition across the beta candidates must be run and inspected
|
||||
(validation-based selection, then test), and runtime measured.
|
||||
Reference in new issue
Block a user