feat: implement reproducible PROTEAN baseline pipeline

docs/research/EXPERIMENT_PROTOCOL.md

@@ -1,10 +1,15 @@
 # Experiment Protocol
 
-Status: frozen design (to be executed in later sessions)
-Machine-readable counterpart: `configs/experiments/experiment_protocol.yaml`
+Status: **FROZEN** reproduction protocol (2026-09-24).
+Machine-readable counterpart: `configs/experiments/experiment_protocol.yaml`.
 
-Values marked **PROVISIONAL** require confirmation before any scientific sweep. Values
-marked **FROZEN** must remain identical across every aggregation variant.
+Values marked **FROZEN** must remain identical across every aggregation variant, beta
+candidate and seed. Values marked **PROJECT DECISION** are choices this project had to make
+because the paper/thesis leave them unspecified; they are reproduction deviations and must
+be reported as such. No value is PROVISIONAL any more.
+
+See `docs/baseline/PROTEAN_BASELINE_INVESTIGATION.md` for the evidence behind each choice and
+`docs/baseline/BASELINE_IMPLEMENTATION.md` for how the protocol is implemented.
 
 ---
 
@@ -24,59 +29,79 @@ Details and mismatches: `docs/research/DATASET_INSPECTION.md`.
 
 FROZEN: symmetric Dirichlet over classes, `num_clients = 10`, `alpha ∈ {0.75, 0.50, 0.25}`.
 
-- Partitions are generated from the **training split only**.
-- Class proportions are drawn per client from `Dirichlet(alpha)` and used to allocate
-  training samples per class to clients.
-- The exact Dirichlet implementation (per-class client allocation vs per-client class
-  sampling) must be fixed and documented, because it changes realised `N_ij`. `PROVISIONAL`.
-- Success criterion for reproducing the paper's setting: at low alpha, several clients miss
-  several classes, consistent with the paper's reported counts (e.g. X-IIoTID `alpha=0.25`:
-  8/10 clients miss 1–5 classes).
+FROZEN algorithm (**PROJECT DECISION**, paper does not specify the allocation procedure):
+
+1. Partitions are generated from the **training split only**.
+2. For each class `c`, draw client proportions `p ~ Dirichlet(alpha * 1_K_clients)`.
+3. Allocate that class's training samples to clients with
+   `counts ~ Multinomial(N_c, p)`, so the counts sum exactly to `N_c`.
+4. Split the class's shuffled sample positions into consecutive blocks of size `counts`.
+   The shuffle is a permutation drawn from the same deterministic RNG.
+5. A client may receive zero samples of a class; that class is then *missing* on the client
+   (`N_ij = 0`), which is the intended source of non-IIDness.
+
+Determinism: the RNG seed is derived as
+`rng_seed = sha256(f"{dataset}|dirichlet|alpha={alpha}|seed={seed}")[:16] mod 2**31`.
+The manifest stores `rng_seed`, so the same partition is reproduced exactly on reload.
+
+Success criterion for reproducing the paper's setting: at low alpha several clients miss
+several classes.
 
 ## 3. Splits
 
-- Test: 20% of each dataset (paper uses 80/20). FROZEN.
-- Validation: a separate split used only for beta selection. The paper has **no** validation
-  set, so this is an explicit project addition. `PROVISIONAL` fraction and procedure.
-- Requirement: train/validation/test come from a single deterministic split with a recorded
-  seed, and the same split is reused for every beta variant.
+FROZEN: a single deterministic **stratified** split per `(dataset, seed)`.
+
+- `test = 0.20` of the cleaned dataset (paper uses 80/20). FROZEN.
+- `validation = 0.10`; `train = 0.70`. **PROJECT DECISION**: the paper has no validation set;
+  this project adds one for beta selection. The test fraction keeps the paper's 20%.
+- Stratification is per class: each class's samples are shuffled with a deterministic RNG and
+  split 70/10/20. This guarantees every class is represented in validation and test, which is
+  required for rare-class evaluation.
+- Split seed: `sha256(f"{dataset}|split|seed={seed}")[:16] mod 2**31`.
+- The exact same split is reused for every aggregation variant and beta candidate.
 
 ## 4. Seeds
 
-- The paper averages over three runs with unnamed seeds. `PROVISIONAL` set: `{0, 1, 2}`.
-- Once frozen, the seed set is identical across all aggregation variants.
-- Each seed deterministically controls: split, Dirichlet partition, model initialisation,
-  batch order/shuffling.
-- A beta run MUST NOT derive a new partition; it loads the manifest for
-  `(dataset, alpha, seed)`.
+FROZEN (`PROJECT DECISION`): `{0, 1, 2}` (the paper averages over three unnamed seeds).
 
-## 5. Beta candidates and modes
+Each seed deterministically controls: split, Dirichlet partition, model initialisation, batch
+order/shuffling. All RNG use is isolated via `protean_research.utils.seeding.derive_seed`.
+A beta run MUST NOT derive a new partition; it loads the manifest for `(dataset, alpha, seed)`.
 
-FROZEN: `beta ∈ {0.00, 0.25, 0.50, 0.75, 1.00}`, one beta for all classes and clients.
+## 5. Aggregation modes
 
-Modes:
+**Baseline checkpoint: only `original_protean` is implemented.** The proposed
+support-weighted aggregation (`support_weighted`) and the `equal_available` control are
+**DEFERRED** to the later beta/controls phase and are not present in executable code.
 
-- `original_protean` — verified baseline semantics. Not yet encodable (see baseline
-  investigation). This is the baseline to reproduce.
-- `support_weighted` — `w_ij = N_ij^beta`, normalised over `S_j` (proposed mechanism).
-- `equal_available` — equal weights over `S_j`; a methodological control introduced only if
-  the baseline participant rule differs from `S_j`. Not to be presented as a novelty.
+- `original_protean` — reproduced baseline. **REPRODUCTION ASSUMPTION (project decision)**:
+  because Eq. (2)/(5.5) is internally inconsistent about missing classes, the baseline
+  participant rule is assumed to be `S_j = { i | N_ij > 0 and a valid C_ij exists }` with
+  **equal weights `1/|S_j|`**, following the thesis statement that prototypes are shared
+  only for locally observed classes. This is not verified against code and must be
+  validated in the baseline-verification phase.
+- `support_weighted` (deferred) — `w_ij = N_ij^beta`, normalised over `S_j`.
+- `equal_available` (deferred) — equal weights over `S_j`; only introduced later if the
+  baseline participant rule is shown to differ from `S_j`.
 
-`S_j = { i | N_ij > 0 and a valid local prototype C_ij exists }`.
+FROZEN candidates for the later phase: `beta ∈ {0.00, 0.25, 0.50, 0.75, 1.00}`, one beta for
+all classes and clients. They are **not** part of the baseline checkpoint.
 
 ## 6. Partition persistence
 
-Each `(dataset, alpha, seed)` produces one persisted manifest containing at least:
+Each `(dataset, alpha, seed)` produces one persisted JSON manifest under
+`artifacts/partitions/` containing:
 
-- dataset name, alpha, seed, num_clients, K;
-- per-client sample indices or a reproducible reference plus a content hash;
-- per-client `N_ij` matrix `[client][class]`;
-- class label mapping;
-- split indices for train/validation/test;
-- preprocessing/config hash and code version.
+- dataset key, alpha, seed, num_clients, num_classes, class names;
+- the Dirichlet `algorithm` id and `rng_seed`;
+- per-client `N_ij` matrix `[client][class]` (training split);
+- split metadata (seed, per-split sizes, content hash) and partition content hash;
+- preprocessing/config/code hashes.
 
-Manifests live under `artifacts/partitions/`. Reloading a manifest must reproduce the same
-partition (tested). Training reads manifests; it does not regenerate them.
+Manifests are small and version-controlled. Full sample-index arrays are **not** stored by
+default (they would be tens of MB per manifest); they are regenerated deterministically from
+the stored recipe and RNG seed and verified against the stored `N_ij` and content hash. This
+is the tested reload path: `load_manifest` -> `reproduce_partition` -> hash match.
 
 ## 7. Rare and unseen classes
 
@@ -96,18 +121,17 @@ Persisted metadata (minimum one row per `(dataset, alpha, seed, client, class)`)
 
 Procedure, strictly ordered:
 
-1. **Candidate evaluation (validation only).** For each beta candidate, train with the
-   paired manifest and compute validation metrics. Primary criterion: rare-class macro-F1,
-   computed per client that has rare classes, then averaged over those clients.
-2. **Beta selection.** Choose the beta with the best mean validation rare-class macro-F1
-   (ties broken by a pre-declared rule, e.g. smaller beta). The result is
-   "best among tested candidates under this procedure", never a global optimum.
+1. **Candidate evaluation (validation only).** For each beta candidate, train with the paired
+   manifest and compute validation metrics. Primary criterion: rare-class macro-F1, computed
+   per client that has rare classes, then averaged over those clients.
+2. **Beta selection.** Choose the beta with the best mean validation rare-class macro-F1 (ties
+   broken by smaller beta). The result is "best among tested candidates under this
+   procedure", never a global optimum.
 3. **Final test evaluation.** Evaluate the selected beta on the test split. Report overall
    macro-F1, rare-class recall, rare-class macro-F1, and per-client results, plus the test
    results of all candidates for transparency.
 
-Test data is never used in steps 1–2. The selection code must not have access to test
-outputs; test evaluation is a separate step keyed on the selected beta.
+Test data is never used in steps 1–2.
 
 ## 9. Required metrics
 
@@ -117,29 +141,28 @@ Per dataset/alpha/seed/beta and per client, record:
 - rare-class recall;
 - overall macro-F1;
 - accuracy (for comparison with the paper);
-- per-class precision/recall/F1 and confusion counts;
+- per-class precision/recall/F1 and support (confusion counts persisted);
 - per-client metrics.
 
-Also record the validation/test rare-class support counts, so rare-class metrics are never
-read without their sample sizes. Results are reported as mean and variation across seeds.
+Also record the validation/test rare-class support counts. Results are reported as mean and
+variation across seeds.
 
 ## 10. Non-IID characterization
 
 For every manifest, record at least: `N_ij`, class total, number of clients containing class
 `j`, `min N_ij`, `max N_ij`, `mean N_ij`, `std N_ij`, coefficient of variation, and
-`max/min` ratio where defined. Alpha alone is insufficient; different seeds realise
-different distributions.
+`max/min` ratio where defined. Alpha alone is insufficient.
 
 ## 11. Direct prototype analysis
 
 Separate the direct mathematical aggregation effect from the end-to-end training effect.
 
-- During a run, persist at least one round's fixed `C_ij` and `N_ij` snapshot.
+- Persist at least one round's fixed `C_ij` and `N_ij` snapshot.
 - Reusing those snapshots, compute `Cbar_j(beta)` for several beta without retraining.
-- Report `|| Cbar_j(beta) - Cbar_j(0) ||`; optionally the normalised aggregation weights,
-  distances to contributing prototypes, pairwise global-prototype displacement, and support
+- Report `|| Cbar_j(beta) - Cbar_j(0) ||`; optionally normalised aggregation weights,
+  distances to contributing prototypes, pairwise global-prototype displacement, support
   imbalance.
-- This analysis complements, and never replaces, end-to-end evaluation.
+- This complements, and never replaces, end-to-end evaluation.
 
 ## 12. Paired comparison guarantee
 
@@ -154,24 +177,66 @@ For each `(dataset, alpha, seed)`, every beta variant and every mode reuses:
 
 A beta run that regenerates a fresh Dirichlet partition is a protocol violation.
 
-## 13. Frozen vs provisional summary
+## 13. Preprocessing (frozen)
 
-| Item | Value | Status |
+**PROJECT DECISION** — the paper specifies no preprocessing. Frozen recipe:
+
+1. Read the raw CSV read-only (`pandas.read_csv`, `low_memory=False`).
+2. Column names are stripped; the empty 5G-NIDD header becomes `Unnamed: 0`.
+3. Exact duplicate rows are dropped (`keep="first"`) before splitting.
+4. Rows whose label is in `drop_labels` are dropped.
+5. Labels are canonicalised: surrounding whitespace removed, runs of whitespace collapsed,
+   and whitespace around `_` removed (fixes `Lateral _movement`).
+6. Feature columns = all columns except the label column, its aliases, and
+   `non_feature_columns`.
+7. `missing_tokens` are replaced with NaN. Categorical missing values become an explicit
+   `__MISSING__` category.
+8. Categorical columns (`categorical_columns`) are one-hot encoded with categories fitted on
+   the **train split only**; unseen categories at transform time become all-zero.
+9. Numeric columns are coerced to numeric, median-imputed and standardized using **train
+   statistics only**.
+10. Output is a deterministic `float32` matrix with a fixed column order.
+
+No statistic from validation or test data ever enters preprocessing.
+
+## 14. Model, local training, inference (frozen)
+
+- Model: 2 Conv1D (64, 128 filters, ReLU, max-pool, dropout 0.2/0.5), flatten, dense 128
+  (embedding), log-softmax head. **PROJECT DECISION**: kernel size 3, padding 1.
+- Local loss: `cross_entropy + lambda * ||phi(x) - Cbar_{y}||^2 + (mu/2) * ||omega_i - omega||^2`,
+  with `mu = 0.1` (paper), `lambda = 1.0` (**PROJECT DECISION**, paper silent). The prototype
+  term is applied per sample only for classes that currently have a global prototype.
+- Optimizer Adam, `lr = 1e-3`, batch size 256, 3 local epochs, 10 rounds
+  (**PROJECT DECISION** where not stated).
+- Local prototypes: mean embedding over the client's full class-`j` training samples,
+  computed at the **end of the local round** (paper Eq. 1). Missing classes produce no
+  prototype.
+- Server: model parameters averaged with equal `1/M` weights (paper Eq. 2/5.5); class
+  prototypes aggregated with the active mode.
+- Round 0: no global prototypes exist; the prototype-alignment term is skipped; the first
+  global prototypes are produced by aggregating round-0 local prototypes.
+- Inference: embedding `z = phi(x)`, prediction `argmin_j ||z - Cbar_j||^2` over classes that
+  have a global prototype.
+
+## 15. Frozen vs decision summary
+
+| Item | Value | Source |
 |---|---|---|
-| Datasets | X-IIoTID, 5G-NIDD | FROZEN |
-| Clients | 10 | FROZEN |
-| Alpha | 0.75, 0.50, 0.25 | FROZEN |
-| Beta candidates | 0, 0.25, 0.50, 0.75, 1.00 | FROZEN |
-| Test fraction | 0.20 | FROZEN |
-| Rounds / local epochs | 10 / 3 | FROZEN |
-| `mu` | 0.1 | FROZEN |
-| `lambda` | unknown | PROVISIONAL |
-| Batch size / learning rate | unknown | PROVISIONAL |
-| Validation fraction / procedure | unknown | PROVISIONAL |
-| Seeds | 0, 1, 2 | PROVISIONAL |
-| Dirichlet allocation algorithm | unknown | PROVISIONAL |
-| Preprocessing recipe | unknown | PROVISIONAL |
-| Missing-class baseline rule | unknown | PROVISIONAL / blocked on baseline |
+| Datasets | X-IIoTID, 5G-NIDD | paper |
+| Clients | 10 | paper |
+| Alpha | 0.75, 0.50, 0.25 | paper |
+| Beta candidates | 0, 0.25, 0.50, 0.75, 1.00 | spec |
+| Test fraction | 0.20 | paper |
+| Rounds / local epochs | 10 / 3 | paper |
+| `mu` | 0.1 | paper |
+| `lambda` | 1.0 | project decision |
+| Batch size / learning rate | 256 / 1e-3 (Adam) | project decision |
+| Validation fraction | 0.10 (stratified) | project decision |
+| Seeds | 0, 1, 2 | project decision |
+| Dirichlet allocation | per-class Dirichlet + multinomial | project decision |
+| Preprocessing recipe | §13 | project decision |
+| Baseline participant rule | `S_j`, equal `1/|S_j|` | project decision |
+| Model aggregation | equal `1/M` | paper Eq. 2/5.5 |
+| Prototype computation | end-of-round mean embedding | paper Eq. 1 |
 
-The provisional items must be resolved and recorded here before the first sweep. Until then
-only smoke tests are permitted.
+Until the baseline is verified, only smoke tests are permitted; no sweep may be launched.

docs/research/IMPLEMENTATION_PLAN.md

@@ -1,6 +1,9 @@
 # Implementation Plan
 
-Status: design document (no full training implemented in this session)
+Status: design document. The data, model, federated and evaluation modules
+described below are now IMPLEMENTED for the baseline (see
+`docs/baseline/BASELINE_IMPLEMENTATION.md`); the analysis/direct-prototype module
+is still future work and beta experiments were not run in this session.
 Scope: architecture for the server-side prototype-aggregation study on top of a reproduced
 PROTEAN baseline.
 
@@ -107,19 +110,18 @@ regenerate a partition.
 - `round.py`: one communication round; records per-round prototype snapshots for direct
   analysis.
 
-### `aggregation/` (exists)
+### `aggregation/` (baseline implemented; proposed method deferred)
 
-Pure, dependency-free functions implementing only the unambiguous modes:
+Pure, dependency-free functions implementing only the reproduced baseline:
 
-- `beta_weights(supports, beta)` → `N_ij ** beta` (unnormalised).
-- `aggregation_weights(prototypes, mode, beta, supports)` → normalised weights summing to 1.
-- `aggregate_prototypes(...)` → `Cbar_j` for one class.
-- `AggregationMode`: `original_protean` (raises `NotImplementedError` until verified),
-  `support_weighted`, `equal_available`.
+- `aggregate_prototypes(prototypes, mode)` → `Cbar_j` for one class (equal mean over `S_j`).
+- `aggregate_class_prototypes(contributions, mode)` → per-class global prototypes; builds
+  `S_j` by dropping clients with `N_ij = 0` or a missing prototype.
+- `AggregationMode`: only `original_protean` in this checkpoint.
 
-Planned addition: a server-facing helper that accepts `{client_id: (C_ij, N_ij)}` for class
-`j`, constructs `S_j` (drop clients with `N_ij = 0` or missing prototype), and delegates to
-`aggregate_prototypes`. This keeps all weighting logic in one tested place.
+The proposed `support_weighted` mode (`N_ij ** beta`) and the `equal_available` control are
+**deferred** to the later beta/controls phase and are not present in executable code. The
+baseline participant rule (`S_j`, equal weights) is a documented reproduction assumption.
 
 ### `evaluation/` (planned)
 
@@ -139,20 +141,23 @@ Planned addition: a server-facing helper that accepts `{client_id: (C_ij, N_ij)}
   `Cbar_j(beta)` for several beta without retraining and report
   `||Cbar_j(beta) - Cbar_j(0)||`, weights, distances, displacement, imbalance.
 
-## 5. Aggregation API concept
+## 5. Aggregation API (baseline)
 
 ```
-aggregate_prototypes(prototypes, mode, beta=None, supports=None) -> list[float]
+aggregate_prototypes(prototypes, mode="original_protean") -> list[float]
+aggregate_class_prototypes(contributions, mode="original_protean") -> dict[int, list[float]]
 ```
 
-Contract:
+Contract (implemented baseline):
 
-- `prototypes[i]` is `C_ij`; `supports[i]` is `N_ij > 0`; order is client order.
-- `support_weighted`: weights `N_ij ** beta`, normalised over `S_j`.
-- `equal_available`: equal weights over `S_j` (control; not for `beta > 0` claims).
-- `original_protean`: only after the baseline missing-class rule is verified and encoded.
-- Validates finiteness, strictly positive supports, equal dimensions; normalised weights sum
-  to 1; never emits NaN/Inf for valid inputs.
+- `prototypes[i]` is `C_ij`; order is client order.
+- `original_protean`: equal mean over the supplied contributor set `S_j`.
+- `aggregate_class_prototypes` builds `S_j` by dropping clients with `N_ij <= 0` or a
+  missing prototype; missing classes are never fabricated.
+- Validates finiteness and equal dimensions; never emits NaN/Inf for valid inputs.
+
+The proposed `support_weighted(beta, supports)` and `equal_available` modes are **planned**
+for the later beta/controls phase and are not implemented in this checkpoint.
 
 ## 6. Experiment orchestration
 
@@ -169,22 +174,24 @@ Contract:
 
 ## 7. Testing strategy
 
-Required tests (from the specification), to be implemented as the code is built:
+Implemented in the baseline checkpoint (see `tests/`):
 
-1. `beta = 0` equals the arithmetic mean over the same `S_j`.
-2. `beta = 1` equals sample-count weighting (equals pooling of included embeddings).
-3. One contributing client gives the same output for all beta.
-4. Equal supports give the same output for all beta.
-5. A missing class does not enter `S_j`.
-6. Normalised weights sum to one.
-7. Valid inputs never produce NaN/Inf.
-8. Partition generation is deterministic for a fixed seed.
-9. Saved partition manifests reload identically.
-10. Baseline behaviour is unchanged by adding the proposed mode.
-11. Validation/test separation is maintained.
+- baseline aggregation is the equal mean over `S_j`;
+- a missing class does not enter `S_j` and is never fabricated;
+- valid inputs never produce NaN/Inf; dimension/finiteness validation;
+- partition generation is deterministic for a fixed seed;
+- saved partition manifests reload identically (hash-verified);
+- rare/unseen semantics; prototype means; metrics on known fixtures;
+- a small end-to-end federated smoke test.
 
-This session adds the dependency-free subset (1–7) in `tests/test_prototype_aggregation.py`.
-Items 8–11 require the future data/federated modules and are deferred.
+Deferred to the beta/controls phase (proposed-method invariants):
+
+- `beta = 0` equals the arithmetic mean over the same `S_j`;
+- `beta = 1` equals sample-count weighting;
+- single-contributor and equal-support invariance across beta;
+- normalised beta weights sum to one;
+- baseline behaviour unchanged by adding the proposed mode;
+- paired beta/validation/test separation.
 
 ## 8. Compute feasibility
 

docs/research/RESEARCH_SPEC_INTERPRETATION.md

@@ -88,9 +88,10 @@ must still be verified.
 the participant rule from the effect of the weights. The spec says such a control is only
 introduced if needed.
 
-`IMPLEMENTATION DECISION` `equal_available` is kept as a dormant, documented control mode.
-It must not be reported as a result unless baseline investigation demonstrates its
-necessity (see `docs/baseline/PROTEAN_BASELINE_INVESTIGATION.md`).
+`IMPLEMENTATION DECISION` The `equal_available` control is **deferred** to the later
+beta/controls phase and is not implemented in the baseline-only checkpoint. It is
+introduced only if baseline investigation demonstrates that the baseline participant rule
+differs from `S_j` (see `docs/baseline/PROTEAN_BASELINE_INVESTIGATION.md`).
 
 ## 5. Rare class and unseen class definitions