feat: implement reproducible PROTEAN baseline pipeline

docs/baseline/BASELINE_IMPLEMENTATION.md

@@ -0,0 +1,210 @@
+# Baseline Implementation
+
+Status: minimum reproducible baseline implemented and smoke-tested (2026-09-24).
+Scope: the pipeline needed to run original-PROTEAN experiments. The proposed
+beta-weighted aggregation and the `equal_available` control are **NOT implemented**
+in this checkpoint; they are deferred to the later beta/controls phase. No beta
+sweep was launched.
+
+Companion documents:
+
+- `docs/baseline/PROTEAN_BASELINE_INVESTIGATION.md` — evidence for baseline semantics.
+- `docs/research/EXPERIMENT_PROTOCOL.md` — protocol and provenance tags.
+- `docs/research/IMPLEMENTATION_PLAN.md` — architecture.
+- `docs/handoff/SESSION_HANDOFF.md` — continuation point.
+
+## 0. Provenance legend
+
+Every decision below is classified as one of:
+
+- **CODE** — confirmed from original PROTEAN code. **None exist**: no official
+  implementation is public (`PROTEAN_BASELINE_INVESTIGATION.md` §2).
+- **PAPER** — stated in the paper and/or the author's PhD thesis.
+- **INFERRED** — not written verbatim but a defensible reading of the paper/thesis.
+- **PROJECT DECISION** — unspecified by the paper; chosen here and reported as a
+  reproduction deviation.
+- **UNRESOLVED** — still unknown.
+
+## 1. Implemented modules
+
+| Module | Responsibility |
+|---|---|
+| `data/loading.py` | spec resolution, read-only CSV read, label canonicalisation, duplicate drop, feature selection |
+| `data/preprocessing.py` | train-fitted median imputation + standardisation, one-hot encoding, serialisable state |
+| `data/splits.py` | deterministic stratified 70/10/20 split, leakage guard |
+| `data/partition.py` | Dirichlet non-IID partition, `N_ij`, content hash, eval-split partition |
+| `data/manifest.py` | manifest save/load, exact reproduction with hash verification |
+| `data/characterization.py` | per-class and per-client non-IID statistics |
+| `data/rare_unseen.py` | per-client rare/unseen class derivation |
+| `data/pipeline.py` | end-to-end deterministic data wiring |
+| `models/protean_cnn.py` | baseline CNN, parameter get/set/average, embedding extraction |
+| `models/prototypes.py` | local prototypes, global prototype collection, nearest-prototype inference |
+| `aggregation/prototype_aggregation.py` | baseline `original_protean` aggregation only |
+| `federated/client.py` | local update (CE + prototype alignment + model alignment) |
+| `federated/server.py` | model averaging (1/M), prototype aggregation, global inference |
+| `federated/round.py` | round orchestration + per-round validation metrics |
+| `evaluation/metrics.py` | accuracy, macro/weighted F1, per-class P/R/F1, rare/unseen metrics |
+| `utils/seeding.py`, `utils/integrity.py` | deterministic seed derivation, hashing |
+| `scripts/run_baseline.py` | one-condition baseline runner / smoke test |
+| `scripts/generate_partitions.py` | generate + verify all `(dataset, alpha, seed)` manifests |
+
+## 2. Preprocessing decisions (all **PROJECT DECISION**)
+
+The paper specifies no preprocessing. Frozen recipe:
+
+1. Raw CSVs read read-only; paths resolve from config with env overrides.
+2. Column names stripped; the empty 5G-NIDD header becomes `Unnamed: 0`.
+3. Exact duplicate rows dropped (`keep="first"`): X-IIoTID 6,222; 5G-NIDD 1.
+4. `crypto-ransomware` dropped from X-IIoTID (**PAPER**).
+5. Labels canonicalised (whitespace collapse, whitespace around `_` removed), fixing
+   `Lateral _movement` -> `Lateral_movement`.
+6. Feature columns exclude the label column, its aliases and configured non-feature
+   columns.
+7. `categorical_columns` one-hot encoded with a `__MISSING__` category; all other
+   columns coerced to numeric, missing tokens -> NaN, median-imputed.
+8. Imputation medians and standardisation statistics fitted on the **train split only**;
+   fitted state persisted and reused for validation/test.
+9. Deterministic `float32` output matrix.
+
+No statistic from validation or test data enters preprocessing.
+
+## 3. Split decisions
+
+| Decision | Value | Provenance |
+|---|---|---|
+| Test fraction | 0.20 | PAPER (80/20) |
+| Validation fraction | 0.10 | PROJECT DECISION (paper has no validation set) |
+| Train fraction | 0.70 | PROJECT DECISION (remainder) |
+| Stratification | per class | PROJECT DECISION (ensures rare classes in val/test) |
+| Split seed | derived from `(dataset, "split", seed)` | PROJECT DECISION |
+
+## 4. Partition decisions
+
+| Decision | Value | Provenance |
+|---|---|---|
+| Algorithm | per-class Dirichlet + multinomial allocation | PROJECT DECISION (paper unspecified) |
+| Partition source | training split only | PAPER |
+| RNG seed | derived from `(dataset, "dirichlet", alpha, seed)` | PROJECT DECISION |
+| Missing classes | `N_ij = 0`, no prototype | PAPER (only observed classes shared) |
+| Persistence | JSON manifest + content hash, indices regenerated & verified | PROJECT DECISION |
+
+Observed non-IID behavior (seed 0): X-IIoTID `alpha=0.75` -> at most 1 missing class per
+client; `alpha=0.25` -> up to 3. Qualitatively consistent with the paper setting.
+
+## 5. Model provenance
+
+| Element | Value | Provenance |
+|---|---|---|
+| Layer set | 2 Conv1D (64, 128) + ReLU + max-pool + dropout (0.2/0.5) + flatten + dense 128 + log-softmax | PAPER |
+| Conv kernel / padding | 3 / 1 | PROJECT DECISION (paper silent) |
+| Embedding `phi` | dense 128 output | PAPER (paper: "first eight of ten layers") |
+| Prototype source | embedding | PAPER / INFERRED |
+
+## 6. Local training and loss provenance
+
+| Element | Value | Provenance |
+|---|---|---|
+| Objective form | CE + `lambda * prototype alignment` + `(mu/2) * model alignment` | PAPER Eq. (3) |
+| `mu` | 0.1 | PAPER |
+| `lambda` | 1.0 | PROJECT DECISION (paper gives no value) |
+| Prototype alignment form | per-sample `\|\|phi(x) - Cbar_y\|\|^2` for classes with a global prototype | INFERRED (FedProto-style reading; paper writes class-mean to class-mean) |
+| Model alignment target | global model of the round | INFERRED |
+| Optimiser / lr / batch size | Adam / 1e-3 / 256 | PROJECT DECISION |
+| Rounds / local epochs | 10 / 3 | PAPER |
+| Local prototype | end-of-round mean embedding per observed class | PAPER Eq. (1) |
+| Round 0 prototypes | none; prototype term skipped; first prototypes from round-0 aggregation | INFERRED / PROJECT DECISION |
+
+## 7. Aggregation provenance — `original_protean` status
+
+**`original_protean` is implemented, but its participant rule is a REPRODUCTION
+ASSUMPTION, not verified original behavior.**
+
+- **PAPER**: Eq. (2)/(5.5) writes `Cbar_j = (1/M) * sum_{i} C_ij` and model averaging
+  `omega = (1/M) * sum_i omega_i`.
+- **PAPER**: the thesis states prototypes are shared only for locally observed classes.
+- **UNRESOLVED**: the paper never defines how a class missing on a client (`N_ij = 0`) is
+  handled, nor whether the denominator is `M` or the number of contributors.
+- **REPRODUCTION ASSUMPTION (PROJECT DECISION)**: the server receives prototypes only from
+  `S_j = { i | N_ij > 0 and a valid C_ij exists }` and averages them with **equal weights
+  `1/|S_j|`**. This follows the thesis sentence above but is not source-confirmed.
+
+Consequences:
+
+- `original_protean` is therefore a *documented interpretation* of the baseline.
+  It must be validated in the baseline-verification phase before any between-method
+  comparison.
+- The model-aggregation rule (equal `1/M`) is implemented separately and is closer to the
+  paper text.
+
+Whether a literal `1/M` reading with omitted classes coincides with this assumption is an
+open question for the verification phase; the `equal_available` control that could
+distinguish them is deliberately deferred.
+
+## 8. Inference provenance
+
+| Element | Value | Provenance |
+|---|---|---|
+| Rule | nearest global prototype by squared L2 | PAPER (thesis Eqs. 5.11–5.13) |
+| Classes considered | those with a global prototype | INFERRED |
+
+## 9. Seeds
+
+- Exact values: **`{0, 1, 2}`**, frozen in `configs/experiments/experiment_protocol.yaml`.
+- Rationale: the paper averages over three random seeds but names none, so the values are
+  **not source-confirmed**. `{0, 1, 2}` is the minimal three-seed set and is an explicit
+  **PROJECT DECISION**.
+- Each seed deterministically controls the split, Dirichlet partition, model
+  initialisation and batch order, via
+  `protean_research.utils.seeding.derive_seed` (never Python `hash()`).
+- The seed set is identical across all baseline conditions.
+
+## 10. Manifests
+
+There are **18** partition manifests under `artifacts/partitions/`, from
+`2 datasets × 3 alphas × 3 seeds = 18` combinations of `(dataset, alpha, seed)`. Each is
+content-hashed, reload-verified, and intended to be reused by every later aggregation
+variant so that comparisons stay paired. Full sample-index arrays are regenerated
+deterministically from the stored recipe/RNG seed and checked against the stored `N_ij`
+and hash, so manifests stay small.
+
+## 11. Tests
+
+`python -m pytest -q` -> **39 tests** (see run result in the handoff). Coverage includes
+dataset config resolution, deterministic label mapping, deterministic/train-only
+preprocessing, deterministic stratified split, deterministic Dirichlet partition,
+manifest reload identity and corruption detection, rare/unseen semantics, local prototype
+means, baseline aggregation (including that `support_weighted` is not implemented), metrics
+on known fixtures, and a small end-to-end federated smoke test.
+
+## 12. Smoke test
+
+```
+.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.5 --seed 0 \
+    --max-rows 3000 --sample-from 60000 --clients 4 --rounds 2 --local-epochs 1
+```
+
+Runs on both datasets through the full path (load -> clean -> split -> preprocess ->
+partition -> local training -> prototypes -> baseline aggregation -> global
+redistribution -> evaluation -> artifacts). Smoke runs write only to ignored directories
+(`artifacts/processed/smoke/`, `results/smoke/`) and are **not** thesis results. The
+smoke test does not invoke the proposed method.
+
+## 13. Structured outputs
+
+Per run under `results/<run_name>/`: `metadata.json`, `history.json`,
+`test_global_metrics.json`, `test_confusion_matrix.json`, `test_per_class.csv`,
+`per_client.csv`, `train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv`.
+Preprocessor state -> `artifacts/processed/`; round-0 prototype snapshot ->
+`artifacts/processed/smoke/analysis/` (smoke) or `artifacts/analysis/` (full run);
+manifests -> `artifacts/partitions/`.
+
+## 14. Remaining assumptions and unresolved issues
+
+1. **Participant rule** (`S_j`, equal weights) is a reproduction assumption (see §7).
+2. `lambda = 1.0`, optimiser and schedule are unverified against the paper.
+3. Prototype-alignment form and round-0 handling are inferred.
+4. Per-client evaluation partitioning (val/test assigned to clients via Dirichlet) is a
+   project decision; the paper does not describe it.
+5. No full-scale run has been timed; compute feasibility of a real sweep is unmeasured.
+6. Whether the literal `1/M` reading differs from the `S_j` assumption is open and must
+   be tested with a deliberate control in the verification phase.

docs/handoff/SESSION_HANDOFF.md

@@ -1,11 +1,11 @@
 # Session Handoff
 
 This is the canonical continuation point for the next agent session. It is written to be
-readable without any prior chat history, terminal scrollback, or git history.
+readable without prior chat history, terminal scrollback, or git history.
 
-Session date: 2026-09-24
-Session type: research study, baseline investigation, architecture planning, dataset
-inspection, minimal bootstrap. **No training was run and no experimental sweep was launched.**
+Last updated: 2026-09-24 (Session 2, baseline-only checkpoint).
+Session 2 type: minimum reproducible baseline implementation + smoke test, then cleanup so
+the checkpoint contains **baseline only**. **No full-scale training and no beta sweep.**
 
 ---
 
@@ -13,7 +13,7 @@ inspection, minimal bootstrap. **No training was run and no experimental sweep w
 
 Build the code and experiment infrastructure for a thesis study that modifies **only the
 server-side prototype aggregation** of PROTEAN (Chennoufi et al., 2025), a federated
-intrusion-detection framework. The proposed mechanism weights each contributing client's
+intrusion-detection framework. The planned mechanism weights each contributing client's
 class prototype by its class sample support:
 
 ```
@@ -22,325 +22,260 @@ S_j = { i | N_ij > 0 and a valid local prototype C_ij exists }
 beta in {0.0, 0.25, 0.5, 0.75, 1.0}
 ```
 
-Goal: characterise when support weighting helps or hurts rare attack detection under
-non-IID data, selecting beta on a validation set. `beta = 0` is a control, **not** assumed
-identical to the original baseline.
+**This checkpoint contains the baseline only.** The proposed beta-weighted aggregation and
+the `equal_available` control are intentionally NOT implemented yet; they are deferred to
+the later beta/controls phase, which must not start until the baseline is verified.
 
 ## 2. Authoritative source
 
-- `planningawal/PROTEAN_Revisi_Lurus.md` — the authoritative research specification.
-- The entire `planningawal/` directory is **immutable / read-only**. Do not edit, create,
-  rename, move, delete, reformat, overwrite, or generate reports inside it. It must not
-  appear in any "files modified" list.
-- Verified unchanged this session: SHA-256
-  `9ba1d7beb94354db0c4b588199fb1e2823aec8a23be1625ec09a756507fbcb89` (12,050 bytes).
+- `planningawal/PROTEAN_Revisi_Lurus.md` — authoritative research specification.
+- `planningawal/` is **immutable / read-only**. Verified unchanged:
+  SHA-256 `9ba1d7beb94354db0c4b588199fb1e2823aec8a23be1625ec09a756507fbcb89`
+  (only file in `planningawal/`).
 
 ## 3. Dataset locations (read-only source data)
 
-- `~/datasets/X-IIoTID/X-IIoTID-dataset.csv`
-- `~/datasets/5G-NIDD/5G-NIDD.csv`
+- `~/datasets/X-IIoTID/X-IIoTID-dataset.csv` — SHA-256
+  `7b9290057ee42e784da3c0d84b781815502c9205c74175c96374e71a5ffd98a0` (mtime 2021-10-07)
+- `~/datasets/5G-NIDD/5G-NIDD.csv` — SHA-256
+  `fa36f80859585f474504ca69eb951a079b35145537976c86b00aae3aab46ee59` (mtime 2023-02-18)
 
-Do not move, rename, overwrite, modify, or copy the full raw files into the repository. Use
-configurable external paths (`configs/datasets/datasets.yaml`, with env overrides).
+Re-verified unchanged. Paths resolve from `configs/datasets/datasets.yaml` with env
+overrides (`PROTEAN_XIIOTID_RAW_PATH`, `PROTEAN_5GNIDD_RAW_PATH`). Raw files are never
+modified or copied into the repo.
 
-Recorded SHA-256:
+## 4. Current repository state
 
-- X-IIoTID: `7b9290057ee42e784da3c0d84b781815502c9205c74175c96374e71a5ffd98a0`
-- 5G-NIDD: `fa36f80859585f474504ca69eb951a079b35145537976c86b00aae3aab46ee59`
-
-## 4. Current repository tree
+Baseline code:
 
 ```
-perdalam-protean/
-├── README.md
-├── pyproject.toml
-├── .gitignore
-├── planningawal/                         # IMMUTABLE (read-only)
-│   └── PROTEAN_Revisi_Lurus.md
-├── configs/
-│   ├── datasets/datasets.yaml
-│   ├── experiments/experiment_protocol.yaml
-│   └── models/protean_cnn.yaml
-├── docs/
-│   ├── research/
-│   │   ├── RESEARCH_SPEC_INTERPRETATION.md
-│   │   ├── DATASET_INSPECTION.md
-│   │   ├── IMPLEMENTATION_PLAN.md
-│   │   └── EXPERIMENT_PROTOCOL.md
-│   ├── baseline/
-│   │   └── PROTEAN_BASELINE_INVESTIGATION.md
-│   └── handoff/
-│       └── SESSION_HANDOFF.md
-├── src/protean_research/
-│   ├── __init__.py
-│   ├── aggregation/
-│   │   ├── __init__.py
-│   │   └── prototype_aggregation.py
-│   └── utils/
-│       ├── __init__.py
-│       └── config.py
-├── tests/
-│   └── test_prototype_aggregation.py
-├── scripts/                              # .gitkeep only (tracked)
-├── artifacts/
-│   ├── partitions/                       # tracked: deterministic manifests (.gitkeep now)
-│   ├── analysis/                         # tracked: small analysis outputs (.gitkeep now)
-│   ├── processed/                        # ignored (regenerable)
-│   └── checkpoints/                      # ignored (regenerable)
-└── results/                              # ignored (regenerable)
+src/protean_research/
+├── data/            loading, preprocessing, splits, partition, manifest,
+│                    characterization, rare_unseen, pipeline
+├── models/          protean_cnn, prototypes
+├── aggregation/     prototype_aggregation (baseline original_protean ONLY)
+├── federated/       client, server, round
+├── evaluation/      metrics
+└── utils/           config, seeding, integrity
+scripts/             run_baseline.py, generate_partitions.py
+tests/               34 baseline-focused tests
+artifacts/partitions/ 18 full-data manifests (tracked, small, content-hashed)
+artifacts/analysis/   18 characterization + 18 rare_unseen CSVs (tracked, small)
+artifacts/processed/  ignored: preprocessor state + smoke partitions/snapshots
+results/              ignored: smoke run outputs
 ```
 
-`.gitignore` deliberately keeps `artifacts/partitions/` and `artifacts/analysis/`
-version-controlled (partition manifests are deterministic, content-hashed, and required for
-paired comparisons), while ignoring `artifacts/processed/`, `artifacts/checkpoints/`,
-`results/`, `logs/`, and `*.log`. `scripts/` is preserved with a `.gitkeep` placeholder.
-`planningawal/` is never ignored.
+Environment: `.venv/` exists (gitignored) with numpy 2.5.3, pandas 3.0.6,
+torch 2.14.0+cpu, pyyaml 6.0.3, pytest 9.1.1. Use `.venv/bin/python`.
 
-Note: the repository was **not** empty at session start. A partial scaffold already existed
-(`configs/*`, `src/protean_research/aggregation/*`, `src/protean_research/utils/config.py`,
-`__init__.py` files, and empty `docs/`, `scripts/`, `tests/`, `artifacts/`, `results/`
-directories). `docs/` was empty and there was no handoff. That scaffold was kept and verified;
-it is not the product of this session.
+## 5. What was completed this session
 
-## 5. Work completed this session
+1. Froze every previously-PROVISIONAL protocol value in
+   `configs/experiments/experiment_protocol.yaml` and `docs/research/EXPERIMENT_PROTOCOL.md`.
+2. Dataset config: explicit `categorical_columns`, `drop_duplicate_rows`, missing tokens,
+   fixed 5G-NIDD `Unnamed: 0` (quoted).
+3. Data foundation (`data/`): read-only loading, deterministic canonicalisation, leak-free
+   train-fitted preprocessing, deterministic stratified 70/10/20 split, Dirichlet partition
+   with content-hashed manifests and `N_ij`, characterization, rare/unseen metadata.
+4. Model (`models/`): baseline CNN, parameter averaging, embedding extraction, local
+   prototypes, global prototype collection, nearest-prototype inference.
+5. Baseline aggregation (`aggregation/`): `original_protean` only. The proposed
+   `support_weighted` and the `equal_available` control were **removed** so the checkpoint
+   is baseline only; an unavailable-mode guard raises `ValueError`.
+6. Federated loop (`federated/`): local update (CE + prototype alignment + model
+   alignment), server model/prototype aggregation, round orchestration, round-0 snapshot.
+7. Evaluation (`evaluation/`): accuracy, macro/weighted F1, per-class P/R/F1, rare/unseen.
+8. Scripts: `generate_partitions.py` (18 manifests, full data, reload-verified);
+   `run_baseline.py` (one baseline condition; no beta/mode options).
+9. Tests: 34 baseline-only tests, all passing.
+10. Smoke test: baseline-only on both datasets.
+11. Docs: `docs/baseline/BASELINE_IMPLEMENTATION.md` rewritten with provenance tags.
 
-1. Read the full specification and produced a tagged interpretation
-   (`docs/research/RESEARCH_SPEC_INTERPRETATION.md`).
-2. Investigated authoritative PROTEAN sources (paper, ESORICS/Springer, HAL, author PhD
-   thesis) and searched for official code.
-3. Established baseline aggregation semantics question-by-question with evidence tags
-   (`docs/baseline/PROTEAN_BASELINE_INVESTIGATION.md`).
-4. Inspected both raw datasets in full (read-only) and recorded shapes, label distributions,
-   missingness, duplicates, identifiers, and mismatches
-   (`docs/research/DATASET_INSPECTION.md`).
-5. Verified the pre-existing config/path-resolution scaffold and aggregation module by running
-   them.
-6. Wrote the implementation architecture (`docs/research/IMPLEMENTATION_PLAN.md`) and the
-   experiment protocol (`docs/research/EXPERIMENT_PROTOCOL.md`).
-7. Added a minimal bootstrap: `pyproject.toml`, `README.md`, and dependency-free unit tests
-   for the aggregation math (`tests/test_prototype_aggregation.py`, 9 tests, all passing).
-8. Recorded dataset and source hashes for integrity verification.
-9. Wrote this handoff.
+## 6. Baseline decisions and provenance
 
-## 6. Evidence and findings
+Full detail in `docs/baseline/BASELINE_IMPLEMENTATION.md`. Summary:
 
-### 6.1 Official PROTEAN source status
+- **No official PROTEAN code exists**, so no decision is source-confirmed by code.
+- **PAPER**: architecture, `mu = 0.1`, 10 rounds / 3 local epochs, 80/20 test, equal `1/M`
+  model averaging, end-of-round mean-embedding prototypes, nearest-prototype inference.
+- **REPRODUCTION ASSUMPTION (project decision)**: `original_protean` participant rule is
+  `S_j` with equal `1/|S_j|` weights. The paper writes `1/M` but is ambiguous for classes
+  missing on a client; this assumption follows the thesis sentence that prototypes are
+  shared only for locally observed classes. It is **not** treated as verified.
+- **PROJECT DECISION**: `lambda = 1.0`; Adam `lr = 1e-3`; batch size 256; kernel size 3;
+  stratified 70/10/20 split with an added validation set; seeds `{0,1,2}`;
+  preprocessing recipe; per-class Dirichlet allocation; per-client eval partitioning.
+- **INFERRED**: per-sample prototype alignment (FedProto-style reading of Eq. 3); round-0
+  handling (no prototypes, prototype term skipped).
+- **UNRESOLVED**: whether a literal `1/M` missing-class reading differs from `S_j`.
 
-- **No official code exists publicly (as of 2026-09-24).**
-- Author GitHub `sarachn` (Sara Chennoufi) has only `FATE-LLM` (fork) and `SecLLMHolmes` —
-  neither is PROTEAN.
-- GitHub repository search `PROTEAN federated prototype` → 0 results; `grep.app` → none.
-- The unrelated GitHub user `Chennoufi` has only a `septembre` repo — do not chase it.
-- No code/artifact/supplementary link in the arXiv HTML, HAL record, or thesis text.
-- **Consequence: the baseline must be reproduced from the paper/thesis.**
+## 7. Seeds (explicit)
 
-Primary sources:
+- Exact values: **`{0, 1, 2}`**.
+- Rationale: the paper averages over three random seeds but names none, so the values are
+  **not source-confirmed**. `{0, 1, 2}` is the minimal three-seed set and is an explicit
+  **project decision**.
+- Each seed deterministically controls the split, Dirichlet partition, model
+  initialisation and batch order, via `protean_research.utils.seeding.derive_seed`.
+- The set is identical across all baseline conditions.
 
-- arXiv:2507.05524v1 (7 Jul 2025), ESORICS 2025 pp. 103–125,
-  DOI `10.1007/978-3-032-07884-1_6`.
-- HAL `hal-05379203`.
-- Author PhD thesis `tel-05490927` (195 pages, 2025) — restates PROTEAN with extra detail.
+## 8. Manifests (18)
 
-### 6.2 Baseline semantics discovered
+`artifacts/partitions/` holds **18** manifests from `2 datasets × 3 alphas × 3 seeds`.
+They are content-hashed, reload-verified, and reused by later variants for pairing. Sample
+indices are regenerated deterministically from the stored recipe/RNG seed and checked
+against the stored `N_ij` and hash, keeping manifests small.
 
-Confirmed from the paper/thesis:
+## 9. Verification performed this session
 
-- Local prototype (Eq. 1 / 5.4): mean embedding over the client's class-`j` samples.
-- Server aggregation (Eq. 2 / 5.5): `omega = (1/M) sum_i omega_i`,
-  `Cbar_j = (1/M) sum_i C_ij` (equal, no `N_ij`).
-- Local loss (Eq. 3 / 5.6): cross-entropy + `lambda * ||C_ij - Cbar_j||^2` +
-  `(mu/2) * ||omega_i - omega^{t-1}||^2`, with `mu = 0.1`.
-- Inference: nearest global prototype by squared L2 (thesis Eqs. 5.11–5.13).
-- Model: 2 Conv1D (64, 128 filters) + ReLU + maxpool + dropout (0.2, 0.5), flatten,
-  dense 128, log-softmax head.
-- Setup: 10 clients, 80/20 train/test, 10 rounds, 3 local epochs,
-  `alpha in {0.75, 0.5, 0.25}`.
-- Thesis (Sec. 5.3.6): "prototypes are shared only for locally observed classes".
+- `python -m pytest -q` -> **34 passed**.
+- Full partition generation on both datasets: 18 manifests, each reload-verified.
+- Non-IID check (seed 0): X-IIoTID `alpha=0.75` max 1 missing class/client, `alpha=0.25`
+  up to 3; 5G-NIDD similar. Qualitatively consistent with the paper setting.
+- Baseline-only smoke on X-IIoTID (3000 rows, 4 clients, 2 rounds, 1 epoch): full path
+  succeeds, ~4.5 s.
+- Baseline-only smoke on 5G-NIDD (4000 rows): succeeds.
+- No beta/support_weighted code is invoked or present.
+- Integrity: `planningawal/` and both raw datasets retain their recorded SHA-256 values.
 
-### 6.3 Missing-class findings (critical)
+## 10. Files added this session
 
-- Equation (2)/5.5 sums over all `M` participants, but the paper never states how a missing
-  class prototype is handled. The thesis implies absent classes are not shared, but the
-  server denominator (`M` vs number of contributors) is undefined.
-- Therefore `beta = 0` over `S_j` is **not** automatically equal to the original baseline;
-  equality requires the same participant rule. This is the central baseline ambiguity.
-- `original_protean` is deliberately **not implemented** in
-  `src/protean_research/aggregation/prototype_aggregation.py`; it raises
-  `NotImplementedError` with an explanatory message.
-- Additional unknowns: round-0 global-prototype initialisation; how the alignment loss sums
-  over `j = 1..K` when a client has `N_ij = 0`; per-batch vs end-of-round prototypes;
-  whether model aggregation is equal `1/M` or FedAvg sample-weighted (the thesis intro says
-  FedAvg is a weighted average, contradicting Eq. 5.5); numeric value of `lambda`; seed values;
-  feature preprocessing.
+Source: `src/protean_research/data/{__init__,loading,preprocessing,splits,partition,manifest,
+characterization,rare_unseen,pipeline}.py`,
+`src/protean_research/models/{__init__,protean_cnn,prototypes}.py`,
+`src/protean_research/federated/{__init__,client,server,round}.py`,
+`src/protean_research/evaluation/{__init__,metrics}.py`,
+`src/protean_research/utils/{seeding,integrity}.py`.
 
-### 6.4 Dataset findings
+Scripts: `scripts/run_baseline.py`, `scripts/generate_partitions.py`.
 
-- X-IIoTID: 820,834 × 68; labels `class1` (19), `class2` (10 general), `class3` (binary).
-  `class2` benign = `Normal`. After dropping `crypto-ransomware`: 9 classes. `'-'`
-  placeholders in 42 columns; 6,222 duplicate rows; label typo `Lateral _movement`.
-  Identifiers: `Date`, `Timestamp`, `Scr_IP`, `Scr_port`, `Des_IP`, `Des_port`.
-- 5G-NIDD: 1,215,890 × 52; `Attack Type` (9 values, benign = `Benign`), `Label` binary,
-  `Attack Tool` (6). 8 attack classes + Benign = 9. Heavy structural missingness (e.g. `dVid`
-  99.83%, `dTos` 77.56%); 1 duplicate row. Identifiers: empty-header index, `Seq`, `Offset`.
-- Both map to PROTEAN's 9-class setup, but PROTEAN's exact preprocessing is unspecified and
-  must be frozen here.
+Tests: `tests/test_data_pipeline.py`, `tests/test_partition.py`, `tests/test_prototypes.py`,
+`tests/test_baseline_aggregation.py`, `tests/test_metrics.py`, `tests/test_federated_smoke.py`.
 
-## 7. Decisions made
+Docs: `docs/baseline/BASELINE_IMPLEMENTATION.md`.
 
-1. **Treat `planningawal/` as immutable** and keep all outputs elsewhere.
-2. **Do not implement `original_protean` until its semantics are verified.** Encode only the
-   unambiguous modes (`support_weighted`, `equal_available`). Rationale: avoid silently
-   assuming baseline behaviour.
-3. **Keep `equal_available` as a dormant control**, introduced only if baseline investigation
-   shows the participant rule differs from `S_j` (per the specification).
-4. **Centralise dataset paths** in `configs/datasets/datasets.yaml` with env overrides; never
-   hardcode in source.
-5. **Separate raw and generated data**; generated outputs under `artifacts/`.
-6. **Use `class2` for X-IIoTID and `Attack Type` for 5G-NIDD** as the multi-class labels, drop
-   `crypto-ransomware`, treat `Normal`/`Benign` as benign. Matches the paper.
-7. **Keep the pre-existing scaffold** (configs + aggregation + utils) and extend it minimally
-   rather than restructuring; no evidence justified a different layout.
-8. **Add dependency-free unit tests** for aggregation invariants now; defer data/federated
-   tests until those modules exist.
-9. **Do not initialize git during the planning session** (not requested at the time). A
-   follow-up checkpoint task then initialized Git and added remote `origin`; see Section 11.
-10. **Mark seeds, `lambda`, validation fraction, Dirichlet allocation, and preprocessing as
-    PROVISIONAL**; they must be frozen before any sweep.
+Artifacts (tracked, small): `artifacts/partitions/*.json` (18),
+`artifacts/analysis/*.csv` (36).
 
-## 8. Unresolved questions
+## 11. Files modified this session
 
-1. Baseline missing-class rule and prototype-aggregation denominator (`M` vs `|S_j|`).
-2. Round-0 initialisation of global prototypes for all `K` classes.
-3. `lambda` value (paper gives `mu = 0.1` only).
-4. Whether model-parameter aggregation is equal or FedAvg sample-weighted.
-5. How the prototype alignment loss treats `N_ij = 0` classes.
-6. Per-batch vs end-of-round local prototype computation.
-7. Validation split fraction and procedure (the paper has no validation set).
-8. Exact Dirichlet allocation algorithm (affects realised `N_ij`).
-9. Exact seed values and whether metrics are averaged per client then per seed.
-10. Feature preprocessing: encoding, scaling, `'-'`/NaN handling, duplicate-row policy.
-11. ~~Whether to initialize git to protect the immutable source.~~ Resolved: Git was
-    initialized in the follow-up checkpoint task (see Section 11).
-12. Compute budget: no timing measured; a profiling smoke test is required before any sweep.
-
-## 9. Files created this session
-
-- `docs/research/RESEARCH_SPEC_INTERPRETATION.md`
-- `docs/research/DATASET_INSPECTION.md`
-- `docs/research/IMPLEMENTATION_PLAN.md`
+- `configs/datasets/datasets.yaml`
+- `configs/experiments/experiment_protocol.yaml`
 - `docs/research/EXPERIMENT_PROTOCOL.md`
-- `docs/baseline/PROTEAN_BASELINE_INVESTIGATION.md`
-- `docs/handoff/SESSION_HANDOFF.md`
+- `docs/research/IMPLEMENTATION_PLAN.md`
 - `README.md`
-- `pyproject.toml`
-- `.gitignore`
-- `tests/test_prototype_aggregation.py`
+- `src/protean_research/aggregation/__init__.py`
+- `src/protean_research/aggregation/prototype_aggregation.py`
+- `src/protean_research/models/prototypes.py`
+- `src/protean_research/federated/server.py`
+- `src/protean_research/federated/round.py`
+- `scripts/run_baseline.py`
+- tests (baseline-only aggregation suite; beta tests removed)
 
-## 10. Files modified this session
+`planningawal/` was not touched.
 
-- None. No existing file was edited. `planningawal/` was not touched (and must not be).
+## 12. Removed from this checkpoint (deferred to beta/controls phase)
 
-Pre-existing files (present at session start, verified but not modified):
-`configs/datasets/datasets.yaml`, `configs/experiments/experiment_protocol.yaml`,
-`configs/models/protean_cnn.yaml`, `src/protean_research/__init__.py`,
-`src/protean_research/aggregation/__init__.py`,
-`src/protean_research/aggregation/prototype_aggregation.py`,
-`src/protean_research/utils/__init__.py`, `src/protean_research/utils/config.py`.
+- `support_weighted` aggregation code and `beta` parameters.
+- `equal_available` mode (redundant with the baseline `S_j` rule; only needed if a later
+  investigation shows the baseline participant rule differs).
+- `--mode` / `--beta` CLI options in `run_baseline.py`.
+- Beta-specific tests (`beta = 0/1`, single-contributor/equal-support beta invariance,
+  normalised beta weights, the `original_protean == support_weighted(beta=0)` equivalence
+  test). The former `tests/test_prototype_aggregation.py` was removed and folded into a
+  baseline-only `tests/test_baseline_aggregation.py`.
+- Any claim that `original_protean == support_weighted(beta=0)` demonstrates equivalence
+  with original PROTEAN. That was at most an internal implementation-equivalence check and
+  is no longer present in code or docs.
 
-## 11. Git status
+## 13. Git status
 
-The planning session ran with **no Git repository**. A follow-up checkpoint task then
-initialized Git:
+Not committed (no commit was requested). `git status --short` shows modified tracked files
+(configs, docs, aggregation package, federated modules, README, run script) plus untracked
+source/tests/scripts/docs, the 18 partition manifests and 36 analysis CSVs. `AGENTS.md` was
+already untracked before this session. `.venv/`, `artifacts/processed/`, `results/`,
+`logs/`, and `*.log` are ignored. No checkpoints or raw datasets are tracked.
 
-- Branch: `main`
-- Remote `origin`: `git@github.com:willtanoe/prototype-agregation.git`
-- Initial checkpoint commit: `chore: checkpoint PROTEAN research planning and architecture`
-  (exact hash reported in the checkpoint task's terminal summary; deliberately not embedded
-  here to avoid a recursive "hash inside the commit" loop).
-- `.gitkeep` placeholders were added for `scripts/`, `artifacts/partitions/`, and
-  `artifacts/analysis/`. `artifacts/processed/`, `artifacts/checkpoints/`, `results/`,
-  `logs/`, and `*.log` are ignored.
-- `planningawal/PROTEAN_Revisi_Lurus.md` is tracked exactly as-is and was not modified.
-- Raw datasets are outside the repository and are not tracked.
+## 14. Unresolved issues / blockers
 
-Because this is the initial commit, `planningawal/` was verified with SHA-256 hashing and
-directory listing rather than `git diff`.
+1. **Baseline is implemented but not yet VALIDATED.** The `original_protean` participant
+   rule is a reproduction assumption, not verified behavior.
+2. `lambda`, optimiser/schedule, and round-0 handling are unverified.
+3. No full-scale run has been timed; compute feasibility of a real sweep is unmeasured
+   (X-IIoTID ~570k train rows, 5G-NIDD ~851k; CPU-only).
+4. Per-client evaluation partitioning is a project decision.
+5. Prototype-alignment form (per-sample vs class-mean) is a documented inference.
+6. Whether the literal `1/M` missing-class reading differs from the `S_j` assumption is
+   open; a deliberate control is needed in the verification phase.
 
-## 12. Verification performed
+## 15. Exact next task
 
-- Read the full specification (118 lines).
-- Read the full paper HTML and extracted the author PhD thesis text (195 pages).
-- GitHub API searches (author account, repositories), DuckDuckGo, `grep.app`.
-- Full pandas load of both datasets; recorded shape, dtypes, label counts, missingness,
-  duplicates, identifier columns.
-- SHA-256 of both datasets and of `planningawal/PROTEAN_Revisi_Lurus.md`.
-- Ran the pre-existing aggregation module and config resolver (modes, weights, path
-  resolution, `original_protean` raises `NotImplementedError`).
-- Ran `python -m pytest -q` in a temporary venv: **9 passed**.
-- Confirmed no training or sweep was launched.
-- Final check: `planningawal/` contains only `PROTEAN_Revisi_Lurus.md` with the same hash;
-  raw datasets retain the same hashes and mtimes.
+**Baseline validation and reproduction verification — NOT beta weighting.**
 
-Checkpoint task (follow-up):
+Next session should:
 
-- `git rev-parse --is-inside-work-tree` before init: not a repository.
-- Searched the whole tree for `*.csv`, `*.parquet`, `*.feather`, `*.pkl`, `*.pickle`, model
-  checkpoints: none present; largest file is the 12 KB spec.
-- Scanned tracked text for secret patterns (`API_KEY`, `SECRET`, `TOKEN`, `PASSWORD`,
-  `PRIVATE KEY`, `BEGIN OPENSSH`, `BEGIN RSA`, `sk-`, `ghp_`, `github_pat_`): none found.
-- `find planningawal -maxdepth 2 -type f` → only `PROTEAN_Revisi_Lurus.md`.
-- Re-ran `pytest`: 9 passed.
+1. Run one full-data baseline condition (e.g. X-IIoTID, `alpha=0.5`, `seed=0`,
+   `original_protean`) with measured wall-clock and memory; record it.
+2. Inspect the run: per-class precision/recall/F1, per-client rare metrics, confusion
+   matrix. Check plausibility and missing-class-client behavior.
+3. Stress-test the `S_j` reproduction assumption: compare against a literal `1/M`
+   omitted-class reading to quantify the effect of the assumption (introduce the control
+   only for this investigation, clearly marked).
+4. Add a deterministic end-to-end baseline regression test pinning a small result.
+5. Only after the baseline is demonstrably verified and recorded, begin the paired beta
+   comparison on a single `(dataset, alpha, seed)` before any grid.
 
-## 13. Exact next task
+Do **not** launch the full grid, and do not change the architecture, loss coefficients, or
+inference rule.
 
-**Implement the deterministic data foundation, and freeze the remaining protocol choices
-before coding.**
+## 16. Important commands
 
-Concretely, the next session should:
+```bash
+# Tests
+.venv/bin/python -m pytest -q
 
-1. In `docs/research/EXPERIMENT_PROTOCOL.md`, replace the PROVISIONAL entries with explicit
-   frozen decisions and rationale for: preprocessing recipe, duplicate-row policy,
-   `'-'`/NaN handling, the Dirichlet allocation algorithm, the train/validation/test split
-   procedure and fraction, the seed set, `lambda`, batch size, and learning rate. These are
-   project decisions because the paper does not specify them; label them as reproduction
-   deviations.
-2. Create `src/protean_research/data/` with `loading.py`, `preprocessing.py`, `splits.py`,
-   `partition.py`, `manifest.py`, `characterization.py`, `rare_unseen.py` implementing the
-   flow in `docs/research/IMPLEMENTATION_PLAN.md` Section 3.
-3. Persist one partition manifest per `(dataset, alpha, seed)` under `artifacts/partitions/`
-   with a content hash, and derive rare/unseen metadata from the training split.
-4. Add tests for determinism of partition generation (strategy item 8) and identical manifest
-   reload (item 9), plus the non-IID characterization fields.
+# Generate / verify all partition manifests (full data, ~30 s)
+.venv/bin/python scripts/generate_partitions.py
 
-Do **not** start model training or a beta sweep. The baseline missing-class rule is still
-unresolved, so `original_protean` remains unimplemented.
+# Baseline smoke on both datasets (writes only to ignored dirs)
+.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.5 --seed 0 \
+    --max-rows 3000 --sample-from 60000 --clients 4 --rounds 2 --local-epochs 1
+.venv/bin/python scripts/run_baseline.py --dataset 5gnidd --alpha 0.25 --seed 1 \
+    --max-rows 4000 --sample-from 60000 --clients 4 --rounds 2 --local-epochs 1
 
-## 14. Recommended next-session prompt
+# Full-data baseline for one condition (next session; expect long CPU runtime)
+.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.5 --seed 0 \
+    --clients 10 --rounds 10 --local-epochs 3
+```
+
+## 17. Recommended next-session prompt
 
 ```
-You are continuing the perdalam-protean research repository.
+You are continuing the perdalam-protean research repository (Session 3).
 
-FIRST, read these two files completely:
+FIRST, read completely:
   1. docs/handoff/SESSION_HANDOFF.md
-  2. planningawal/PROTEAN_Revisi_Lurus.md
+  2. docs/baseline/BASELINE_IMPLEMENTATION.md
+  3. docs/research/EXPERIMENT_PROTOCOL.md
 
-Then read, as needed:
-  - docs/research/IMPLEMENTATION_PLAN.md
-  - docs/research/EXPERIMENT_PROTOCOL.md
-  - docs/research/DATASET_INSPECTION.md
+Then read as needed:
+  - planningawal/PROTEAN_Revisi_Lurus.md
   - docs/baseline/PROTEAN_BASELINE_INVESTIGATION.md
+  - docs/research/IMPLEMENTATION_PLAN.md
 
 Critical rules:
   - planningawal/ is immutable. Do not edit/create/move/rename/delete anything inside it.
   - ~/datasets/X-IIoTID/X-IIoTID-dataset.csv and ~/datasets/5G-NIDD/5G-NIDD.csv are
     read-only external source data. Do not modify or copy them into the repo.
-  - Do NOT run full training or an experimental sweep.
+  - This checkpoint is BASELINE ONLY. Do NOT add the proposed support_weighted/beta code
+    or the equal_available control until the baseline is verified.
+  - Do NOT launch the full alpha x seed x dataset x beta sweep.
 
-Your task is Section 13 of the handoff: first freeze the PROVISIONAL protocol decisions in
-docs/research/EXPERIMENT_PROTOCOL.md, then implement the deterministic data foundation in
-src/protean_research/data/ with partition-manifest persistence, rare/unseen metadata, and
-tests for partition determinism and manifest reload.
+Your task is Section 15 of the handoff: BASELINE VALIDATION AND REPRODUCTION
+VERIFICATION. Run one full-data baseline condition with measured runtime, inspect its
+per-class/per-client behavior, stress-test the S_j participant-rule assumption, and add a
+deterministic end-to-end baseline regression test. Record everything in
+docs/baseline/BASELINE_IMPLEMENTATION.md and update docs/handoff/SESSION_HANDOFF.md.
 
-Start by reading the handoff, then continue from the documented state. Update
-docs/handoff/SESSION_HANDOFF.md at the end of your session.
+Use .venv/bin/python for all Python commands.
 ```

docs/research/EXPERIMENT_PROTOCOL.md

@@ -1,10 +1,15 @@
 # Experiment Protocol
 
-Status: frozen design (to be executed in later sessions)
-Machine-readable counterpart: `configs/experiments/experiment_protocol.yaml`
+Status: **FROZEN** reproduction protocol (2026-09-24).
+Machine-readable counterpart: `configs/experiments/experiment_protocol.yaml`.
 
-Values marked **PROVISIONAL** require confirmation before any scientific sweep. Values
-marked **FROZEN** must remain identical across every aggregation variant.
+Values marked **FROZEN** must remain identical across every aggregation variant, beta
+candidate and seed. Values marked **PROJECT DECISION** are choices this project had to make
+because the paper/thesis leave them unspecified; they are reproduction deviations and must
+be reported as such. No value is PROVISIONAL any more.
+
+See `docs/baseline/PROTEAN_BASELINE_INVESTIGATION.md` for the evidence behind each choice and
+`docs/baseline/BASELINE_IMPLEMENTATION.md` for how the protocol is implemented.
 
 ---
 
@@ -24,59 +29,79 @@ Details and mismatches: `docs/research/DATASET_INSPECTION.md`.
 
 FROZEN: symmetric Dirichlet over classes, `num_clients = 10`, `alpha ∈ {0.75, 0.50, 0.25}`.
 
-- Partitions are generated from the **training split only**.
-- Class proportions are drawn per client from `Dirichlet(alpha)` and used to allocate
-  training samples per class to clients.
-- The exact Dirichlet implementation (per-class client allocation vs per-client class
-  sampling) must be fixed and documented, because it changes realised `N_ij`. `PROVISIONAL`.
-- Success criterion for reproducing the paper's setting: at low alpha, several clients miss
-  several classes, consistent with the paper's reported counts (e.g. X-IIoTID `alpha=0.25`:
-  8/10 clients miss 1–5 classes).
+FROZEN algorithm (**PROJECT DECISION**, paper does not specify the allocation procedure):
+
+1. Partitions are generated from the **training split only**.
+2. For each class `c`, draw client proportions `p ~ Dirichlet(alpha * 1_K_clients)`.
+3. Allocate that class's training samples to clients with
+   `counts ~ Multinomial(N_c, p)`, so the counts sum exactly to `N_c`.
+4. Split the class's shuffled sample positions into consecutive blocks of size `counts`.
+   The shuffle is a permutation drawn from the same deterministic RNG.
+5. A client may receive zero samples of a class; that class is then *missing* on the client
+   (`N_ij = 0`), which is the intended source of non-IIDness.
+
+Determinism: the RNG seed is derived as
+`rng_seed = sha256(f"{dataset}|dirichlet|alpha={alpha}|seed={seed}")[:16] mod 2**31`.
+The manifest stores `rng_seed`, so the same partition is reproduced exactly on reload.
+
+Success criterion for reproducing the paper's setting: at low alpha several clients miss
+several classes.
 
 ## 3. Splits
 
-- Test: 20% of each dataset (paper uses 80/20). FROZEN.
-- Validation: a separate split used only for beta selection. The paper has **no** validation
-  set, so this is an explicit project addition. `PROVISIONAL` fraction and procedure.
-- Requirement: train/validation/test come from a single deterministic split with a recorded
-  seed, and the same split is reused for every beta variant.
+FROZEN: a single deterministic **stratified** split per `(dataset, seed)`.
+
+- `test = 0.20` of the cleaned dataset (paper uses 80/20). FROZEN.
+- `validation = 0.10`; `train = 0.70`. **PROJECT DECISION**: the paper has no validation set;
+  this project adds one for beta selection. The test fraction keeps the paper's 20%.
+- Stratification is per class: each class's samples are shuffled with a deterministic RNG and
+  split 70/10/20. This guarantees every class is represented in validation and test, which is
+  required for rare-class evaluation.
+- Split seed: `sha256(f"{dataset}|split|seed={seed}")[:16] mod 2**31`.
+- The exact same split is reused for every aggregation variant and beta candidate.
 
 ## 4. Seeds
 
-- The paper averages over three runs with unnamed seeds. `PROVISIONAL` set: `{0, 1, 2}`.
-- Once frozen, the seed set is identical across all aggregation variants.
-- Each seed deterministically controls: split, Dirichlet partition, model initialisation,
-  batch order/shuffling.
-- A beta run MUST NOT derive a new partition; it loads the manifest for
-  `(dataset, alpha, seed)`.
+FROZEN (`PROJECT DECISION`): `{0, 1, 2}` (the paper averages over three unnamed seeds).
 
-## 5. Beta candidates and modes
+Each seed deterministically controls: split, Dirichlet partition, model initialisation, batch
+order/shuffling. All RNG use is isolated via `protean_research.utils.seeding.derive_seed`.
+A beta run MUST NOT derive a new partition; it loads the manifest for `(dataset, alpha, seed)`.
 
-FROZEN: `beta ∈ {0.00, 0.25, 0.50, 0.75, 1.00}`, one beta for all classes and clients.
+## 5. Aggregation modes
 
-Modes:
+**Baseline checkpoint: only `original_protean` is implemented.** The proposed
+support-weighted aggregation (`support_weighted`) and the `equal_available` control are
+**DEFERRED** to the later beta/controls phase and are not present in executable code.
 
-- `original_protean` — verified baseline semantics. Not yet encodable (see baseline
-  investigation). This is the baseline to reproduce.
-- `support_weighted` — `w_ij = N_ij^beta`, normalised over `S_j` (proposed mechanism).
-- `equal_available` — equal weights over `S_j`; a methodological control introduced only if
-  the baseline participant rule differs from `S_j`. Not to be presented as a novelty.
+- `original_protean` — reproduced baseline. **REPRODUCTION ASSUMPTION (project decision)**:
+  because Eq. (2)/(5.5) is internally inconsistent about missing classes, the baseline
+  participant rule is assumed to be `S_j = { i | N_ij > 0 and a valid C_ij exists }` with
+  **equal weights `1/|S_j|`**, following the thesis statement that prototypes are shared
+  only for locally observed classes. This is not verified against code and must be
+  validated in the baseline-verification phase.
+- `support_weighted` (deferred) — `w_ij = N_ij^beta`, normalised over `S_j`.
+- `equal_available` (deferred) — equal weights over `S_j`; only introduced later if the
+  baseline participant rule is shown to differ from `S_j`.
 
-`S_j = { i | N_ij > 0 and a valid local prototype C_ij exists }`.
+FROZEN candidates for the later phase: `beta ∈ {0.00, 0.25, 0.50, 0.75, 1.00}`, one beta for
+all classes and clients. They are **not** part of the baseline checkpoint.
 
 ## 6. Partition persistence
 
-Each `(dataset, alpha, seed)` produces one persisted manifest containing at least:
+Each `(dataset, alpha, seed)` produces one persisted JSON manifest under
+`artifacts/partitions/` containing:
 
-- dataset name, alpha, seed, num_clients, K;
-- per-client sample indices or a reproducible reference plus a content hash;
-- per-client `N_ij` matrix `[client][class]`;
-- class label mapping;
-- split indices for train/validation/test;
-- preprocessing/config hash and code version.
+- dataset key, alpha, seed, num_clients, num_classes, class names;
+- the Dirichlet `algorithm` id and `rng_seed`;
+- per-client `N_ij` matrix `[client][class]` (training split);
+- split metadata (seed, per-split sizes, content hash) and partition content hash;
+- preprocessing/config/code hashes.
 
-Manifests live under `artifacts/partitions/`. Reloading a manifest must reproduce the same
-partition (tested). Training reads manifests; it does not regenerate them.
+Manifests are small and version-controlled. Full sample-index arrays are **not** stored by
+default (they would be tens of MB per manifest); they are regenerated deterministically from
+the stored recipe and RNG seed and verified against the stored `N_ij` and content hash. This
+is the tested reload path: `load_manifest` -> `reproduce_partition` -> hash match.
 
 ## 7. Rare and unseen classes
 
@@ -96,18 +121,17 @@ Persisted metadata (minimum one row per `(dataset, alpha, seed, client, class)`)
 
 Procedure, strictly ordered:
 
-1. **Candidate evaluation (validation only).** For each beta candidate, train with the
-   paired manifest and compute validation metrics. Primary criterion: rare-class macro-F1,
-   computed per client that has rare classes, then averaged over those clients.
-2. **Beta selection.** Choose the beta with the best mean validation rare-class macro-F1
-   (ties broken by a pre-declared rule, e.g. smaller beta). The result is
-   "best among tested candidates under this procedure", never a global optimum.
+1. **Candidate evaluation (validation only).** For each beta candidate, train with the paired
+   manifest and compute validation metrics. Primary criterion: rare-class macro-F1, computed
+   per client that has rare classes, then averaged over those clients.
+2. **Beta selection.** Choose the beta with the best mean validation rare-class macro-F1 (ties
+   broken by smaller beta). The result is "best among tested candidates under this
+   procedure", never a global optimum.
 3. **Final test evaluation.** Evaluate the selected beta on the test split. Report overall
    macro-F1, rare-class recall, rare-class macro-F1, and per-client results, plus the test
    results of all candidates for transparency.
 
-Test data is never used in steps 1–2. The selection code must not have access to test
-outputs; test evaluation is a separate step keyed on the selected beta.
+Test data is never used in steps 1–2.
 
 ## 9. Required metrics
 
@@ -117,29 +141,28 @@ Per dataset/alpha/seed/beta and per client, record:
 - rare-class recall;
 - overall macro-F1;
 - accuracy (for comparison with the paper);
-- per-class precision/recall/F1 and confusion counts;
+- per-class precision/recall/F1 and support (confusion counts persisted);
 - per-client metrics.
 
-Also record the validation/test rare-class support counts, so rare-class metrics are never
-read without their sample sizes. Results are reported as mean and variation across seeds.
+Also record the validation/test rare-class support counts. Results are reported as mean and
+variation across seeds.
 
 ## 10. Non-IID characterization
 
 For every manifest, record at least: `N_ij`, class total, number of clients containing class
 `j`, `min N_ij`, `max N_ij`, `mean N_ij`, `std N_ij`, coefficient of variation, and
-`max/min` ratio where defined. Alpha alone is insufficient; different seeds realise
-different distributions.
+`max/min` ratio where defined. Alpha alone is insufficient.
 
 ## 11. Direct prototype analysis
 
 Separate the direct mathematical aggregation effect from the end-to-end training effect.
 
-- During a run, persist at least one round's fixed `C_ij` and `N_ij` snapshot.
+- Persist at least one round's fixed `C_ij` and `N_ij` snapshot.
 - Reusing those snapshots, compute `Cbar_j(beta)` for several beta without retraining.
-- Report `|| Cbar_j(beta) - Cbar_j(0) ||`; optionally the normalised aggregation weights,
-  distances to contributing prototypes, pairwise global-prototype displacement, and support
+- Report `|| Cbar_j(beta) - Cbar_j(0) ||`; optionally normalised aggregation weights,
+  distances to contributing prototypes, pairwise global-prototype displacement, support
   imbalance.
-- This analysis complements, and never replaces, end-to-end evaluation.
+- This complements, and never replaces, end-to-end evaluation.
 
 ## 12. Paired comparison guarantee
 
@@ -154,24 +177,66 @@ For each `(dataset, alpha, seed)`, every beta variant and every mode reuses:
 
 A beta run that regenerates a fresh Dirichlet partition is a protocol violation.
 
-## 13. Frozen vs provisional summary
+## 13. Preprocessing (frozen)
 
-| Item | Value | Status |
+**PROJECT DECISION** — the paper specifies no preprocessing. Frozen recipe:
+
+1. Read the raw CSV read-only (`pandas.read_csv`, `low_memory=False`).
+2. Column names are stripped; the empty 5G-NIDD header becomes `Unnamed: 0`.
+3. Exact duplicate rows are dropped (`keep="first"`) before splitting.
+4. Rows whose label is in `drop_labels` are dropped.
+5. Labels are canonicalised: surrounding whitespace removed, runs of whitespace collapsed,
+   and whitespace around `_` removed (fixes `Lateral _movement`).
+6. Feature columns = all columns except the label column, its aliases, and
+   `non_feature_columns`.
+7. `missing_tokens` are replaced with NaN. Categorical missing values become an explicit
+   `__MISSING__` category.
+8. Categorical columns (`categorical_columns`) are one-hot encoded with categories fitted on
+   the **train split only**; unseen categories at transform time become all-zero.
+9. Numeric columns are coerced to numeric, median-imputed and standardized using **train
+   statistics only**.
+10. Output is a deterministic `float32` matrix with a fixed column order.
+
+No statistic from validation or test data ever enters preprocessing.
+
+## 14. Model, local training, inference (frozen)
+
+- Model: 2 Conv1D (64, 128 filters, ReLU, max-pool, dropout 0.2/0.5), flatten, dense 128
+  (embedding), log-softmax head. **PROJECT DECISION**: kernel size 3, padding 1.
+- Local loss: `cross_entropy + lambda * ||phi(x) - Cbar_{y}||^2 + (mu/2) * ||omega_i - omega||^2`,
+  with `mu = 0.1` (paper), `lambda = 1.0` (**PROJECT DECISION**, paper silent). The prototype
+  term is applied per sample only for classes that currently have a global prototype.
+- Optimizer Adam, `lr = 1e-3`, batch size 256, 3 local epochs, 10 rounds
+  (**PROJECT DECISION** where not stated).
+- Local prototypes: mean embedding over the client's full class-`j` training samples,
+  computed at the **end of the local round** (paper Eq. 1). Missing classes produce no
+  prototype.
+- Server: model parameters averaged with equal `1/M` weights (paper Eq. 2/5.5); class
+  prototypes aggregated with the active mode.
+- Round 0: no global prototypes exist; the prototype-alignment term is skipped; the first
+  global prototypes are produced by aggregating round-0 local prototypes.
+- Inference: embedding `z = phi(x)`, prediction `argmin_j ||z - Cbar_j||^2` over classes that
+  have a global prototype.
+
+## 15. Frozen vs decision summary
+
+| Item | Value | Source |
 |---|---|---|
-| Datasets | X-IIoTID, 5G-NIDD | FROZEN |
-| Clients | 10 | FROZEN |
-| Alpha | 0.75, 0.50, 0.25 | FROZEN |
-| Beta candidates | 0, 0.25, 0.50, 0.75, 1.00 | FROZEN |
-| Test fraction | 0.20 | FROZEN |
-| Rounds / local epochs | 10 / 3 | FROZEN |
-| `mu` | 0.1 | FROZEN |
-| `lambda` | unknown | PROVISIONAL |
-| Batch size / learning rate | unknown | PROVISIONAL |
-| Validation fraction / procedure | unknown | PROVISIONAL |
-| Seeds | 0, 1, 2 | PROVISIONAL |
-| Dirichlet allocation algorithm | unknown | PROVISIONAL |
-| Preprocessing recipe | unknown | PROVISIONAL |
-| Missing-class baseline rule | unknown | PROVISIONAL / blocked on baseline |
+| Datasets | X-IIoTID, 5G-NIDD | paper |
+| Clients | 10 | paper |
+| Alpha | 0.75, 0.50, 0.25 | paper |
+| Beta candidates | 0, 0.25, 0.50, 0.75, 1.00 | spec |
+| Test fraction | 0.20 | paper |
+| Rounds / local epochs | 10 / 3 | paper |
+| `mu` | 0.1 | paper |
+| `lambda` | 1.0 | project decision |
+| Batch size / learning rate | 256 / 1e-3 (Adam) | project decision |
+| Validation fraction | 0.10 (stratified) | project decision |
+| Seeds | 0, 1, 2 | project decision |
+| Dirichlet allocation | per-class Dirichlet + multinomial | project decision |
+| Preprocessing recipe | §13 | project decision |
+| Baseline participant rule | `S_j`, equal `1/|S_j|` | project decision |
+| Model aggregation | equal `1/M` | paper Eq. 2/5.5 |
+| Prototype computation | end-of-round mean embedding | paper Eq. 1 |
 
-The provisional items must be resolved and recorded here before the first sweep. Until then
-only smoke tests are permitted.
+Until the baseline is verified, only smoke tests are permitted; no sweep may be launched.

docs/research/IMPLEMENTATION_PLAN.md

@@ -1,6 +1,9 @@
 # Implementation Plan
 
-Status: design document (no full training implemented in this session)
+Status: design document. The data, model, federated and evaluation modules
+described below are now IMPLEMENTED for the baseline (see
+`docs/baseline/BASELINE_IMPLEMENTATION.md`); the analysis/direct-prototype module
+is still future work and beta experiments were not run in this session.
 Scope: architecture for the server-side prototype-aggregation study on top of a reproduced
 PROTEAN baseline.
 
@@ -107,19 +110,18 @@ regenerate a partition.
 - `round.py`: one communication round; records per-round prototype snapshots for direct
   analysis.
 
-### `aggregation/` (exists)
+### `aggregation/` (baseline implemented; proposed method deferred)
 
-Pure, dependency-free functions implementing only the unambiguous modes:
+Pure, dependency-free functions implementing only the reproduced baseline:
 
-- `beta_weights(supports, beta)` → `N_ij ** beta` (unnormalised).
-- `aggregation_weights(prototypes, mode, beta, supports)` → normalised weights summing to 1.
-- `aggregate_prototypes(...)` → `Cbar_j` for one class.
-- `AggregationMode`: `original_protean` (raises `NotImplementedError` until verified),
-  `support_weighted`, `equal_available`.
+- `aggregate_prototypes(prototypes, mode)` → `Cbar_j` for one class (equal mean over `S_j`).
+- `aggregate_class_prototypes(contributions, mode)` → per-class global prototypes; builds
+  `S_j` by dropping clients with `N_ij = 0` or a missing prototype.
+- `AggregationMode`: only `original_protean` in this checkpoint.
 
-Planned addition: a server-facing helper that accepts `{client_id: (C_ij, N_ij)}` for class
-`j`, constructs `S_j` (drop clients with `N_ij = 0` or missing prototype), and delegates to
-`aggregate_prototypes`. This keeps all weighting logic in one tested place.
+The proposed `support_weighted` mode (`N_ij ** beta`) and the `equal_available` control are
+**deferred** to the later beta/controls phase and are not present in executable code. The
+baseline participant rule (`S_j`, equal weights) is a documented reproduction assumption.
 
 ### `evaluation/` (planned)
 
@@ -139,20 +141,23 @@ Planned addition: a server-facing helper that accepts `{client_id: (C_ij, N_ij)}
   `Cbar_j(beta)` for several beta without retraining and report
   `||Cbar_j(beta) - Cbar_j(0)||`, weights, distances, displacement, imbalance.
 
-## 5. Aggregation API concept
+## 5. Aggregation API (baseline)
 
 ```
-aggregate_prototypes(prototypes, mode, beta=None, supports=None) -> list[float]
+aggregate_prototypes(prototypes, mode="original_protean") -> list[float]
+aggregate_class_prototypes(contributions, mode="original_protean") -> dict[int, list[float]]
 ```
 
-Contract:
+Contract (implemented baseline):
 
-- `prototypes[i]` is `C_ij`; `supports[i]` is `N_ij > 0`; order is client order.
-- `support_weighted`: weights `N_ij ** beta`, normalised over `S_j`.
-- `equal_available`: equal weights over `S_j` (control; not for `beta > 0` claims).
-- `original_protean`: only after the baseline missing-class rule is verified and encoded.
-- Validates finiteness, strictly positive supports, equal dimensions; normalised weights sum
-  to 1; never emits NaN/Inf for valid inputs.
+- `prototypes[i]` is `C_ij`; order is client order.
+- `original_protean`: equal mean over the supplied contributor set `S_j`.
+- `aggregate_class_prototypes` builds `S_j` by dropping clients with `N_ij <= 0` or a
+  missing prototype; missing classes are never fabricated.
+- Validates finiteness and equal dimensions; never emits NaN/Inf for valid inputs.
+
+The proposed `support_weighted(beta, supports)` and `equal_available` modes are **planned**
+for the later beta/controls phase and are not implemented in this checkpoint.
 
 ## 6. Experiment orchestration
 
@@ -169,22 +174,24 @@ Contract:
 
 ## 7. Testing strategy
 
-Required tests (from the specification), to be implemented as the code is built:
+Implemented in the baseline checkpoint (see `tests/`):
 
-1. `beta = 0` equals the arithmetic mean over the same `S_j`.
-2. `beta = 1` equals sample-count weighting (equals pooling of included embeddings).
-3. One contributing client gives the same output for all beta.
-4. Equal supports give the same output for all beta.
-5. A missing class does not enter `S_j`.
-6. Normalised weights sum to one.
-7. Valid inputs never produce NaN/Inf.
-8. Partition generation is deterministic for a fixed seed.
-9. Saved partition manifests reload identically.
-10. Baseline behaviour is unchanged by adding the proposed mode.
-11. Validation/test separation is maintained.
+- baseline aggregation is the equal mean over `S_j`;
+- a missing class does not enter `S_j` and is never fabricated;
+- valid inputs never produce NaN/Inf; dimension/finiteness validation;
+- partition generation is deterministic for a fixed seed;
+- saved partition manifests reload identically (hash-verified);
+- rare/unseen semantics; prototype means; metrics on known fixtures;
+- a small end-to-end federated smoke test.
 
-This session adds the dependency-free subset (1–7) in `tests/test_prototype_aggregation.py`.
-Items 8–11 require the future data/federated modules and are deferred.
+Deferred to the beta/controls phase (proposed-method invariants):
+
+- `beta = 0` equals the arithmetic mean over the same `S_j`;
+- `beta = 1` equals sample-count weighting;
+- single-contributor and equal-support invariance across beta;
+- normalised beta weights sum to one;
+- baseline behaviour unchanged by adding the proposed mode;
+- paired beta/validation/test separation.
 
 ## 8. Compute feasibility
 

docs/research/RESEARCH_SPEC_INTERPRETATION.md

@@ -88,9 +88,10 @@ must still be verified.
 the participant rule from the effect of the weights. The spec says such a control is only
 introduced if needed.
 
-`IMPLEMENTATION DECISION` `equal_available` is kept as a dormant, documented control mode.
-It must not be reported as a result unless baseline investigation demonstrates its
-necessity (see `docs/baseline/PROTEAN_BASELINE_INVESTIGATION.md`).
+`IMPLEMENTATION DECISION` The `equal_available` control is **deferred** to the later
+beta/controls phase and is not implemented in the baseline-only checkpoint. It is
+introduced only if baseline investigation demonstrates that the baseline participant rule
+differs from `S_j` (see `docs/baseline/PROTEAN_BASELINE_INVESTIGATION.md`).
 
 ## 5. Rare class and unseen class definitions