willtanoe/thesis-prototype-agregation
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: implement reproducible PROTEAN baseline pipeline

Browse Source
This commit is contained in:
willtanoe committed 2026-09-24 17:39:00 +00:00
1 parent 2cc6a9859e
commit 0aa8e2ba79
93 files changed
+8173 -658
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/baseline/BASELINE_IMPLEMENTATION.md
+210
View File
@@ -0,0 +1,210 @@
# Baseline Implementation
Status: minimum reproducible baseline implemented and smoke-tested (2026-09-24).
Scope: the pipeline needed to run original-PROTEAN experiments. The proposed
beta-weighted aggregation and the `equal_available` control are **NOT implemented**
in this checkpoint; they are deferred to the later beta/controls phase. No beta
sweep was launched.
Companion documents:
- `docs/baseline/PROTEAN_BASELINE_INVESTIGATION.md` — evidence for baseline semantics.
- `docs/research/EXPERIMENT_PROTOCOL.md` — protocol and provenance tags.
- `docs/research/IMPLEMENTATION_PLAN.md` — architecture.
- `docs/handoff/SESSION_HANDOFF.md` — continuation point.
## 0. Provenance legend
Every decision below is classified as one of:
- **CODE** — confirmed from original PROTEAN code. **None exist**: no official
implementation is public (`PROTEAN_BASELINE_INVESTIGATION.md` §2).
- **PAPER** — stated in the paper and/or the author's PhD thesis.
- **INFERRED** — not written verbatim but a defensible reading of the paper/thesis.
- **PROJECT DECISION** — unspecified by the paper; chosen here and reported as a
reproduction deviation.
- **UNRESOLVED** — still unknown.
## 1. Implemented modules
| Module | Responsibility |
|---|---|
| `data/loading.py` | spec resolution, read-only CSV read, label canonicalisation, duplicate drop, feature selection |
| `data/preprocessing.py` | train-fitted median imputation + standardisation, one-hot encoding, serialisable state |
| `data/splits.py` | deterministic stratified 70/10/20 split, leakage guard |
| `data/partition.py` | Dirichlet non-IID partition, `N_ij`, content hash, eval-split partition |
| `data/manifest.py` | manifest save/load, exact reproduction with hash verification |
| `data/characterization.py` | per-class and per-client non-IID statistics |
| `data/rare_unseen.py` | per-client rare/unseen class derivation |
| `data/pipeline.py` | end-to-end deterministic data wiring |
| `models/protean_cnn.py` | baseline CNN, parameter get/set/average, embedding extraction |
| `models/prototypes.py` | local prototypes, global prototype collection, nearest-prototype inference |
| `aggregation/prototype_aggregation.py` | baseline `original_protean` aggregation only |
| `federated/client.py` | local update (CE + prototype alignment + model alignment) |
| `federated/server.py` | model averaging (1/M), prototype aggregation, global inference |
| `federated/round.py` | round orchestration + per-round validation metrics |
| `evaluation/metrics.py` | accuracy, macro/weighted F1, per-class P/R/F1, rare/unseen metrics |
| `utils/seeding.py`, `utils/integrity.py` | deterministic seed derivation, hashing |
| `scripts/run_baseline.py` | one-condition baseline runner / smoke test |
| `scripts/generate_partitions.py` | generate + verify all `(dataset, alpha, seed)` manifests |
## 2. Preprocessing decisions (all **PROJECT DECISION**)
The paper specifies no preprocessing. Frozen recipe:
1. Raw CSVs read read-only; paths resolve from config with env overrides.
2. Column names stripped; the empty 5G-NIDD header becomes `Unnamed: 0`.
3. Exact duplicate rows dropped (`keep="first"`): X-IIoTID 6,222; 5G-NIDD 1.
4. `crypto-ransomware` dropped from X-IIoTID (**PAPER**).
5. Labels canonicalised (whitespace collapse, whitespace around `_` removed), fixing
`Lateral _movement` -> `Lateral_movement`.
6. Feature columns exclude the label column, its aliases and configured non-feature
columns.
7. `categorical_columns` one-hot encoded with a `__MISSING__` category; all other
columns coerced to numeric, missing tokens -> NaN, median-imputed.
8. Imputation medians and standardisation statistics fitted on the **train split only**;
fitted state persisted and reused for validation/test.
9. Deterministic `float32` output matrix.
No statistic from validation or test data enters preprocessing.
## 3. Split decisions
| Decision | Value | Provenance |
|---|---|---|
| Test fraction | 0.20 | PAPER (80/20) |
| Validation fraction | 0.10 | PROJECT DECISION (paper has no validation set) |
| Train fraction | 0.70 | PROJECT DECISION (remainder) |
| Stratification | per class | PROJECT DECISION (ensures rare classes in val/test) |
| Split seed | derived from `(dataset, "split", seed)` | PROJECT DECISION |
## 4. Partition decisions
| Decision | Value | Provenance |
|---|---|---|
| Algorithm | per-class Dirichlet + multinomial allocation | PROJECT DECISION (paper unspecified) |
| Partition source | training split only | PAPER |
| RNG seed | derived from `(dataset, "dirichlet", alpha, seed)` | PROJECT DECISION |
| Missing classes | `N_ij = 0`, no prototype | PAPER (only observed classes shared) |
| Persistence | JSON manifest + content hash, indices regenerated & verified | PROJECT DECISION |
Observed non-IID behavior (seed 0): X-IIoTID `alpha=0.75` -> at most 1 missing class per
client; `alpha=0.25` -> up to 3. Qualitatively consistent with the paper setting.
## 5. Model provenance
| Element | Value | Provenance |
|---|---|---|
| Layer set | 2 Conv1D (64, 128) + ReLU + max-pool + dropout (0.2/0.5) + flatten + dense 128 + log-softmax | PAPER |
| Conv kernel / padding | 3 / 1 | PROJECT DECISION (paper silent) |
| Embedding `phi` | dense 128 output | PAPER (paper: "first eight of ten layers") |
| Prototype source | embedding | PAPER / INFERRED |
## 6. Local training and loss provenance
| Element | Value | Provenance |
|---|---|---|
| Objective form | CE + `lambda * prototype alignment` + `(mu/2) * model alignment` | PAPER Eq. (3) |
| `mu` | 0.1 | PAPER |
| `lambda` | 1.0 | PROJECT DECISION (paper gives no value) |
| Prototype alignment form | per-sample `\|\|phi(x) - Cbar_y\|\|^2` for classes with a global prototype | INFERRED (FedProto-style reading; paper writes class-mean to class-mean) |
| Model alignment target | global model of the round | INFERRED |
| Optimiser / lr / batch size | Adam / 1e-3 / 256 | PROJECT DECISION |
| Rounds / local epochs | 10 / 3 | PAPER |
| Local prototype | end-of-round mean embedding per observed class | PAPER Eq. (1) |
| Round 0 prototypes | none; prototype term skipped; first prototypes from round-0 aggregation | INFERRED / PROJECT DECISION |
## 7. Aggregation provenance — `original_protean` status
**`original_protean` is implemented, but its participant rule is a REPRODUCTION
ASSUMPTION, not verified original behavior.**
- **PAPER**: Eq. (2)/(5.5) writes `Cbar_j = (1/M) * sum_{i} C_ij` and model averaging
`omega = (1/M) * sum_i omega_i`.
- **PAPER**: the thesis states prototypes are shared only for locally observed classes.
- **UNRESOLVED**: the paper never defines how a class missing on a client (`N_ij = 0`) is
handled, nor whether the denominator is `M` or the number of contributors.
- **REPRODUCTION ASSUMPTION (PROJECT DECISION)**: the server receives prototypes only from
`S_j = { i | N_ij > 0 and a valid C_ij exists }` and averages them with **equal weights
`1/|S_j|`**. This follows the thesis sentence above but is not source-confirmed.
Consequences:
- `original_protean` is therefore a *documented interpretation* of the baseline.
It must be validated in the baseline-verification phase before any between-method
comparison.
- The model-aggregation rule (equal `1/M`) is implemented separately and is closer to the
paper text.
Whether a literal `1/M` reading with omitted classes coincides with this assumption is an
open question for the verification phase; the `equal_available` control that could
distinguish them is deliberately deferred.
## 8. Inference provenance
| Element | Value | Provenance |
|---|---|---|
| Rule | nearest global prototype by squared L2 | PAPER (thesis Eqs. 5.11–5.13) |
| Classes considered | those with a global prototype | INFERRED |
## 9. Seeds
- Exact values: **`{0, 1, 2}`**, frozen in `configs/experiments/experiment_protocol.yaml`.
- Rationale: the paper averages over three random seeds but names none, so the values are
**not source-confirmed**. `{0, 1, 2}` is the minimal three-seed set and is an explicit
**PROJECT DECISION**.
- Each seed deterministically controls the split, Dirichlet partition, model
initialisation and batch order, via
`protean_research.utils.seeding.derive_seed` (never Python `hash()`).
- The seed set is identical across all baseline conditions.
## 10. Manifests
There are **18** partition manifests under `artifacts/partitions/`, from
`2 datasets × 3 alphas × 3 seeds = 18` combinations of `(dataset, alpha, seed)`. Each is
content-hashed, reload-verified, and intended to be reused by every later aggregation
variant so that comparisons stay paired. Full sample-index arrays are regenerated
deterministically from the stored recipe/RNG seed and checked against the stored `N_ij`
and hash, so manifests stay small.
## 11. Tests
`python -m pytest -q` -> **39 tests** (see run result in the handoff). Coverage includes
dataset config resolution, deterministic label mapping, deterministic/train-only
preprocessing, deterministic stratified split, deterministic Dirichlet partition,
manifest reload identity and corruption detection, rare/unseen semantics, local prototype
means, baseline aggregation (including that `support_weighted` is not implemented), metrics
on known fixtures, and a small end-to-end federated smoke test.
## 12. Smoke test
```
.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.5 --seed 0 \
--max-rows 3000 --sample-from 60000 --clients 4 --rounds 2 --local-epochs 1
```
Runs on both datasets through the full path (load -> clean -> split -> preprocess ->
partition -> local training -> prototypes -> baseline aggregation -> global
redistribution -> evaluation -> artifacts). Smoke runs write only to ignored directories
(`artifacts/processed/smoke/`, `results/smoke/`) and are **not** thesis results. The
smoke test does not invoke the proposed method.
## 13. Structured outputs
Per run under `results/<run_name>/`: `metadata.json`, `history.json`,
`test_global_metrics.json`, `test_confusion_matrix.json`, `test_per_class.csv`,
`per_client.csv`, `train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv`.
Preprocessor state -> `artifacts/processed/`; round-0 prototype snapshot ->
`artifacts/processed/smoke/analysis/` (smoke) or `artifacts/analysis/` (full run);
manifests -> `artifacts/partitions/`.
## 14. Remaining assumptions and unresolved issues
1. **Participant rule** (`S_j`, equal weights) is a reproduction assumption (see §7).
2. `lambda = 1.0`, optimiser and schedule are unverified against the paper.
3. Prototype-alignment form and round-0 handling are inferred.
4. Per-client evaluation partitioning (val/test assigned to clients via Dirichlet) is a
project decision; the paper does not describe it.
5. No full-scale run has been timed; compute feasibility of a real sweep is unmeasured.
6. Whether the literal `1/M` reading differs from the `S_j` assumption is open and must
be tested with a deliberate control in the verification phase.