# PROTEAN Prototype Aggregation Study Research code for **server-side prototype aggregation in PROTEAN**, a federated intrusion-detection framework, under non-IID client data. The study investigates a beta-controlled, class-specific sample-support weighting of local class prototypes for **rare attack detection**. This repository is a finished experimental study: the planned federated grid is complete and audited, and the remaining work is thesis writing. Raw datasets are external and immutable; all scientific outputs are derived from audited, machine-readable artifacts. ## Research Question / Motivation Each client `i` forms a local prototype `C_ij` for class `j` from `N_ij` local training samples. `N_ij` can differ by orders of magnitude across clients (non-IID), yet a simple server-side mean gives every contributing client the same weight regardless of support. This study asks whether, and under what conditions, weighting each client's contribution by its class-specific sample support changes the global prototype and rare-class detection behavior. `N_ij` is **class-specific sample support, not prototype quality**. A client with few samples is not assumed to have a worse prototype; low-support clients may carry variation unavailable at high-support clients. ## Proposed Aggregation ``` Cbar_j(beta) = ( sum_{i in S_j} N_ij^beta * C_ij ) / ( sum_{i in S_j} N_ij^beta ) S_j = { i | N_ij > 0 and a valid local prototype C_ij exists } beta in {0.00, 0.25, 0.50, 0.75, 1.00} ``` - `beta = 0`: equal weighting over the contributors `S_j`. - `beta = 1`: contribution proportional to class-specific support. - `0 < beta < 1`: intermediate weighting. `S_j` (participant set and denominator rule) is a **documented reproduction assumption**: no verified official PROTEAN code is available. Beta selection, all comparisons, and the `beta = 0` equivalence result are conditional on this shared rule. The modification is narrowly scoped to the server-side aggregation step; client training, model architecture, and loss are unchanged. ## Experimental Protocol | Item | Value | |---|---| | Datasets | X-IIoTID (`class2`, benign `Normal`, K=9); 5G-NIDD (`Attack Type`, benign `Benign`, K=9) | | Dirichlet alpha | 0.25, 0.50, 0.75 | | Seeds | 0, 1, 2 (project-defined) | | Clients | 10 | | Rounds | 10 | | Local epochs | 3 | | Batch size | 256 | | Optimizer / learning rate | Adam / 1e-3 | | FedProx mu | 0.1 | | Prototype alignment lambda | 1.0 | | Beta candidates | 0.00, 0.25, 0.50, 0.75, 1.00 | | Modes | `original_protean`, `support_weighted` | | Splits | train / validation / test = 70 / 10 / 20 (stratified) | | Selection | pooled validation per-client-mean rare-class macro-F1 across seeds | | Test | used only after the beta is fixed | Each `(dataset, alpha, seed)` uses one deterministic frozen partition; all compared configurations within it share the same manifest, split, class mapping, and initialization. ## Completed Experiment Grid ``` 2 datasets x 3 alpha conditions x 3 seeds x (1 original PROTEAN + 5 beta candidates) = 108 full federated runs ``` All 108 planned runs completed and audited. Smoke runs are excluded and are not scientific experiments. Realized non-IID characteristics (not Dirichlet alpha alone) are recorded per condition. ## Final Pooled Beta Selections Validation-only, pooled across clients and seeds; best among the tested candidates. | Dataset | alpha=0.25 | alpha=0.50 | alpha=0.75 | |---|---|---|---| | X-IIoTID | 0.00 | 0.75 | 0.50 | | 5G-NIDD | 0.00 | 1.00 | 0.50 | Caveats: these are the best among the five tested candidates, **not global optima**; no universal or monotonic beta relationship is established; only two datasets and three alpha levels were evaluated. ## Main Findings 1. Realized support imbalance and direct prototype displacement from `beta = 0` **co-vary descriptively** within both datasets (larger imbalance accompanies larger displacement). 2. For a fixed snapshot, prototype displacement increases **monotonically** with beta in every condition. 3. Rare-class detection response is **non-monotonic** and condition-dependent; the direct aggregation effect and the end-to-end training effect are distinct. 4. The selected beta **changes across conditions and datasets**. 5. **No universal beta** was found. 6. Both datasets selected **beta = 0 at alpha = 0.25** (strongest realized imbalance, largest displacement); this is a descriptive pattern, not a rule. 7. Under the shared `S_j` reproduction rule, `beta = 0` is **semantically** the equal mean over `S_j`, but downstream bitwise identity with `original_protean` is not guaranteed because of floating-point operation ordering. Not supported by the evidence (must not be claimed): support weighting improves rare detection in general; the selected beta beats original PROTEAN; alpha causally determines beta; `N_ij` measures prototype quality; larger displacement means better detection; a global optimum exists; results generalize beyond the evaluated scope. ## Repository Structure ``` src/protean_research/ data, models, aggregation, federated, evaluation, utils scripts/ training, partition, and analysis scripts configs/ datasets, experiments, model configuration docs/research/ protocol, per-condition reports, cross-alpha/cross-dataset analysis, audits docs/baseline/ PROTEAN baseline investigation and validation docs/thesis/ Chapter 4 blueprint and claims register docs/handoff/ session handoff (start here for a new session) artifacts/analysis/ machine-readable analysis outputs (committed) artifacts/figures/final/ thesis-ready figures artifacts/partitions/ deterministic frozen Dirichlet partition manifests results/ local scientific run outputs (intentionally not committed) ``` `results/` holds local run outputs and is git-ignored; committed scientific outputs are the artifact tables and figures under `artifacts/`. Raw datasets are never copied into the repository. ## Key Scripts | Script | Role | |---|---| | `scripts/run_baseline.py` | full baseline / support-weighted training runner | | `scripts/generate_partitions.py` | deterministic frozen Dirichlet manifests | | `scripts/analyze_multiseed_beta.py` | pooled validation beta selection | | `scripts/analyze_partition_heterogeneity.py` | realized non-IID characterization | | `scripts/analyze_prototype_effect.py` | fixed-snapshot prototype displacement | | `scripts/analyze_cross_alpha.py` | cross-alpha synthesis | | `scripts/verify_beta0_equivalence.py` | beta=0 vs original semantic/numerical audit | | `scripts/validate_sj_participant_rule.py` | `S_j` reproduction-assumption analysis | ## Important Research Documents - `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md` — final six-condition synthesis - `docs/research/CROSS_DATASET_BETA_ANALYSIS.md` — cross-dataset comparison - `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md` — X-IIoTID cross-alpha analysis - `docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md` — 5G-NIDD cross-alpha analysis - `docs/research/EXPERIMENT_PROTOCOL.md` — frozen protocol - `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md` — Chapter 4 blueprint - `docs/thesis/CLAIMS_REGISTER.md` — claim / evidence / status register - `docs/handoff/SESSION_HANDOFF.md` — current state and next steps ## Reproducibility Notes - Raw datasets are external and immutable; they are never modified, renamed, or committed. Their SHA-256 hashes are recorded in the audit documents. - Frozen partition manifests are generated once and reused; a run that regenerates a fresh Dirichlet split is a protocol violation. - Paired seeds, partitions, class mappings, and hyperparameters are shared across all compared configurations within a condition. - Beta selection uses validation only; test metrics are read only after the beta is fixed. - Analysis outputs are machine-readable (CSV/JSON), and figures load from those artifacts. - `src/`, model, and aggregation code were frozen during the final experiments (no training tuning driven by test results). ## Limitations - The `S_j` participant/denominator rule remains a reproduction assumption, materially exercised in a substantial fraction of class-condition cells. - Two datasets, three alpha values, three project-defined seeds, 10 clients, one architecture. - The beta grid is discrete; intermediate values were not tested. - No inferential trend testing (three seeds are underpowered for it). - `beta = 0` is not bitwise identical to `original_protean` (floating-point ordering). - `N_ij` is sample support, not prototype quality; prototype displacement is a representation metric, not detection quality. ## Status - Experimental grid: COMPLETE (108 full runs) - Audit status: PASS (two conditions with documentation-only minor corrections) - Final synthesis: COMPLETE - Additional training planned: NO ## Quickstart ```bash # unit + integration tests (expected: 75 passed) .venv/bin/python -m pytest # generate / verify deterministic partition manifests (full data) .venv/bin/python scripts/generate_partitions.py # pooled multi-seed validation analysis (read-only, no retraining) .venv/bin/python scripts/analyze_multiseed_beta.py --dataset 5gnidd --alpha 0.25 --seeds 0 1 2 # one full paired training run (CPU, long) .venv/bin/python scripts/run_baseline.py --dataset 5gnidd --alpha 0.25 --seed 1 \ --clients 10 --rounds 10 --local-epochs 3 \ --aggregation-mode support_weighted --beta 1.0 ``` See `docs/handoff/SESSION_HANDOFF.md` before making methodological changes. The entire `planningawal/` directory is immutable research source material.