thesis-prototype-agregation/README.md

PROTEAN Prototype Aggregation Study

Research code for server-side prototype aggregation in PROTEAN, a federated intrusion-detection framework, under non-IID client data. The study investigates a beta-controlled, class-specific sample-support weighting of local class prototypes for rare attack detection.

This repository is a finished experimental study: the planned federated grid is complete and audited, and the remaining work is thesis writing. Raw datasets are external and immutable; all scientific outputs are derived from audited, machine-readable artifacts.

Research Question / Motivation

Each client i forms a local prototype C_ij for class j from N_ij local training samples. N_ij can differ by orders of magnitude across clients (non-IID), yet a simple server-side mean gives every contributing client the same weight regardless of support. This study asks whether, and under what conditions, weighting each client's contribution by its class-specific sample support changes the global prototype and rare-class detection behavior.

N_ij is class-specific sample support, not prototype quality. A client with few samples is not assumed to have a worse prototype; low-support clients may carry variation unavailable at high-support clients.

Proposed Aggregation

Cbar_j(beta) = ( sum_{i in S_j} N_ij^beta * C_ij ) / ( sum_{i in S_j} N_ij^beta )
S_j = { i | N_ij > 0 and a valid local prototype C_ij exists }
beta in {0.00, 0.25, 0.50, 0.75, 1.00}

• beta = 0: equal weighting over the contributors S_j.
• beta = 1: contribution proportional to class-specific support.
• 0 < beta < 1: intermediate weighting.

S_j (participant set and denominator rule) is a documented reproduction assumption: no verified official PROTEAN code is available. Beta selection, all comparisons, and the beta = 0 equivalence result are conditional on this shared rule. The modification is narrowly scoped to the server-side aggregation step; client training, model architecture, and loss are unchanged.

Experimental Protocol

Item	Value
Datasets	X-IIoTID (class2, benign Normal, K=9); 5G-NIDD (Attack Type, benign Benign, K=9)
Dirichlet alpha	0.25, 0.50, 0.75
Seeds	0, 1, 2 (project-defined)
Clients	10
Rounds	10
Local epochs	3
Batch size	256
Optimizer / learning rate	Adam / 1e-3
FedProx mu	0.1
Prototype alignment lambda	1.0
Beta candidates	0.00, 0.25, 0.50, 0.75, 1.00
Modes	original_protean, support_weighted
Splits	train / validation / test = 70 / 10 / 20 (stratified)
Selection	pooled validation per-client-mean rare-class macro-F1 across seeds
Test	used only after the beta is fixed

Each (dataset, alpha, seed) uses one deterministic frozen partition; all compared configurations within it share the same manifest, split, class mapping, and initialization.

Completed Experiment Grid

2 datasets x 3 alpha conditions x 3 seeds x (1 original PROTEAN + 5 beta candidates)
= 108 full federated runs

All 108 planned runs completed and audited. Smoke runs are excluded and are not scientific experiments. Realized non-IID characteristics (not Dirichlet alpha alone) are recorded per condition.

Final Pooled Beta Selections

Validation-only, pooled across clients and seeds; best among the tested candidates.

Dataset	alpha=0.25	alpha=0.50	alpha=0.75
X-IIoTID	0.00	0.75	0.50
5G-NIDD	0.00	1.00	0.50

Caveats: these are the best among the five tested candidates, not global optima; no universal or monotonic beta relationship is established; only two datasets and three alpha levels were evaluated.

Main Findings

1. Realized support imbalance and direct prototype displacement from beta = 0 co-vary descriptively within both datasets (larger imbalance accompanies larger displacement).
2. For a fixed snapshot, prototype displacement increases monotonically with beta in every condition.
3. Rare-class detection response is non-monotonic and condition-dependent; the direct aggregation effect and the end-to-end training effect are distinct.
4. The selected beta changes across conditions and datasets.
5. No universal beta was found.
6. Both datasets selected beta = 0 at alpha = 0.25 (strongest realized imbalance, largest displacement); this is a descriptive pattern, not a rule.
7. Under the shared S_j reproduction rule, beta = 0 is semantically the equal mean over S_j, but downstream bitwise identity with original_protean is not guaranteed because of floating-point operation ordering.

Not supported by the evidence (must not be claimed): support weighting improves rare detection in general; the selected beta beats original PROTEAN; alpha causally determines beta; N_ij measures prototype quality; larger displacement means better detection; a global optimum exists; results generalize beyond the evaluated scope.

Repository Structure

src/protean_research/   data, models, aggregation, federated, evaluation, utils
scripts/                training, partition, and analysis scripts
configs/                datasets, experiments, model configuration
docs/research/          protocol, per-condition reports, cross-alpha/cross-dataset analysis, audits
docs/baseline/          PROTEAN baseline investigation and validation
docs/thesis/            Chapter 4 blueprint and claims register
docs/handoff/           session handoff (start here for a new session)
artifacts/analysis/     machine-readable analysis outputs (committed)
artifacts/figures/final/ thesis-ready figures
artifacts/partitions/   deterministic frozen Dirichlet partition manifests
results/                local scientific run outputs (intentionally not committed)

results/ holds local run outputs and is git-ignored; committed scientific outputs are the artifact tables and figures under artifacts/. Raw datasets are never copied into the repository.

Key Scripts

Script	Role
scripts/run_baseline.py	full baseline / support-weighted training runner
scripts/generate_partitions.py	deterministic frozen Dirichlet manifests
scripts/analyze_multiseed_beta.py	pooled validation beta selection
scripts/analyze_partition_heterogeneity.py	realized non-IID characterization
scripts/analyze_prototype_effect.py	fixed-snapshot prototype displacement
scripts/analyze_cross_alpha.py	cross-alpha synthesis
scripts/verify_beta0_equivalence.py	beta=0 vs original semantic/numerical audit
scripts/validate_sj_participant_rule.py	S_j reproduction-assumption analysis

Important Research Documents

• docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md — final six-condition synthesis
• docs/research/CROSS_DATASET_BETA_ANALYSIS.md — cross-dataset comparison
• docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md — X-IIoTID cross-alpha analysis
• docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md — 5G-NIDD cross-alpha analysis
• docs/research/EXPERIMENT_PROTOCOL.md — frozen protocol
• docs/thesis/CHAPTER4_RESULTS_OUTLINE.md — Chapter 4 blueprint
• docs/thesis/CLAIMS_REGISTER.md — claim / evidence / status register
• docs/handoff/SESSION_HANDOFF.md — current state and next steps

Reproducibility Notes

• Raw datasets are external and immutable; they are never modified, renamed, or committed. Their SHA-256 hashes are recorded in the audit documents.
• Frozen partition manifests are generated once and reused; a run that regenerates a fresh Dirichlet split is a protocol violation.
• Paired seeds, partitions, class mappings, and hyperparameters are shared across all compared configurations within a condition.
• Beta selection uses validation only; test metrics are read only after the beta is fixed.
• Analysis outputs are machine-readable (CSV/JSON), and figures load from those artifacts.
• src/, model, and aggregation code were frozen during the final experiments (no training tuning driven by test results).

Limitations

• The S_j participant/denominator rule remains a reproduction assumption, materially exercised in a substantial fraction of class-condition cells.
• Two datasets, three alpha values, three project-defined seeds, 10 clients, one architecture.
• The beta grid is discrete; intermediate values were not tested.
• No inferential trend testing (three seeds are underpowered for it).
• beta = 0 is not bitwise identical to original_protean (floating-point ordering).
• N_ij is sample support, not prototype quality; prototype displacement is a representation metric, not detection quality.

Status

• Experimental grid: COMPLETE (108 full runs)
• Audit status: PASS (two conditions with documentation-only minor corrections)
• Final synthesis: COMPLETE
• Additional training planned: NO

Quickstart

# unit + integration tests (expected: 75 passed)
.venv/bin/python -m pytest

# generate / verify deterministic partition manifests (full data)
.venv/bin/python scripts/generate_partitions.py

# pooled multi-seed validation analysis (read-only, no retraining)
.venv/bin/python scripts/analyze_multiseed_beta.py --dataset 5gnidd --alpha 0.25 --seeds 0 1 2

# one full paired training run (CPU, long)
.venv/bin/python scripts/run_baseline.py --dataset 5gnidd --alpha 0.25 --seed 1 \
    --clients 10 --rounds 10 --local-epochs 3 \
    --aggregation-mode support_weighted --beta 1.0

See docs/handoff/SESSION_HANDOFF.md before making methodological changes. The entire planningawal/ directory is immutable research source material.