thesis-prototype-agregation/README.md

# PROTEAN Prototype Aggregation Study

Research code for **server-side prototype aggregation in PROTEAN**, a federated intrusion-detection
framework, under non-IID client data. The study investigates a beta-controlled, class-specific
sample-support weighting of local class prototypes for **rare attack detection**.

This repository is a finished experimental study: the planned federated grid is complete and
audited, and the remaining work is thesis writing. Raw datasets are external and immutable; all
scientific outputs are derived from audited, machine-readable artifacts.

## Research Question / Motivation

Each client `i` forms a local prototype `C_ij` for class `j` from `N_ij` local training samples.
`N_ij` can differ by orders of magnitude across clients (non-IID), yet a simple server-side mean
gives every contributing client the same weight regardless of support. This study asks whether,
and under what conditions, weighting each client's contribution by its class-specific sample
support changes the global prototype and rare-class detection behavior.

`N_ij` is **class-specific sample support, not prototype quality**. A client with few samples is
not assumed to have a worse prototype; low-support clients may carry variation unavailable at
high-support clients.

## Proposed Aggregation

```
Cbar_j(beta) = ( sum_{i in S_j} N_ij^beta * C_ij ) / ( sum_{i in S_j} N_ij^beta )
S_j = { i | N_ij > 0 and a valid local prototype C_ij exists }
beta in {0.00, 0.25, 0.50, 0.75, 1.00}
```

- `beta = 0`: equal weighting over the contributors `S_j`.
- `beta = 1`: contribution proportional to class-specific support.
- `0 < beta < 1`: intermediate weighting.

`S_j` (participant set and denominator rule) is a **documented reproduction assumption**: no
verified official PROTEAN code is available. Beta selection, all comparisons, and the `beta = 0`
equivalence result are conditional on this shared rule. The modification is narrowly scoped to
the server-side aggregation step; client training, model architecture, and loss are unchanged.

## Experimental Protocol

| Item | Value |
|---|---|
| Datasets | X-IIoTID (`class2`, benign `Normal`, K=9); 5G-NIDD (`Attack Type`, benign `Benign`, K=9) |
| Dirichlet alpha | 0.25, 0.50, 0.75 |
| Seeds | 0, 1, 2 (project-defined) |
| Clients | 10 |
| Rounds | 10 |
| Local epochs | 3 |
| Batch size | 256 |
| Optimizer / learning rate | Adam / 1e-3 |
| FedProx mu | 0.1 |
| Prototype alignment lambda | 1.0 |
| Beta candidates | 0.00, 0.25, 0.50, 0.75, 1.00 |
| Modes | `original_protean`, `support_weighted` |
| Splits | train / validation / test = 70 / 10 / 20 (stratified) |
| Selection | pooled validation per-client-mean rare-class macro-F1 across seeds |
| Test | used only after the beta is fixed |

Each `(dataset, alpha, seed)` uses one deterministic frozen partition; all compared
configurations within it share the same manifest, split, class mapping, and initialization.

## Completed Experiment Grid

```
2 datasets x 3 alpha conditions x 3 seeds x (1 original PROTEAN + 5 beta candidates)
= 108 full federated runs
```

All 108 planned runs completed and audited. Smoke runs are excluded and are not scientific
experiments. Realized non-IID characteristics (not Dirichlet alpha alone) are recorded per
condition.

## Final Pooled Beta Selections

Validation-only, pooled across clients and seeds; best among the tested candidates.

| Dataset | alpha=0.25 | alpha=0.50 | alpha=0.75 |
|---|---|---|---|
| X-IIoTID | 0.00 | 0.75 | 0.50 |
| 5G-NIDD  | 0.00 | 1.00 | 0.50 |

Caveats: these are the best among the five tested candidates, **not global optima**; no universal
or monotonic beta relationship is established; only two datasets and three alpha levels were
evaluated.

## Main Findings

1. Realized support imbalance and direct prototype displacement from `beta = 0` **co-vary
   descriptively** within both datasets (larger imbalance accompanies larger displacement).
2. For a fixed snapshot, prototype displacement increases **monotonically** with beta in every
   condition.
3. Rare-class detection response is **non-monotonic** and condition-dependent; the direct
   aggregation effect and the end-to-end training effect are distinct.
4. The selected beta **changes across conditions and datasets**.
5. **No universal beta** was found.
6. Both datasets selected **beta = 0 at alpha = 0.25** (strongest realized imbalance, largest
   displacement); this is a descriptive pattern, not a rule.
7. Under the shared `S_j` reproduction rule, `beta = 0` is **semantically** the equal mean over
   `S_j`, but downstream bitwise identity with `original_protean` is not guaranteed because of
   floating-point operation ordering.

Not supported by the evidence (must not be claimed): support weighting improves rare detection
in general; the selected beta beats original PROTEAN; alpha causally determines beta; `N_ij`
measures prototype quality; larger displacement means better detection; a global optimum exists;
results generalize beyond the evaluated scope.

## Repository Structure

```
src/protean_research/   data, models, aggregation, federated, evaluation, utils
scripts/                training, partition, and analysis scripts
configs/                datasets, experiments, model configuration
docs/research/          protocol, per-condition reports, cross-alpha/cross-dataset analysis, audits
docs/baseline/          PROTEAN baseline investigation and validation
docs/thesis/            Chapter 4 blueprint and claims register
docs/handoff/           session handoff (start here for a new session)
artifacts/analysis/     machine-readable analysis outputs (committed)
artifacts/figures/final/ thesis-ready figures
artifacts/partitions/   deterministic frozen Dirichlet partition manifests
results/                local scientific run outputs (intentionally not committed)
```

`results/` holds local run outputs and is git-ignored; committed scientific outputs are the
artifact tables and figures under `artifacts/`. Raw datasets are never copied into the repository.

## Key Scripts

| Script | Role |
|---|---|
| `scripts/run_baseline.py` | full baseline / support-weighted training runner |
| `scripts/generate_partitions.py` | deterministic frozen Dirichlet manifests |
| `scripts/analyze_multiseed_beta.py` | pooled validation beta selection |
| `scripts/analyze_partition_heterogeneity.py` | realized non-IID characterization |
| `scripts/analyze_prototype_effect.py` | fixed-snapshot prototype displacement |
| `scripts/analyze_cross_alpha.py` | cross-alpha synthesis |
| `scripts/verify_beta0_equivalence.py` | beta=0 vs original semantic/numerical audit |
| `scripts/validate_sj_participant_rule.py` | `S_j` reproduction-assumption analysis |

## Important Research Documents

- `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md` — final six-condition synthesis
- `docs/research/CROSS_DATASET_BETA_ANALYSIS.md` — cross-dataset comparison
- `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md` — X-IIoTID cross-alpha analysis
- `docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md` — 5G-NIDD cross-alpha analysis
- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen protocol
- `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md` — Chapter 4 blueprint
- `docs/thesis/CLAIMS_REGISTER.md` — claim / evidence / status register
- `docs/handoff/SESSION_HANDOFF.md` — current state and next steps

## Reproducibility Notes

- Raw datasets are external and immutable; they are never modified, renamed, or committed.
  Their SHA-256 hashes are recorded in the audit documents.
- Frozen partition manifests are generated once and reused; a run that regenerates a fresh
  Dirichlet split is a protocol violation.
- Paired seeds, partitions, class mappings, and hyperparameters are shared across all compared
  configurations within a condition.
- Beta selection uses validation only; test metrics are read only after the beta is fixed.
- Analysis outputs are machine-readable (CSV/JSON), and figures load from those artifacts.
- `src/`, model, and aggregation code were frozen during the final experiments (no training
  tuning driven by test results).

## Limitations

- The `S_j` participant/denominator rule remains a reproduction assumption, materially exercised
  in a substantial fraction of class-condition cells.
- Two datasets, three alpha values, three project-defined seeds, 10 clients, one architecture.
- The beta grid is discrete; intermediate values were not tested.
- No inferential trend testing (three seeds are underpowered for it).
- `beta = 0` is not bitwise identical to `original_protean` (floating-point ordering).
- `N_ij` is sample support, not prototype quality; prototype displacement is a representation
  metric, not detection quality.

## Status

- Experimental grid: COMPLETE (108 full runs)
- Audit status: PASS (two conditions with documentation-only minor corrections)
- Final synthesis: COMPLETE
- Additional training planned: NO

## Quickstart

```bash
# unit + integration tests (expected: 75 passed)
.venv/bin/python -m pytest

# generate / verify deterministic partition manifests (full data)
.venv/bin/python scripts/generate_partitions.py

# pooled multi-seed validation analysis (read-only, no retraining)
.venv/bin/python scripts/analyze_multiseed_beta.py --dataset 5gnidd --alpha 0.25 --seeds 0 1 2

# one full paired training run (CPU, long)
.venv/bin/python scripts/run_baseline.py --dataset 5gnidd --alpha 0.25 --seed 1 \
    --clients 10 --rounds 10 --local-epochs 3 \
    --aggregation-mode support_weighted --beta 1.0
```

See `docs/handoff/SESSION_HANDOFF.md` before making methodological changes. The entire
`planningawal/` directory is immutable research source material.