diff --git a/README.md b/README.md index 2348c58..8873c18 100644 --- a/README.md +++ b/README.md @@ -1,76 +1,201 @@ -# perdalam-protean +# PROTEAN Prototype Aggregation Study -Research code for **server-side prototype aggregation in PROTEAN** using class-specific -sample support, for rare attack detection under non-IID federated intrusion detection. +Research code for **server-side prototype aggregation in PROTEAN**, a federated intrusion-detection +framework, under non-IID client data. The study investigates a beta-controlled, class-specific +sample-support weighting of local class prototypes for **rare attack detection**. -The planned study modifies only how the server combines local class prototypes. The -proposed support-weighted aggregation is: +This repository is a finished experimental study: the planned federated grid is complete and +audited, and the remaining work is thesis writing. Raw datasets are external and immutable; all +scientific outputs are derived from audited, machine-readable artifacts. + +## Research Question / Motivation + +Each client `i` forms a local prototype `C_ij` for class `j` from `N_ij` local training samples. +`N_ij` can differ by orders of magnitude across clients (non-IID), yet a simple server-side mean +gives every contributing client the same weight regardless of support. This study asks whether, +and under what conditions, weighting each client's contribution by its class-specific sample +support changes the global prototype and rare-class detection behavior. + +`N_ij` is **class-specific sample support, not prototype quality**. A client with few samples is +not assumed to have a worse prototype; low-support clients may carry variation unavailable at +high-support clients. + +## Proposed Aggregation ``` -Cbar_j(beta) = ( sum_{i in S_j} N_ij^beta C_ij ) / ( sum_{i in S_j} N_ij^beta ) +Cbar_j(beta) = ( sum_{i in S_j} N_ij^beta * C_ij ) / ( sum_{i in S_j} N_ij^beta ) +S_j = { i | N_ij > 0 and a valid local prototype C_ij exists } +beta in {0.00, 0.25, 0.50, 0.75, 1.00} ``` -with `S_j = { i | N_ij > 0 and C_ij exists }` and candidate -`beta in {0.0, 0.25, 0.5, 0.75, 1.0}`. +- `beta = 0`: equal weighting over the contributors `S_j`. +- `beta = 1`: contribution proportional to class-specific support. +- `0 < beta < 1`: intermediate weighting. -**Current checkpoint: baseline only.** The proposed beta-weighted aggregation and the -`equal_available` control are **not implemented** in this checkpoint; they are deferred to -the later beta/controls phase, which must not begin until the baseline is verified. See -`docs/handoff/SESSION_HANDOFF.md`. +`S_j` (participant set and denominator rule) is a **documented reproduction assumption**: no +verified official PROTEAN code is available. Beta selection, all comparisons, and the `beta = 0` +equivalence result are conditional on this shared rule. The modification is narrowly scoped to +the server-side aggregation step; client training, model architecture, and loss are unchanged. -## Immutable source +## Experimental Protocol -- `planningawal/PROTEAN_Revisi_Lurus.md` is the authoritative research specification and is - **read-only**. Do not edit, add, rename, move, or delete anything under `planningawal/`. +| Item | Value | +|---|---| +| Datasets | X-IIoTID (`class2`, benign `Normal`, K=9); 5G-NIDD (`Attack Type`, benign `Benign`, K=9) | +| Dirichlet alpha | 0.25, 0.50, 0.75 | +| Seeds | 0, 1, 2 (project-defined) | +| Clients | 10 | +| Rounds | 10 | +| Local epochs | 3 | +| Batch size | 256 | +| Optimizer / learning rate | Adam / 1e-3 | +| FedProx mu | 0.1 | +| Prototype alignment lambda | 1.0 | +| Beta candidates | 0.00, 0.25, 0.50, 0.75, 1.00 | +| Modes | `original_protean`, `support_weighted` | +| Splits | train / validation / test = 70 / 10 / 20 (stratified) | +| Selection | pooled validation per-client-mean rare-class macro-F1 across seeds | +| Test | used only after the beta is fixed | -## External raw datasets (read-only, never copied into the repo) +Each `(dataset, alpha, seed)` uses one deterministic frozen partition; all compared +configurations within it share the same manifest, split, class mapping, and initialization. -- `~/datasets/X-IIoTID/X-IIoTID-dataset.csv` -- `~/datasets/5G-NIDD/5G-NIDD.csv` - -Paths are configured in `configs/datasets/datasets.yaml` (with environment overrides) and are -never hardcoded in source. Generated artifacts go under `artifacts/`. - -## Layout +## Completed Experiment Grid ``` -docs/research/ spec interpretation, dataset inspection, implementation plan, protocol -docs/baseline/ PROTEAN baseline investigation + baseline implementation notes -docs/handoff/ session handoff (start here for a new session) -configs/ datasets, experiments, model -src/protean_research/ data, models, aggregation, federated, evaluation, utils (baseline) -scripts/ run_baseline.py, generate_partitions.py -tests/ focused unit + integration tests -artifacts/ results/ generated outputs (not source) +2 datasets x 3 alpha conditions x 3 seeds x (1 original PROTEAN + 5 beta candidates) += 108 full federated runs ``` +All 108 planned runs completed and audited. Smoke runs are excluded and are not scientific +experiments. Realized non-IID characteristics (not Dirichlet alpha alone) are recorded per +condition. + +## Final Pooled Beta Selections + +Validation-only, pooled across clients and seeds; best among the tested candidates. + +| Dataset | alpha=0.25 | alpha=0.50 | alpha=0.75 | +|---|---|---|---| +| X-IIoTID | 0.00 | 0.75 | 0.50 | +| 5G-NIDD | 0.00 | 1.00 | 0.50 | + +Caveats: these are the best among the five tested candidates, **not global optima**; no universal +or monotonic beta relationship is established; only two datasets and three alpha levels were +evaluated. + +## Main Findings + +1. Realized support imbalance and direct prototype displacement from `beta = 0` **co-vary + descriptively** within both datasets (larger imbalance accompanies larger displacement). +2. For a fixed snapshot, prototype displacement increases **monotonically** with beta in every + condition. +3. Rare-class detection response is **non-monotonic** and condition-dependent; the direct + aggregation effect and the end-to-end training effect are distinct. +4. The selected beta **changes across conditions and datasets**. +5. **No universal beta** was found. +6. Both datasets selected **beta = 0 at alpha = 0.25** (strongest realized imbalance, largest + displacement); this is a descriptive pattern, not a rule. +7. Under the shared `S_j` reproduction rule, `beta = 0` is **semantically** the equal mean over + `S_j`, but downstream bitwise identity with `original_protean` is not guaranteed because of + floating-point operation ordering. + +Not supported by the evidence (must not be claimed): support weighting improves rare detection +in general; the selected beta beats original PROTEAN; alpha causally determines beta; `N_ij` +measures prototype quality; larger displacement means better detection; a global optimum exists; +results generalize beyond the evaluated scope. + +## Repository Structure + +``` +src/protean_research/ data, models, aggregation, federated, evaluation, utils +scripts/ training, partition, and analysis scripts +configs/ datasets, experiments, model configuration +docs/research/ protocol, per-condition reports, cross-alpha/cross-dataset analysis, audits +docs/baseline/ PROTEAN baseline investigation and validation +docs/thesis/ Chapter 4 blueprint and claims register +docs/handoff/ session handoff (start here for a new session) +artifacts/analysis/ machine-readable analysis outputs (committed) +artifacts/figures/final/ thesis-ready figures +artifacts/partitions/ deterministic frozen Dirichlet partition manifests +results/ local scientific run outputs (intentionally not committed) +``` + +`results/` holds local run outputs and is git-ignored; committed scientific outputs are the +artifact tables and figures under `artifacts/`. Raw datasets are never copied into the repository. + +## Key Scripts + +| Script | Role | +|---|---| +| `scripts/run_baseline.py` | full baseline / support-weighted training runner | +| `scripts/generate_partitions.py` | deterministic frozen Dirichlet manifests | +| `scripts/analyze_multiseed_beta.py` | pooled validation beta selection | +| `scripts/analyze_partition_heterogeneity.py` | realized non-IID characterization | +| `scripts/analyze_prototype_effect.py` | fixed-snapshot prototype displacement | +| `scripts/analyze_cross_alpha.py` | cross-alpha synthesis | +| `scripts/verify_beta0_equivalence.py` | beta=0 vs original semantic/numerical audit | +| `scripts/validate_sj_participant_rule.py` | `S_j` reproduction-assumption analysis | + +## Important Research Documents + +- `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md` — final six-condition synthesis +- `docs/research/CROSS_DATASET_BETA_ANALYSIS.md` — cross-dataset comparison +- `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md` — X-IIoTID cross-alpha analysis +- `docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md` — 5G-NIDD cross-alpha analysis +- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen protocol +- `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md` — Chapter 4 blueprint +- `docs/thesis/CLAIMS_REGISTER.md` — claim / evidence / status register +- `docs/handoff/SESSION_HANDOFF.md` — current state and next steps + +## Reproducibility Notes + +- Raw datasets are external and immutable; they are never modified, renamed, or committed. + Their SHA-256 hashes are recorded in the audit documents. +- Frozen partition manifests are generated once and reused; a run that regenerates a fresh + Dirichlet split is a protocol violation. +- Paired seeds, partitions, class mappings, and hyperparameters are shared across all compared + configurations within a condition. +- Beta selection uses validation only; test metrics are read only after the beta is fixed. +- Analysis outputs are machine-readable (CSV/JSON), and figures load from those artifacts. +- `src/`, model, and aggregation code were frozen during the final experiments (no training + tuning driven by test results). + +## Limitations + +- The `S_j` participant/denominator rule remains a reproduction assumption, materially exercised + in a substantial fraction of class-condition cells. +- Two datasets, three alpha values, three project-defined seeds, 10 clients, one architecture. +- The beta grid is discrete; intermediate values were not tested. +- No inferential trend testing (three seeds are underpowered for it). +- `beta = 0` is not bitwise identical to `original_protean` (floating-point ordering). +- `N_ij` is sample support, not prototype quality; prototype displacement is a representation + metric, not detection quality. + ## Status -The minimum reproducible **baseline** is implemented and smoke-tested. No full-scale -training and no beta sweep have been run. The original PROTEAN baseline has **no public -official code**, so it is reproduced from the paper/thesis with explicitly documented -reproduction assumptions (`docs/baseline/BASELINE_IMPLEMENTATION.md`). - -Read `docs/handoff/SESSION_HANDOFF.md` first, then -`planningawal/PROTEAN_Revisi_Lurus.md`. +- Experimental grid: COMPLETE (108 full runs) +- Audit status: PASS (two conditions with documentation-only minor corrections) +- Final synthesis: COMPLETE +- Additional training planned: NO ## Quickstart ```bash -# inside the repo, with the project virtualenv (numpy, pandas, torch, pyyaml, pytest) -.venv/bin/python -m pytest -q +# unit + integration tests (expected: 75 passed) +.venv/bin/python -m pytest -# baseline smoke test (writes only to ignored directories) -.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.5 --seed 0 \ - --max-rows 3000 --sample-from 60000 --clients 4 --rounds 2 --local-epochs 1 - -# generate + verify all deterministic partition manifests (full data) +# generate / verify deterministic partition manifests (full data) .venv/bin/python scripts/generate_partitions.py + +# pooled multi-seed validation analysis (read-only, no retraining) +.venv/bin/python scripts/analyze_multiseed_beta.py --dataset 5gnidd --alpha 0.25 --seeds 0 1 2 + +# one full paired training run (CPU, long) +.venv/bin/python scripts/run_baseline.py --dataset 5gnidd --alpha 0.25 --seed 1 \ + --clients 10 --rounds 10 --local-epochs 3 \ + --aggregation-mode support_weighted --beta 1.0 ``` -The baseline aggregation is dependency-free and testable without pandas/torch: - -```python -from protean_research.aggregation import aggregate_prototypes -aggregate_prototypes([[0.0, 2.0], [2.0, 0.0]]) # equal mean over the contributor set S_j -``` +See `docs/handoff/SESSION_HANDOFF.md` before making methodological changes. The entire +`planningawal/` directory is immutable research source material.