From 8255393da9eb68378a6266765a69f6d3b6bcf1ce Mon Sep 17 00:00:00 2001 From: willtanoe Date: Fri, 2 Oct 2026 14:35:14 +0700 Subject: [PATCH] docs: refresh README for finalized research state --- README.md | 258 ++++++++++++++++++++++++++++++------------------------ 1 file changed, 142 insertions(+), 116 deletions(-) diff --git a/README.md b/README.md index 8873c18..14e25ea 100644 --- a/README.md +++ b/README.md @@ -1,25 +1,52 @@ # PROTEAN Prototype Aggregation Study -Research code for **server-side prototype aggregation in PROTEAN**, a federated intrusion-detection -framework, under non-IID client data. The study investigates a beta-controlled, class-specific -sample-support weighting of local class prototypes for **rare attack detection**. +Research code and audited artifacts for a master's thesis on **server-side prototype aggregation in +PROTEAN** (Chennoufi et al., 2025), a federated intrusion-detection framework, under non-IID client +data. The study investigates a **beta-controlled, class-specific sample-support weighting** of local +class prototypes for **rare attack detection**. -This repository is a finished experimental study: the planned federated grid is complete and -audited, and the remaining work is thesis writing. Raw datasets are external and immutable; all -scientific outputs are derived from audited, machine-readable artifacts. +The experimental study is **complete** (108 full runs, six non-IID conditions, both datasets) and the +thesis proposal Markdown is **finalized**. No further training is planned unless new academic +feedback explicitly requires it. -## Research Question / Motivation +## Final Research Title -Each client `i` forms a local prototype `C_ij` for class `j` from `N_ij` local training samples. -`N_ij` can differ by orders of magnitude across clients (non-IID), yet a simple server-side mean -gives every contributing client the same weight regardless of support. This study asks whether, -and under what conditions, weighting each client's contribution by its class-specific sample -support changes the global prototype and rare-class detection behavior. +**Optimasi Agregasi Prototype pada PROTEAN melalui Pembobotan Berbasis Dukungan Sampel Kelas untuk +Deteksi Serangan Langka pada Data Non-IID** -`N_ij` is **class-specific sample support, not prototype quality**. A client with few samples is -not assumed to have a worse prototype; low-support clients may carry variation unavailable at +*Optimizing Prototype Aggregation in PROTEAN through Class-Sample-Support Weighting for Rare Attack +Detection under Non-IID Data Distributions.* + +Here **"Optimasi" means empirical tuning/selection of `beta` among the predefined candidate values**, +not a global mathematical optimum and not a universal beta. The selected value is the best among the +tested candidates under the defined validation procedure for each condition. + +## Research Scope + +- **Base method:** PROTEAN, a federated intrusion-detection framework for non-IID data. +- **Focus:** the **server-side prototype aggregation** step only. Client training, model + architecture, loss, and inference are unchanged. +- **Proposed mechanism:** weight each contributing client's class prototype by `N_ij^beta`, where + `N_ij` is that client's class-specific training support and `beta` is a single tunable exponent + in `[0, 1]`. +- **Analysis chain:** support imbalance -> contribution weight -> prototype displacement -> + rare-class detection -> per-client effect. + +`N_ij` is **class-specific sample support, not prototype quality**. A client with few samples is not +assumed to have a worse prototype; low-support clients may carry variation unavailable at high-support clients. +## Why This Problem Matters + +Federated IDS data is naturally non-IID: each client observes different traffic and different attack +types. Rare attack classes appear in little of the data, so aggregate metrics can look good while +the model fails on them. PROTEAN builds a global prototype per class on the server and uses it both +for local prototype alignment and for nearest-prototype inference. Its reference aggregation gives +every contributing client equal weight, even though the local prototypes may be built from very +different numbers of samples. This study asks whether, and under which realized non-IID conditions, +weighting contributions by class sample support changes the global prototype and rare-class +detection behavior. + ## Proposed Aggregation ``` @@ -32,78 +59,56 @@ beta in {0.00, 0.25, 0.50, 0.75, 1.00} - `beta = 1`: contribution proportional to class-specific support. - `0 < beta < 1`: intermediate weighting. -`S_j` (participant set and denominator rule) is a **documented reproduction assumption**: no -verified official PROTEAN code is available. Beta selection, all comparisons, and the `beta = 0` -equivalence result are conditional on this shared rule. The modification is narrowly scoped to -the server-side aggregation step; client training, model architecture, and loss are unchanged. +`S_j` (participant set and denominator rule) is a **documented reproduction assumption**: no verified +official PROTEAN code is available. `beta = 0` is semantically the equal mean over `S_j`, but is not +guaranteed to be bitwise identical to the frozen `original_protean` baseline because of +floating-point operation ordering. -## Experimental Protocol +## Experimental Setup | Item | Value | |---|---| | Datasets | X-IIoTID (`class2`, benign `Normal`, K=9); 5G-NIDD (`Attack Type`, benign `Benign`, K=9) | -| Dirichlet alpha | 0.25, 0.50, 0.75 | +| Non-IID conditions | symmetric Dirichlet `alpha` = 0.75, 0.50, 0.25 | | Seeds | 0, 1, 2 (project-defined) | -| Clients | 10 | -| Rounds | 10 | -| Local epochs | 3 | -| Batch size | 256 | -| Optimizer / learning rate | Adam / 1e-3 | -| FedProx mu | 0.1 | -| Prototype alignment lambda | 1.0 | -| Beta candidates | 0.00, 0.25, 0.50, 0.75, 1.00 | -| Modes | `original_protean`, `support_weighted` | -| Splits | train / validation / test = 70 / 10 / 20 (stratified) | +| Clients / rounds / local epochs | 10 / 10 / 3 | +| Batch size / optimizer / learning rate | 256 / Adam / 1e-3 | +| FedProx `mu` / prototype-alignment `lambda` | 0.1 / 1.0 | +| Compared configurations | `original_protean` + `support_weighted` at beta 0, 0.25, 0.50, 0.75, 1.00 | +| Splits | stratified train / validation / test = 70 / 10 / 20 | | Selection | pooled validation per-client-mean rare-class macro-F1 across seeds | -| Test | used only after the beta is fixed | +| Test | read only after the selected beta is fixed | -Each `(dataset, alpha, seed)` uses one deterministic frozen partition; all compared -configurations within it share the same manifest, split, class mapping, and initialization. +Each `(dataset, alpha, seed)` uses one deterministic frozen partition; all compared configurations +within it share the same manifest, split, class mapping, and initialization. -## Completed Experiment Grid +## Experiment Status -``` -2 datasets x 3 alpha conditions x 3 seeds x (1 original PROTEAN + 5 beta candidates) -= 108 full federated runs -``` +- **Complete: 108 full runs** = 2 datasets × 3 alpha conditions × 3 seeds × + (1 original PROTEAN + 5 beta settings). +- All conditions are trained, audited, and synthesized. +- Smoke runs are excluded and are not scientific experiments. +- **No further training planned** unless new academic feedback requires it. -All 108 planned runs completed and audited. Smoke runs are excluded and are not scientific -experiments. Realized non-IID characteristics (not Dirichlet alpha alone) are recorded per -condition. +## Final Scientific Position -## Final Pooled Beta Selections +High-level, evidence-safe conclusions only (full tables live in the synthesis documents): -Validation-only, pooled across clients and seeds; best among the tested candidates. +- Realized support imbalance, missing client-class cells, and client-size ratio vary by condition; + Dirichlet `alpha` is a design knob, not the realized heterogeneity itself. +- Increasing `beta` changes the normalized contribution weights and therefore the position of each + global prototype; for a fixed snapshot, displacement grows with `beta`. +- The rare-class detection response is **condition-dependent and non-monotone** in `beta`; the direct + aggregation effect and the end-to-end training effect are distinct. +- **No universal beta exists.** The validation-selected beta differs across datasets and non-IID + conditions and is not monotone in `alpha`. +- Stronger support weighting can **help, have limited effect, or hurt** rare-class behavior, + depending on the condition, and can redistribute benefit across clients within one condition. -| Dataset | alpha=0.25 | alpha=0.50 | alpha=0.75 | -|---|---|---|---| -| X-IIoTID | 0.00 | 0.75 | 0.50 | -| 5G-NIDD | 0.00 | 1.00 | 0.50 | - -Caveats: these are the best among the five tested candidates, **not global optima**; no universal -or monotonic beta relationship is established; only two datasets and three alpha levels were -evaluated. - -## Main Findings - -1. Realized support imbalance and direct prototype displacement from `beta = 0` **co-vary - descriptively** within both datasets (larger imbalance accompanies larger displacement). -2. For a fixed snapshot, prototype displacement increases **monotonically** with beta in every - condition. -3. Rare-class detection response is **non-monotonic** and condition-dependent; the direct - aggregation effect and the end-to-end training effect are distinct. -4. The selected beta **changes across conditions and datasets**. -5. **No universal beta** was found. -6. Both datasets selected **beta = 0 at alpha = 0.25** (strongest realized imbalance, largest - displacement); this is a descriptive pattern, not a rule. -7. Under the shared `S_j` reproduction rule, `beta = 0` is **semantically** the equal mean over - `S_j`, but downstream bitwise identity with `original_protean` is not guaranteed because of - floating-point operation ordering. - -Not supported by the evidence (must not be claimed): support weighting improves rare detection -in general; the selected beta beats original PROTEAN; alpha causally determines beta; `N_ij` -measures prototype quality; larger displacement means better detection; a global optimum exists; -results generalize beyond the evaluated scope. +Not supported by the evidence (must not be claimed): support weighting always improves rare-class +detection; the selected beta universally beats `original_protean`; `alpha` causally determines beta; +`N_ij` measures prototype quality; larger prototype displacement means better detection; a global +optimum exists; results generalize beyond the two evaluated datasets. ## Repository Structure @@ -111,24 +116,23 @@ results generalize beyond the evaluated scope. src/protean_research/ data, models, aggregation, federated, evaluation, utils scripts/ training, partition, and analysis scripts configs/ datasets, experiments, model configuration -docs/research/ protocol, per-condition reports, cross-alpha/cross-dataset analysis, audits -docs/baseline/ PROTEAN baseline investigation and validation +docs/proposal/ thesis proposal and its internal audit aids +docs/research/ frozen protocol, per-condition reports, cross-alpha / cross-dataset synthesis docs/thesis/ Chapter 4 blueprint and claims register +docs/baseline/ PROTEAN baseline investigation and validation docs/handoff/ session handoff (start here for a new session) -artifacts/analysis/ machine-readable analysis outputs (committed) +planningawal/ immutable research specification (do not modify) +artifacts/analysis/ committed machine-readable analysis outputs artifacts/figures/final/ thesis-ready figures artifacts/partitions/ deterministic frozen Dirichlet partition manifests -results/ local scientific run outputs (intentionally not committed) +results/ local run outputs (git-ignored, intentionally not committed) ``` -`results/` holds local run outputs and is git-ignored; committed scientific outputs are the -artifact tables and figures under `artifacts/`. Raw datasets are never copied into the repository. - ## Key Scripts | Script | Role | |---|---| -| `scripts/run_baseline.py` | full baseline / support-weighted training runner | +| `scripts/run_baseline.py` | full baseline / support-weighted training runner (frozen; grid complete) | | `scripts/generate_partitions.py` | deterministic frozen Dirichlet manifests | | `scripts/analyze_multiseed_beta.py` | pooled validation beta selection | | `scripts/analyze_partition_heterogeneity.py` | realized non-IID characterization | @@ -137,47 +141,73 @@ artifact tables and figures under `artifacts/`. Raw datasets are never copied in | `scripts/verify_beta0_equivalence.py` | beta=0 vs original semantic/numerical audit | | `scripts/validate_sj_participant_rule.py` | `S_j` reproduction-assumption analysis | -## Important Research Documents +## Important Project Documents -- `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md` — final six-condition synthesis +**Proposal** + +- `docs/proposal/THESIS_PROPOSAL_DRAFT.md` — final Markdown proposal source for the current scope +- `docs/proposal/PROPOSAL_SOURCE_MAP.md` — claim-to-evidence map (internal) +- `docs/proposal/LITERATURE_MATRIX.md` — literature matrix (internal) +- `docs/proposal/REFERENCES_VERIFIED.md` — bibliography verification record (internal) + +**Research synthesis** + +- `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md` — authoritative six-condition synthesis - `docs/research/CROSS_DATASET_BETA_ANALYSIS.md` — cross-dataset comparison - `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md` — X-IIoTID cross-alpha analysis - `docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md` — 5G-NIDD cross-alpha analysis -- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen protocol +- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen reproduction protocol + +**Thesis support** + - `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md` — Chapter 4 blueprint - `docs/thesis/CLAIMS_REGISTER.md` — claim / evidence / status register -- `docs/handoff/SESSION_HANDOFF.md` — current state and next steps -## Reproducibility Notes +**Handoff and planning** -- Raw datasets are external and immutable; they are never modified, renamed, or committed. - Their SHA-256 hashes are recorded in the audit documents. -- Frozen partition manifests are generated once and reused; a run that regenerates a fresh - Dirichlet split is a protocol violation. -- Paired seeds, partitions, class mappings, and hyperparameters are shared across all compared - configurations within a condition. -- Beta selection uses validation only; test metrics are read only after the beta is fixed. -- Analysis outputs are machine-readable (CSV/JSON), and figures load from those artifacts. -- `src/`, model, and aggregation code were frozen during the final experiments (no training - tuning driven by test results). +- `docs/handoff/SESSION_HANDOFF.md` — current state and next steps (start here) +- `planningawal/PROTEAN_Revisi_Lurus.md` — immutable research specification + +> `planningawal/` is **immutable**. Read it, cite it, but never edit, rename, move, or add files to +> it. + +## Testing and Reproducibility + +```bash +# unit + integration tests (expected: 75 passed) +.venv/bin/python -m pytest +``` + +- **Paired comparison:** within each `(dataset, alpha, seed)`, every compared configuration reuses + the same partition, split, class mapping, and model initialization. +- **Frozen manifests:** partitions are generated once and reused; regenerating a fresh Dirichlet + split is a protocol violation. +- **Immutable raw data:** datasets live outside the repository and are never modified or committed. +- **Validation-only selection:** beta is chosen from validation data only; test data is never used + for any model-selection decision. +- **Machine-readable outputs:** analysis is stored as CSV/JSON under `artifacts/analysis/`, and the + final figures load from those artifacts. + +## Frozen / Do Not Modify + +Treat the following as finished unless there is an explicit, documented reason: + +- `planningawal/` — immutable research source. +- Raw datasets under `~/datasets/` — read-only, external, never copied into the repository. +- Completed scientific results and the final synthesis documents. +- Frozen experiment artifacts: partition manifests, analysis outputs, and final figures. +- `src/`, model, and aggregation code used for the 108-run grid. +- The finalized proposal Markdown. ## Limitations -- The `S_j` participant/denominator rule remains a reproduction assumption, materially exercised - in a substantial fraction of class-condition cells. +- The `S_j` participant/denominator rule remains a reproduction assumption and is materially + exercised in a substantial fraction of class-condition cells. - Two datasets, three alpha values, three project-defined seeds, 10 clients, one architecture. - The beta grid is discrete; intermediate values were not tested. - No inferential trend testing (three seeds are underpowered for it). - `beta = 0` is not bitwise identical to `original_protean` (floating-point ordering). -- `N_ij` is sample support, not prototype quality; prototype displacement is a representation - metric, not detection quality. - -## Status - -- Experimental grid: COMPLETE (108 full runs) -- Audit status: PASS (two conditions with documentation-only minor corrections) -- Final synthesis: COMPLETE -- Additional training planned: NO +- Prototype displacement is a representation metric, not detection quality. ## Quickstart @@ -185,17 +215,13 @@ artifact tables and figures under `artifacts/`. Raw datasets are never copied in # unit + integration tests (expected: 75 passed) .venv/bin/python -m pytest -# generate / verify deterministic partition manifests (full data) -.venv/bin/python scripts/generate_partitions.py - -# pooled multi-seed validation analysis (read-only, no retraining) +# read-only pooled validation analysis (no retraining required) .venv/bin/python scripts/analyze_multiseed_beta.py --dataset 5gnidd --alpha 0.25 --seeds 0 1 2 -# one full paired training run (CPU, long) -.venv/bin/python scripts/run_baseline.py --dataset 5gnidd --alpha 0.25 --seed 1 \ - --clients 10 --rounds 10 --local-epochs 3 \ - --aggregation-mode support_weighted --beta 1.0 +# read-only cross-alpha / cross-dataset synthesis +.venv/bin/python scripts/analyze_partition_heterogeneity.py --dataset 5gnidd --alphas 0.25 0.5 0.75 +.venv/bin/python scripts/analyze_cross_alpha.py --dataset 5gnidd --alphas 0.25 0.5 0.75 ``` -See `docs/handoff/SESSION_HANDOFF.md` before making methodological changes. The entire -`planningawal/` directory is immutable research source material. +See `docs/handoff/SESSION_HANDOFF.md` before making any methodological change. The next work is +academic writing and formatting, not new experiments.