228 lines
11 KiB
Markdown
228 lines
11 KiB
Markdown
# PROTEAN Prototype Aggregation Study
|
||
|
||
Research code and audited artifacts for a master's thesis on **server-side prototype aggregation in
|
||
PROTEAN** (Chennoufi et al., 2025), a federated intrusion-detection framework, under non-IID client
|
||
data. The study investigates a **beta-controlled, class-specific sample-support weighting** of local
|
||
class prototypes for **rare attack detection**.
|
||
|
||
The experimental study is **complete** (108 full runs, six non-IID conditions, both datasets) and the
|
||
thesis proposal Markdown is **finalized**. No further training is planned unless new academic
|
||
feedback explicitly requires it.
|
||
|
||
## Final Research Title
|
||
|
||
**Optimasi Agregasi Prototype pada PROTEAN melalui Pembobotan Berbasis Dukungan Sampel Kelas untuk
|
||
Deteksi Serangan Langka pada Data Non-IID**
|
||
|
||
*Optimizing Prototype Aggregation in PROTEAN through Class-Sample-Support Weighting for Rare Attack
|
||
Detection under Non-IID Data Distributions.*
|
||
|
||
Here **"Optimasi" means empirical tuning/selection of `beta` among the predefined candidate values**,
|
||
not a global mathematical optimum and not a universal beta. The selected value is the best among the
|
||
tested candidates under the defined validation procedure for each condition.
|
||
|
||
## Research Scope
|
||
|
||
- **Base method:** PROTEAN, a federated intrusion-detection framework for non-IID data.
|
||
- **Focus:** the **server-side prototype aggregation** step only. Client training, model
|
||
architecture, loss, and inference are unchanged.
|
||
- **Proposed mechanism:** weight each contributing client's class prototype by `N_ij^beta`, where
|
||
`N_ij` is that client's class-specific training support and `beta` is a single tunable exponent
|
||
in `[0, 1]`.
|
||
- **Analysis chain:** support imbalance -> contribution weight -> prototype displacement ->
|
||
rare-class detection -> per-client effect.
|
||
|
||
`N_ij` is **class-specific sample support, not prototype quality**. A client with few samples is not
|
||
assumed to have a worse prototype; low-support clients may carry variation unavailable at
|
||
high-support clients.
|
||
|
||
## Why This Problem Matters
|
||
|
||
Federated IDS data is naturally non-IID: each client observes different traffic and different attack
|
||
types. Rare attack classes appear in little of the data, so aggregate metrics can look good while
|
||
the model fails on them. PROTEAN builds a global prototype per class on the server and uses it both
|
||
for local prototype alignment and for nearest-prototype inference. Its reference aggregation gives
|
||
every contributing client equal weight, even though the local prototypes may be built from very
|
||
different numbers of samples. This study asks whether, and under which realized non-IID conditions,
|
||
weighting contributions by class sample support changes the global prototype and rare-class
|
||
detection behavior.
|
||
|
||
## Proposed Aggregation
|
||
|
||
```
|
||
Cbar_j(beta) = ( sum_{i in S_j} N_ij^beta * C_ij ) / ( sum_{i in S_j} N_ij^beta )
|
||
S_j = { i | N_ij > 0 and a valid local prototype C_ij exists }
|
||
beta in {0.00, 0.25, 0.50, 0.75, 1.00}
|
||
```
|
||
|
||
- `beta = 0`: equal weighting over the contributors `S_j`.
|
||
- `beta = 1`: contribution proportional to class-specific support.
|
||
- `0 < beta < 1`: intermediate weighting.
|
||
|
||
`S_j` (participant set and denominator rule) is a **documented reproduction assumption**: no verified
|
||
official PROTEAN code is available. `beta = 0` is semantically the equal mean over `S_j`, but is not
|
||
guaranteed to be bitwise identical to the frozen `original_protean` baseline because of
|
||
floating-point operation ordering.
|
||
|
||
## Experimental Setup
|
||
|
||
| Item | Value |
|
||
|---|---|
|
||
| Datasets | X-IIoTID (`class2`, benign `Normal`, K=9); 5G-NIDD (`Attack Type`, benign `Benign`, K=9) |
|
||
| Non-IID conditions | symmetric Dirichlet `alpha` = 0.75, 0.50, 0.25 |
|
||
| Seeds | 0, 1, 2 (project-defined) |
|
||
| Clients / rounds / local epochs | 10 / 10 / 3 |
|
||
| Batch size / optimizer / learning rate | 256 / Adam / 1e-3 |
|
||
| FedProx `mu` / prototype-alignment `lambda` | 0.1 / 1.0 |
|
||
| Compared configurations | `original_protean` + `support_weighted` at beta 0, 0.25, 0.50, 0.75, 1.00 |
|
||
| Splits | stratified train / validation / test = 70 / 10 / 20 |
|
||
| Selection | pooled validation per-client-mean rare-class macro-F1 across seeds |
|
||
| Test | read only after the selected beta is fixed |
|
||
|
||
Each `(dataset, alpha, seed)` uses one deterministic frozen partition; all compared configurations
|
||
within it share the same manifest, split, class mapping, and initialization.
|
||
|
||
## Experiment Status
|
||
|
||
- **Complete: 108 full runs** = 2 datasets × 3 alpha conditions × 3 seeds ×
|
||
(1 original PROTEAN + 5 beta settings).
|
||
- All conditions are trained, audited, and synthesized.
|
||
- Smoke runs are excluded and are not scientific experiments.
|
||
- **No further training planned** unless new academic feedback requires it.
|
||
|
||
## Final Scientific Position
|
||
|
||
High-level, evidence-safe conclusions only (full tables live in the synthesis documents):
|
||
|
||
- Realized support imbalance, missing client-class cells, and client-size ratio vary by condition;
|
||
Dirichlet `alpha` is a design knob, not the realized heterogeneity itself.
|
||
- Increasing `beta` changes the normalized contribution weights and therefore the position of each
|
||
global prototype; for a fixed snapshot, displacement grows with `beta`.
|
||
- The rare-class detection response is **condition-dependent and non-monotone** in `beta`; the direct
|
||
aggregation effect and the end-to-end training effect are distinct.
|
||
- **No universal beta exists.** The validation-selected beta differs across datasets and non-IID
|
||
conditions and is not monotone in `alpha`.
|
||
- Stronger support weighting can **help, have limited effect, or hurt** rare-class behavior,
|
||
depending on the condition, and can redistribute benefit across clients within one condition.
|
||
|
||
Not supported by the evidence (must not be claimed): support weighting always improves rare-class
|
||
detection; the selected beta universally beats `original_protean`; `alpha` causally determines beta;
|
||
`N_ij` measures prototype quality; larger prototype displacement means better detection; a global
|
||
optimum exists; results generalize beyond the two evaluated datasets.
|
||
|
||
## Repository Structure
|
||
|
||
```
|
||
src/protean_research/ data, models, aggregation, federated, evaluation, utils
|
||
scripts/ training, partition, and analysis scripts
|
||
configs/ datasets, experiments, model configuration
|
||
docs/proposal/ thesis proposal and its internal audit aids
|
||
docs/research/ frozen protocol, per-condition reports, cross-alpha / cross-dataset synthesis
|
||
docs/thesis/ Chapter 4 blueprint and claims register
|
||
docs/baseline/ PROTEAN baseline investigation and validation
|
||
docs/handoff/ session handoff (start here for a new session)
|
||
planningawal/ immutable research specification (do not modify)
|
||
artifacts/analysis/ committed machine-readable analysis outputs
|
||
artifacts/figures/final/ thesis-ready figures
|
||
artifacts/partitions/ deterministic frozen Dirichlet partition manifests
|
||
results/ local run outputs (git-ignored, intentionally not committed)
|
||
```
|
||
|
||
## Key Scripts
|
||
|
||
| Script | Role |
|
||
|---|---|
|
||
| `scripts/run_baseline.py` | full baseline / support-weighted training runner (frozen; grid complete) |
|
||
| `scripts/generate_partitions.py` | deterministic frozen Dirichlet manifests |
|
||
| `scripts/analyze_multiseed_beta.py` | pooled validation beta selection |
|
||
| `scripts/analyze_partition_heterogeneity.py` | realized non-IID characterization |
|
||
| `scripts/analyze_prototype_effect.py` | fixed-snapshot prototype displacement |
|
||
| `scripts/analyze_cross_alpha.py` | cross-alpha synthesis |
|
||
| `scripts/verify_beta0_equivalence.py` | beta=0 vs original semantic/numerical audit |
|
||
| `scripts/validate_sj_participant_rule.py` | `S_j` reproduction-assumption analysis |
|
||
|
||
## Important Project Documents
|
||
|
||
**Proposal**
|
||
|
||
- `docs/proposal/THESIS_PROPOSAL_DRAFT.md` — final Markdown proposal source for the current scope
|
||
- `docs/proposal/PROPOSAL_SOURCE_MAP.md` — claim-to-evidence map (internal)
|
||
- `docs/proposal/LITERATURE_MATRIX.md` — literature matrix (internal)
|
||
- `docs/proposal/REFERENCES_VERIFIED.md` — bibliography verification record (internal)
|
||
|
||
**Research synthesis**
|
||
|
||
- `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md` — authoritative six-condition synthesis
|
||
- `docs/research/CROSS_DATASET_BETA_ANALYSIS.md` — cross-dataset comparison
|
||
- `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md` — X-IIoTID cross-alpha analysis
|
||
- `docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md` — 5G-NIDD cross-alpha analysis
|
||
- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen reproduction protocol
|
||
|
||
**Thesis support**
|
||
|
||
- `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md` — Chapter 4 blueprint
|
||
- `docs/thesis/CLAIMS_REGISTER.md` — claim / evidence / status register
|
||
|
||
**Handoff and planning**
|
||
|
||
- `docs/handoff/SESSION_HANDOFF.md` — current state and next steps (start here)
|
||
- `planningawal/PROTEAN_Revisi_Lurus.md` — immutable research specification
|
||
|
||
> `planningawal/` is **immutable**. Read it, cite it, but never edit, rename, move, or add files to
|
||
> it.
|
||
|
||
## Testing and Reproducibility
|
||
|
||
```bash
|
||
# unit + integration tests (expected: 75 passed)
|
||
.venv/bin/python -m pytest
|
||
```
|
||
|
||
- **Paired comparison:** within each `(dataset, alpha, seed)`, every compared configuration reuses
|
||
the same partition, split, class mapping, and model initialization.
|
||
- **Frozen manifests:** partitions are generated once and reused; regenerating a fresh Dirichlet
|
||
split is a protocol violation.
|
||
- **Immutable raw data:** datasets live outside the repository and are never modified or committed.
|
||
- **Validation-only selection:** beta is chosen from validation data only; test data is never used
|
||
for any model-selection decision.
|
||
- **Machine-readable outputs:** analysis is stored as CSV/JSON under `artifacts/analysis/`, and the
|
||
final figures load from those artifacts.
|
||
|
||
## Frozen / Do Not Modify
|
||
|
||
Treat the following as finished unless there is an explicit, documented reason:
|
||
|
||
- `planningawal/` — immutable research source.
|
||
- Raw datasets under `~/datasets/` — read-only, external, never copied into the repository.
|
||
- Completed scientific results and the final synthesis documents.
|
||
- Frozen experiment artifacts: partition manifests, analysis outputs, and final figures.
|
||
- `src/`, model, and aggregation code used for the 108-run grid.
|
||
- The finalized proposal Markdown.
|
||
|
||
## Limitations
|
||
|
||
- The `S_j` participant/denominator rule remains a reproduction assumption and is materially
|
||
exercised in a substantial fraction of class-condition cells.
|
||
- Two datasets, three alpha values, three project-defined seeds, 10 clients, one architecture.
|
||
- The beta grid is discrete; intermediate values were not tested.
|
||
- No inferential trend testing (three seeds are underpowered for it).
|
||
- `beta = 0` is not bitwise identical to `original_protean` (floating-point ordering).
|
||
- Prototype displacement is a representation metric, not detection quality.
|
||
|
||
## Quickstart
|
||
|
||
```bash
|
||
# unit + integration tests (expected: 75 passed)
|
||
.venv/bin/python -m pytest
|
||
|
||
# read-only pooled validation analysis (no retraining required)
|
||
.venv/bin/python scripts/analyze_multiseed_beta.py --dataset 5gnidd --alpha 0.25 --seeds 0 1 2
|
||
|
||
# read-only cross-alpha / cross-dataset synthesis
|
||
.venv/bin/python scripts/analyze_partition_heterogeneity.py --dataset 5gnidd --alphas 0.25 0.5 0.75
|
||
.venv/bin/python scripts/analyze_cross_alpha.py --dataset 5gnidd --alphas 0.25 0.5 0.75
|
||
```
|
||
|
||
See `docs/handoff/SESSION_HANDOFF.md` before making any methodological change. The next work is
|
||
academic writing and formatting, not new experiments.
|