willtanoe/thesis-prototype-agregation
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: refresh README for finalized research state

Browse Source
This commit is contained in:
willtanoe committed 2026-10-02 14:35:14 +07:00
1 parent eaa9217552
commit 8255393da9
1 file changed
+142 -116
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+142 -116
View File
@@ -1,25 +1,52 @@
# PROTEAN Prototype Aggregation Study
Research code for **server-side prototype aggregation in PROTEAN**, a federated intrusion-detection
framework, under non-IID client data. The study investigates a beta-controlled, class-specific
sample-support weighting of local class prototypes for **rare attack detection**.
Research code and audited artifacts for a master's thesis on **server-side prototype aggregation in
PROTEAN** (Chennoufi et al., 2025), a federated intrusion-detection framework, under non-IID client
data. The study investigates a **beta-controlled, class-specific sample-support weighting** of local
class prototypes for **rare attack detection**.
This repository is a finished experimental study: the planned federated grid is complete and
audited, and the remaining work is thesis writing. Raw datasets are external and immutable; all
scientific outputs are derived from audited, machine-readable artifacts.
The experimental study is **complete** (108 full runs, six non-IID conditions, both datasets) and the
thesis proposal Markdown is **finalized**. No further training is planned unless new academic
feedback explicitly requires it.
## Research Question / Motivation
## Final Research Title
Each client `i` forms a local prototype `C_ij` for class `j` from `N_ij` local training samples.
`N_ij` can differ by orders of magnitude across clients (non-IID), yet a simple server-side mean
gives every contributing client the same weight regardless of support. This study asks whether,
and under what conditions, weighting each client's contribution by its class-specific sample
support changes the global prototype and rare-class detection behavior.
**Optimasi Agregasi Prototype pada PROTEAN melalui Pembobotan Berbasis Dukungan Sampel Kelas untuk
Deteksi Serangan Langka pada Data Non-IID**
`N_ij` is **class-specific sample support, not prototype quality**. A client with few samples is
not assumed to have a worse prototype; low-support clients may carry variation unavailable at
*Optimizing Prototype Aggregation in PROTEAN through Class-Sample-Support Weighting for Rare Attack
Detection under Non-IID Data Distributions.*
Here **"Optimasi" means empirical tuning/selection of `beta` among the predefined candidate values**,
not a global mathematical optimum and not a universal beta. The selected value is the best among the
tested candidates under the defined validation procedure for each condition.
## Research Scope
- **Base method:** PROTEAN, a federated intrusion-detection framework for non-IID data.
- **Focus:** the **server-side prototype aggregation** step only. Client training, model
architecture, loss, and inference are unchanged.
- **Proposed mechanism:** weight each contributing client's class prototype by `N_ij^beta`, where
`N_ij` is that client's class-specific training support and `beta` is a single tunable exponent
in `[0, 1]`.
- **Analysis chain:** support imbalance -> contribution weight -> prototype displacement ->
rare-class detection -> per-client effect.
`N_ij` is **class-specific sample support, not prototype quality**. A client with few samples is not
assumed to have a worse prototype; low-support clients may carry variation unavailable at
high-support clients.
## Why This Problem Matters
Federated IDS data is naturally non-IID: each client observes different traffic and different attack
types. Rare attack classes appear in little of the data, so aggregate metrics can look good while
the model fails on them. PROTEAN builds a global prototype per class on the server and uses it both
for local prototype alignment and for nearest-prototype inference. Its reference aggregation gives
every contributing client equal weight, even though the local prototypes may be built from very
different numbers of samples. This study asks whether, and under which realized non-IID conditions,
weighting contributions by class sample support changes the global prototype and rare-class
detection behavior.
## Proposed Aggregation
```
@@ -32,78 +59,56 @@ beta in {0.00, 0.25, 0.50, 0.75, 1.00}
- `beta = 1`: contribution proportional to class-specific support.
- `0 < beta < 1`: intermediate weighting.
`S_j` (participant set and denominator rule) is a **documented reproduction assumption**: no
verified official PROTEAN code is available. Beta selection, all comparisons, and the `beta = 0`
equivalence result are conditional on this shared rule. The modification is narrowly scoped to
the server-side aggregation step; client training, model architecture, and loss are unchanged.
`S_j` (participant set and denominator rule) is a **documented reproduction assumption**: no verified
official PROTEAN code is available. `beta = 0` is semantically the equal mean over `S_j`, but is not
guaranteed to be bitwise identical to the frozen `original_protean` baseline because of
floating-point operation ordering.
## Experimental Protocol
## Experimental Setup
| Item | Value |
|---|---|
| Datasets | X-IIoTID (`class2`, benign `Normal`, K=9); 5G-NIDD (`Attack Type`, benign `Benign`, K=9) |
| Dirichlet alpha | 0.25, 0.50, 0.75 |
| Non-IID conditions | symmetric Dirichlet `alpha` = 0.75, 0.50, 0.25 |
| Seeds | 0, 1, 2 (project-defined) |
| Clients | 10 |
| Rounds | 10 |
| Local epochs | 3 |
| Batch size | 256 |
| Optimizer / learning rate | Adam / 1e-3 |
| FedProx mu | 0.1 |
| Prototype alignment lambda | 1.0 |
| Beta candidates | 0.00, 0.25, 0.50, 0.75, 1.00 |
| Modes | `original_protean`, `support_weighted` |
| Splits | train / validation / test = 70 / 10 / 20 (stratified) |
| Clients / rounds / local epochs | 10 / 10 / 3 |
| Batch size / optimizer / learning rate | 256 / Adam / 1e-3 |
| FedProx `mu` / prototype-alignment `lambda` | 0.1 / 1.0 |
| Compared configurations | `original_protean` + `support_weighted` at beta 0, 0.25, 0.50, 0.75, 1.00 |
| Splits | stratified train / validation / test = 70 / 10 / 20 |
| Selection | pooled validation per-client-mean rare-class macro-F1 across seeds |
| Test | used only after the beta is fixed |
| Test | read only after the selected beta is fixed |
Each `(dataset, alpha, seed)` uses one deterministic frozen partition; all compared
configurations within it share the same manifest, split, class mapping, and initialization.
Each `(dataset, alpha, seed)` uses one deterministic frozen partition; all compared configurations
within it share the same manifest, split, class mapping, and initialization.
## Completed Experiment Grid
## Experiment Status
```
2 datasets x 3 alpha conditions x 3 seeds x (1 original PROTEAN + 5 beta candidates)
= 108 full federated runs
```
- **Complete: 108 full runs** = 2 datasets × 3 alpha conditions × 3 seeds ×
(1 original PROTEAN + 5 beta settings).
- All conditions are trained, audited, and synthesized.
- Smoke runs are excluded and are not scientific experiments.
- **No further training planned** unless new academic feedback requires it.
All 108 planned runs completed and audited. Smoke runs are excluded and are not scientific
experiments. Realized non-IID characteristics (not Dirichlet alpha alone) are recorded per
condition.
## Final Scientific Position
## Final Pooled Beta Selections
High-level, evidence-safe conclusions only (full tables live in the synthesis documents):
Validation-only, pooled across clients and seeds; best among the tested candidates.
- Realized support imbalance, missing client-class cells, and client-size ratio vary by condition;
Dirichlet `alpha` is a design knob, not the realized heterogeneity itself.
- Increasing `beta` changes the normalized contribution weights and therefore the position of each
global prototype; for a fixed snapshot, displacement grows with `beta`.
- The rare-class detection response is **condition-dependent and non-monotone** in `beta`; the direct
aggregation effect and the end-to-end training effect are distinct.
- **No universal beta exists.** The validation-selected beta differs across datasets and non-IID
conditions and is not monotone in `alpha`.
- Stronger support weighting can **help, have limited effect, or hurt** rare-class behavior,
depending on the condition, and can redistribute benefit across clients within one condition.
| Dataset | alpha=0.25 | alpha=0.50 | alpha=0.75 |
|---|---|---|---|
| X-IIoTID | 0.00 | 0.75 | 0.50 |
| 5G-NIDD | 0.00 | 1.00 | 0.50 |
Caveats: these are the best among the five tested candidates, **not global optima**; no universal
or monotonic beta relationship is established; only two datasets and three alpha levels were
evaluated.
## Main Findings
1. Realized support imbalance and direct prototype displacement from `beta = 0` **co-vary
descriptively** within both datasets (larger imbalance accompanies larger displacement).
2. For a fixed snapshot, prototype displacement increases **monotonically** with beta in every
condition.
3. Rare-class detection response is **non-monotonic** and condition-dependent; the direct
aggregation effect and the end-to-end training effect are distinct.
4. The selected beta **changes across conditions and datasets**.
5. **No universal beta** was found.
6. Both datasets selected **beta = 0 at alpha = 0.25** (strongest realized imbalance, largest
displacement); this is a descriptive pattern, not a rule.
7. Under the shared `S_j` reproduction rule, `beta = 0` is **semantically** the equal mean over
`S_j`, but downstream bitwise identity with `original_protean` is not guaranteed because of
floating-point operation ordering.
Not supported by the evidence (must not be claimed): support weighting improves rare detection
in general; the selected beta beats original PROTEAN; alpha causally determines beta; `N_ij`
measures prototype quality; larger displacement means better detection; a global optimum exists;
results generalize beyond the evaluated scope.
Not supported by the evidence (must not be claimed): support weighting always improves rare-class
detection; the selected beta universally beats `original_protean`; `alpha` causally determines beta;
`N_ij` measures prototype quality; larger prototype displacement means better detection; a global
optimum exists; results generalize beyond the two evaluated datasets.
## Repository Structure
@@ -111,24 +116,23 @@ results generalize beyond the evaluated scope.
src/protean_research/ data, models, aggregation, federated, evaluation, utils
scripts/ training, partition, and analysis scripts
configs/ datasets, experiments, model configuration
docs/research/ protocol, per-condition reports, cross-alpha/cross-dataset analysis, audits
docs/baseline/ PROTEAN baseline investigation and validation
docs/proposal/ thesis proposal and its internal audit aids
docs/research/ frozen protocol, per-condition reports, cross-alpha / cross-dataset synthesis
docs/thesis/ Chapter 4 blueprint and claims register
docs/baseline/ PROTEAN baseline investigation and validation
docs/handoff/ session handoff (start here for a new session)
artifacts/analysis/ machine-readable analysis outputs (committed)
planningawal/ immutable research specification (do not modify)
artifacts/analysis/ committed machine-readable analysis outputs
artifacts/figures/final/ thesis-ready figures
artifacts/partitions/ deterministic frozen Dirichlet partition manifests
results/ local scientific run outputs (intentionally not committed)
results/ local run outputs (git-ignored, intentionally not committed)
```
`results/` holds local run outputs and is git-ignored; committed scientific outputs are the
artifact tables and figures under `artifacts/`. Raw datasets are never copied into the repository.
## Key Scripts
| Script | Role |
|---|---|
| `scripts/run_baseline.py` | full baseline / support-weighted training runner |
| `scripts/run_baseline.py` | full baseline / support-weighted training runner (frozen; grid complete) |
| `scripts/generate_partitions.py` | deterministic frozen Dirichlet manifests |
| `scripts/analyze_multiseed_beta.py` | pooled validation beta selection |
| `scripts/analyze_partition_heterogeneity.py` | realized non-IID characterization |
@@ -137,47 +141,73 @@ artifact tables and figures under `artifacts/`. Raw datasets are never copied in
| `scripts/verify_beta0_equivalence.py` | beta=0 vs original semantic/numerical audit |
| `scripts/validate_sj_participant_rule.py` | `S_j` reproduction-assumption analysis |
## Important Research Documents
## Important Project Documents
- `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md` — final six-condition synthesis
**Proposal**
- `docs/proposal/THESIS_PROPOSAL_DRAFT.md` — final Markdown proposal source for the current scope
- `docs/proposal/PROPOSAL_SOURCE_MAP.md` — claim-to-evidence map (internal)
- `docs/proposal/LITERATURE_MATRIX.md` — literature matrix (internal)
- `docs/proposal/REFERENCES_VERIFIED.md` — bibliography verification record (internal)
**Research synthesis**
- `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md` — authoritative six-condition synthesis
- `docs/research/CROSS_DATASET_BETA_ANALYSIS.md` — cross-dataset comparison
- `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md` — X-IIoTID cross-alpha analysis
- `docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md` — 5G-NIDD cross-alpha analysis
- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen protocol
- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen reproduction protocol
**Thesis support**
- `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md` — Chapter 4 blueprint
- `docs/thesis/CLAIMS_REGISTER.md` — claim / evidence / status register
- `docs/handoff/SESSION_HANDOFF.md` — current state and next steps
## Reproducibility Notes
**Handoff and planning**
- Raw datasets are external and immutable; they are never modified, renamed, or committed.
Their SHA-256 hashes are recorded in the audit documents.
- Frozen partition manifests are generated once and reused; a run that regenerates a fresh
Dirichlet split is a protocol violation.
- Paired seeds, partitions, class mappings, and hyperparameters are shared across all compared
configurations within a condition.
- Beta selection uses validation only; test metrics are read only after the beta is fixed.
- Analysis outputs are machine-readable (CSV/JSON), and figures load from those artifacts.
- `src/`, model, and aggregation code were frozen during the final experiments (no training
tuning driven by test results).
- `docs/handoff/SESSION_HANDOFF.md` — current state and next steps (start here)
- `planningawal/PROTEAN_Revisi_Lurus.md` — immutable research specification
> `planningawal/` is **immutable**. Read it, cite it, but never edit, rename, move, or add files to
> it.
## Testing and Reproducibility
```bash
# unit + integration tests (expected: 75 passed)
.venv/bin/python -m pytest
```
- **Paired comparison:** within each `(dataset, alpha, seed)`, every compared configuration reuses
the same partition, split, class mapping, and model initialization.
- **Frozen manifests:** partitions are generated once and reused; regenerating a fresh Dirichlet
split is a protocol violation.
- **Immutable raw data:** datasets live outside the repository and are never modified or committed.
- **Validation-only selection:** beta is chosen from validation data only; test data is never used
for any model-selection decision.
- **Machine-readable outputs:** analysis is stored as CSV/JSON under `artifacts/analysis/`, and the
final figures load from those artifacts.
## Frozen / Do Not Modify
Treat the following as finished unless there is an explicit, documented reason:
- `planningawal/` — immutable research source.
- Raw datasets under `~/datasets/` — read-only, external, never copied into the repository.
- Completed scientific results and the final synthesis documents.
- Frozen experiment artifacts: partition manifests, analysis outputs, and final figures.
- `src/`, model, and aggregation code used for the 108-run grid.
- The finalized proposal Markdown.
## Limitations
- The `S_j` participant/denominator rule remains a reproduction assumption, materially exercised
in a substantial fraction of class-condition cells.
- The `S_j` participant/denominator rule remains a reproduction assumption and is materially
exercised in a substantial fraction of class-condition cells.
- Two datasets, three alpha values, three project-defined seeds, 10 clients, one architecture.
- The beta grid is discrete; intermediate values were not tested.
- No inferential trend testing (three seeds are underpowered for it).
- `beta = 0` is not bitwise identical to `original_protean` (floating-point ordering).
- `N_ij` is sample support, not prototype quality; prototype displacement is a representation
metric, not detection quality.
## Status
- Experimental grid: COMPLETE (108 full runs)
- Audit status: PASS (two conditions with documentation-only minor corrections)
- Final synthesis: COMPLETE
- Additional training planned: NO
- Prototype displacement is a representation metric, not detection quality.
## Quickstart
@@ -185,17 +215,13 @@ artifact tables and figures under `artifacts/`. Raw datasets are never copied in
# unit + integration tests (expected: 75 passed)
.venv/bin/python -m pytest
# generate / verify deterministic partition manifests (full data)
.venv/bin/python scripts/generate_partitions.py
# pooled multi-seed validation analysis (read-only, no retraining)
# read-only pooled validation analysis (no retraining required)
.venv/bin/python scripts/analyze_multiseed_beta.py --dataset 5gnidd --alpha 0.25 --seeds 0 1 2
# one full paired training run (CPU, long)
.venv/bin/python scripts/run_baseline.py --dataset 5gnidd --alpha 0.25 --seed 1 \
--clients 10 --rounds 10 --local-epochs 3 \
--aggregation-mode support_weighted --beta 1.0
# read-only cross-alpha / cross-dataset synthesis
.venv/bin/python scripts/analyze_partition_heterogeneity.py --dataset 5gnidd --alphas 0.25 0.5 0.75
.venv/bin/python scripts/analyze_cross_alpha.py --dataset 5gnidd --alphas 0.25 0.5 0.75
```
See `docs/handoff/SESSION_HANDOFF.md` before making methodological changes. The entire
`planningawal/` directory is immutable research source material.
See `docs/handoff/SESSION_HANDOFF.md` before making any methodological change. The next work is
academic writing and formatting, not new experiments.