willtanoe/thesis-prototype-agregation
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update readme for completed research study

Browse Source
This commit is contained in:
willtanoe committed 2026-10-01 01:07:23 +07:00
1 parent 8a571a2387
commit 2ab6d74433
1 file changed
+176 -51
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+176 -51
View File
@@ -1,76 +1,201 @@
# perdalam-protean
# PROTEAN Prototype Aggregation Study
Research code for **server-side prototype aggregation in PROTEAN** using class-specific
sample support, for rare attack detection under non-IID federated intrusion detection.
Research code for **server-side prototype aggregation in PROTEAN**, a federated intrusion-detection
framework, under non-IID client data. The study investigates a beta-controlled, class-specific
sample-support weighting of local class prototypes for **rare attack detection**.
The planned study modifies only how the server combines local class prototypes. The
proposed support-weighted aggregation is:
This repository is a finished experimental study: the planned federated grid is complete and
audited, and the remaining work is thesis writing. Raw datasets are external and immutable; all
scientific outputs are derived from audited, machine-readable artifacts.
## Research Question / Motivation
Each client `i` forms a local prototype `C_ij` for class `j` from `N_ij` local training samples.
`N_ij` can differ by orders of magnitude across clients (non-IID), yet a simple server-side mean
gives every contributing client the same weight regardless of support. This study asks whether,
and under what conditions, weighting each client's contribution by its class-specific sample
support changes the global prototype and rare-class detection behavior.
`N_ij` is **class-specific sample support, not prototype quality**. A client with few samples is
not assumed to have a worse prototype; low-support clients may carry variation unavailable at
high-support clients.
## Proposed Aggregation
```
Cbar_j(beta) = ( sum_{i in S_j} N_ij^beta C_ij ) / ( sum_{i in S_j} N_ij^beta )
Cbar_j(beta) = ( sum_{i in S_j} N_ij^beta * C_ij ) / ( sum_{i in S_j} N_ij^beta )
S_j = { i | N_ij > 0 and a valid local prototype C_ij exists }
beta in {0.00, 0.25, 0.50, 0.75, 1.00}
```
with `S_j = { i | N_ij > 0 and C_ij exists }` and candidate
`beta in {0.0, 0.25, 0.5, 0.75, 1.0}`.
- `beta = 0`: equal weighting over the contributors `S_j`.
- `beta = 1`: contribution proportional to class-specific support.
- `0 < beta < 1`: intermediate weighting.
**Current checkpoint: baseline only.** The proposed beta-weighted aggregation and the
`equal_available` control are **not implemented** in this checkpoint; they are deferred to
the later beta/controls phase, which must not begin until the baseline is verified. See
`docs/handoff/SESSION_HANDOFF.md`.
`S_j` (participant set and denominator rule) is a **documented reproduction assumption**: no
verified official PROTEAN code is available. Beta selection, all comparisons, and the `beta = 0`
equivalence result are conditional on this shared rule. The modification is narrowly scoped to
the server-side aggregation step; client training, model architecture, and loss are unchanged.
## Immutable source
## Experimental Protocol
- `planningawal/PROTEAN_Revisi_Lurus.md` is the authoritative research specification and is
**read-only**. Do not edit, add, rename, move, or delete anything under `planningawal/`.
| Item | Value |
|---|---|
| Datasets | X-IIoTID (`class2`, benign `Normal`, K=9); 5G-NIDD (`Attack Type`, benign `Benign`, K=9) |
| Dirichlet alpha | 0.25, 0.50, 0.75 |
| Seeds | 0, 1, 2 (project-defined) |
| Clients | 10 |
| Rounds | 10 |
| Local epochs | 3 |
| Batch size | 256 |
| Optimizer / learning rate | Adam / 1e-3 |
| FedProx mu | 0.1 |
| Prototype alignment lambda | 1.0 |
| Beta candidates | 0.00, 0.25, 0.50, 0.75, 1.00 |
| Modes | `original_protean`, `support_weighted` |
| Splits | train / validation / test = 70 / 10 / 20 (stratified) |
| Selection | pooled validation per-client-mean rare-class macro-F1 across seeds |
| Test | used only after the beta is fixed |
## External raw datasets (read-only, never copied into the repo)
Each `(dataset, alpha, seed)` uses one deterministic frozen partition; all compared
configurations within it share the same manifest, split, class mapping, and initialization.
- `~/datasets/X-IIoTID/X-IIoTID-dataset.csv`
- `~/datasets/5G-NIDD/5G-NIDD.csv`
Paths are configured in `configs/datasets/datasets.yaml` (with environment overrides) and are
never hardcoded in source. Generated artifacts go under `artifacts/`.
## Layout
## Completed Experiment Grid
```
docs/research/ spec interpretation, dataset inspection, implementation plan, protocol
docs/baseline/ PROTEAN baseline investigation + baseline implementation notes
docs/handoff/ session handoff (start here for a new session)
configs/ datasets, experiments, model
src/protean_research/ data, models, aggregation, federated, evaluation, utils (baseline)
scripts/ run_baseline.py, generate_partitions.py
tests/ focused unit + integration tests
artifacts/ results/ generated outputs (not source)
2 datasets x 3 alpha conditions x 3 seeds x (1 original PROTEAN + 5 beta candidates)
= 108 full federated runs
```
All 108 planned runs completed and audited. Smoke runs are excluded and are not scientific
experiments. Realized non-IID characteristics (not Dirichlet alpha alone) are recorded per
condition.
## Final Pooled Beta Selections
Validation-only, pooled across clients and seeds; best among the tested candidates.
| Dataset | alpha=0.25 | alpha=0.50 | alpha=0.75 |
|---|---|---|---|
| X-IIoTID | 0.00 | 0.75 | 0.50 |
| 5G-NIDD | 0.00 | 1.00 | 0.50 |
Caveats: these are the best among the five tested candidates, **not global optima**; no universal
or monotonic beta relationship is established; only two datasets and three alpha levels were
evaluated.
## Main Findings
1. Realized support imbalance and direct prototype displacement from `beta = 0` **co-vary
descriptively** within both datasets (larger imbalance accompanies larger displacement).
2. For a fixed snapshot, prototype displacement increases **monotonically** with beta in every
condition.
3. Rare-class detection response is **non-monotonic** and condition-dependent; the direct
aggregation effect and the end-to-end training effect are distinct.
4. The selected beta **changes across conditions and datasets**.
5. **No universal beta** was found.
6. Both datasets selected **beta = 0 at alpha = 0.25** (strongest realized imbalance, largest
displacement); this is a descriptive pattern, not a rule.
7. Under the shared `S_j` reproduction rule, `beta = 0` is **semantically** the equal mean over
`S_j`, but downstream bitwise identity with `original_protean` is not guaranteed because of
floating-point operation ordering.
Not supported by the evidence (must not be claimed): support weighting improves rare detection
in general; the selected beta beats original PROTEAN; alpha causally determines beta; `N_ij`
measures prototype quality; larger displacement means better detection; a global optimum exists;
results generalize beyond the evaluated scope.
## Repository Structure
```
src/protean_research/ data, models, aggregation, federated, evaluation, utils
scripts/ training, partition, and analysis scripts
configs/ datasets, experiments, model configuration
docs/research/ protocol, per-condition reports, cross-alpha/cross-dataset analysis, audits
docs/baseline/ PROTEAN baseline investigation and validation
docs/thesis/ Chapter 4 blueprint and claims register
docs/handoff/ session handoff (start here for a new session)
artifacts/analysis/ machine-readable analysis outputs (committed)
artifacts/figures/final/ thesis-ready figures
artifacts/partitions/ deterministic frozen Dirichlet partition manifests
results/ local scientific run outputs (intentionally not committed)
```
`results/` holds local run outputs and is git-ignored; committed scientific outputs are the
artifact tables and figures under `artifacts/`. Raw datasets are never copied into the repository.
## Key Scripts
| Script | Role |
|---|---|
| `scripts/run_baseline.py` | full baseline / support-weighted training runner |
| `scripts/generate_partitions.py` | deterministic frozen Dirichlet manifests |
| `scripts/analyze_multiseed_beta.py` | pooled validation beta selection |
| `scripts/analyze_partition_heterogeneity.py` | realized non-IID characterization |
| `scripts/analyze_prototype_effect.py` | fixed-snapshot prototype displacement |
| `scripts/analyze_cross_alpha.py` | cross-alpha synthesis |
| `scripts/verify_beta0_equivalence.py` | beta=0 vs original semantic/numerical audit |
| `scripts/validate_sj_participant_rule.py` | `S_j` reproduction-assumption analysis |
## Important Research Documents
- `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md` — final six-condition synthesis
- `docs/research/CROSS_DATASET_BETA_ANALYSIS.md` — cross-dataset comparison
- `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md` — X-IIoTID cross-alpha analysis
- `docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md` — 5G-NIDD cross-alpha analysis
- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen protocol
- `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md` — Chapter 4 blueprint
- `docs/thesis/CLAIMS_REGISTER.md` — claim / evidence / status register
- `docs/handoff/SESSION_HANDOFF.md` — current state and next steps
## Reproducibility Notes
- Raw datasets are external and immutable; they are never modified, renamed, or committed.
Their SHA-256 hashes are recorded in the audit documents.
- Frozen partition manifests are generated once and reused; a run that regenerates a fresh
Dirichlet split is a protocol violation.
- Paired seeds, partitions, class mappings, and hyperparameters are shared across all compared
configurations within a condition.
- Beta selection uses validation only; test metrics are read only after the beta is fixed.
- Analysis outputs are machine-readable (CSV/JSON), and figures load from those artifacts.
- `src/`, model, and aggregation code were frozen during the final experiments (no training
tuning driven by test results).
## Limitations
- The `S_j` participant/denominator rule remains a reproduction assumption, materially exercised
in a substantial fraction of class-condition cells.
- Two datasets, three alpha values, three project-defined seeds, 10 clients, one architecture.
- The beta grid is discrete; intermediate values were not tested.
- No inferential trend testing (three seeds are underpowered for it).
- `beta = 0` is not bitwise identical to `original_protean` (floating-point ordering).
- `N_ij` is sample support, not prototype quality; prototype displacement is a representation
metric, not detection quality.
## Status
The minimum reproducible **baseline** is implemented and smoke-tested. No full-scale
training and no beta sweep have been run. The original PROTEAN baseline has **no public
official code**, so it is reproduced from the paper/thesis with explicitly documented
reproduction assumptions (`docs/baseline/BASELINE_IMPLEMENTATION.md`).
Read `docs/handoff/SESSION_HANDOFF.md` first, then
`planningawal/PROTEAN_Revisi_Lurus.md`.
- Experimental grid: COMPLETE (108 full runs)
- Audit status: PASS (two conditions with documentation-only minor corrections)
- Final synthesis: COMPLETE
- Additional training planned: NO
## Quickstart
```bash
# inside the repo, with the project virtualenv (numpy, pandas, torch, pyyaml, pytest)
.venv/bin/python -m pytest -q
# unit + integration tests (expected: 75 passed)
.venv/bin/python -m pytest
# baseline smoke test (writes only to ignored directories)
.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.5 --seed 0 \
--max-rows 3000 --sample-from 60000 --clients 4 --rounds 2 --local-epochs 1
# generate + verify all deterministic partition manifests (full data)
# generate / verify deterministic partition manifests (full data)
.venv/bin/python scripts/generate_partitions.py
# pooled multi-seed validation analysis (read-only, no retraining)
.venv/bin/python scripts/analyze_multiseed_beta.py --dataset 5gnidd --alpha 0.25 --seeds 0 1 2
# one full paired training run (CPU, long)
.venv/bin/python scripts/run_baseline.py --dataset 5gnidd --alpha 0.25 --seed 1 \
--clients 10 --rounds 10 --local-epochs 3 \
--aggregation-mode support_weighted --beta 1.0
```
The baseline aggregation is dependency-free and testable without pandas/torch:
```python
from protean_research.aggregation import aggregate_prototypes
aggregate_prototypes([[0.0, 2.0], [2.0, 0.0]]) # equal mean over the contributor set S_j
```
See `docs/handoff/SESSION_HANDOFF.md` before making methodological changes. The entire
`planningawal/` directory is immutable research source material.