docs: document completed PROTEAN experiments and findings
This commit is contained in:
willtanoe committed
1 parent
8255393da9
commit
8a9bdcf63c
1 file changed
+632
-174
@@ -1,227 +1,685 @@
|
||||
# PROTEAN Prototype Aggregation Study
|
||||
|
||||
Research code and audited artifacts for a master's thesis on **server-side prototype aggregation in
|
||||
PROTEAN** (Chennoufi et al., 2025), a federated intrusion-detection framework, under non-IID client
|
||||
data. The study investigates a **beta-controlled, class-specific sample-support weighting** of local
|
||||
class prototypes for **rare attack detection**.
|
||||
Research code and audited artifacts for a master's thesis on **server-side prototype aggregation
|
||||
in PROTEAN** (Chennoufi et al., 2025), a federated intrusion-detection framework, under non-IID
|
||||
client data. The study evaluates a **beta-controlled, class-specific sample-support weighting** of
|
||||
local class prototypes for **rare attack detection**.
|
||||
|
||||
The experimental study is **complete** (108 full runs, six non-IID conditions, both datasets) and the
|
||||
thesis proposal Markdown is **finalized**. No further training is planned unless new academic
|
||||
feedback explicitly requires it.
|
||||
**Research status.** The experimental grid is **complete** (108 full federated runs), the results
|
||||
are synthesized and audited, and the thesis proposal Markdown is finalized. No further training is
|
||||
planned unless new academic feedback explicitly requires it. The repository is in a documentation
|
||||
and thesis-writing stage.
|
||||
|
||||
## Final Research Title
|
||||
**Final title.** *Optimasi Agregasi Prototype pada PROTEAN melalui Pembobotan Berbasis Dukungan
|
||||
Sampel Kelas untuk Deteksi Serangan Langka pada Data Non-IID.* Here "Optimasi" means empirical
|
||||
selection of `beta` among the predefined candidate values, not a global mathematical optimum.
|
||||
|
||||
**Optimasi Agregasi Prototype pada PROTEAN melalui Pembobotan Berbasis Dukungan Sampel Kelas untuk
|
||||
Deteksi Serangan Langka pada Data Non-IID**
|
||||
---
|
||||
|
||||
*Optimizing Prototype Aggregation in PROTEAN through Class-Sample-Support Weighting for Rare Attack
|
||||
Detection under Non-IID Data Distributions.*
|
||||
## 1. Research Overview
|
||||
|
||||
Here **"Optimasi" means empirical tuning/selection of `beta` among the predefined candidate values**,
|
||||
not a global mathematical optimum and not a universal beta. The selected value is the best among the
|
||||
tested candidates under the defined validation procedure for each condition.
|
||||
PROTEAN is a federated intrusion-detection method that shares **class prototypes** between clients
|
||||
and the server in addition to model parameters. Each client `i` builds a local prototype `C_ij` for
|
||||
class `j` from its own training samples; the server aggregates these into a global prototype
|
||||
`Cbar_j` that is used both for local prototype alignment during training and as the classifier at
|
||||
inference (nearest prototype in embedding space).
|
||||
|
||||
## Research Scope
|
||||
This thesis modifies **only the server-side prototype aggregation**. The original PROTEAN
|
||||
aggregation gives every contributing client equal weight. The proposed aggregation weights each
|
||||
client's prototype by `N_ij^beta`, where `N_ij` is that client's class-specific training support and
|
||||
`beta` is a single exponent in `[0, 1]`. Everything else — local prototype formation, model
|
||||
aggregation, architecture, loss, and inference rule — is unchanged relative to the reproduced
|
||||
baseline.
|
||||
|
||||
- **Base method:** PROTEAN, a federated intrusion-detection framework for non-IID data.
|
||||
- **Focus:** the **server-side prototype aggregation** step only. Client training, model
|
||||
architecture, loss, and inference are unchanged.
|
||||
- **Proposed mechanism:** weight each contributing client's class prototype by `N_ij^beta`, where
|
||||
`N_ij` is that client's class-specific training support and `beta` is a single tunable exponent
|
||||
in `[0, 1]`.
|
||||
- **Analysis chain:** support imbalance -> contribution weight -> prototype displacement ->
|
||||
rare-class detection -> per-client effect.
|
||||
`N_ij` is **class-specific sample support, not prototype quality**. A client with few samples is
|
||||
not assumed to have a worse prototype.
|
||||
|
||||
`N_ij` is **class-specific sample support, not prototype quality**. A client with few samples is not
|
||||
assumed to have a worse prototype; low-support clients may carry variation unavailable at
|
||||
high-support clients.
|
||||
## 2. Problem Statement and Motivation
|
||||
|
||||
## Why This Problem Matters
|
||||
Federated IDS data is non-IID: clients observe different traffic mixtures and different attack
|
||||
types, so class support varies strongly across clients. Rare attack classes appear in little of
|
||||
the data, so aggregate metrics can look acceptable while the model fails on exactly the classes
|
||||
that matter for detection.
|
||||
|
||||
Federated IDS data is naturally non-IID: each client observes different traffic and different attack
|
||||
types. Rare attack classes appear in little of the data, so aggregate metrics can look good while
|
||||
the model fails on them. PROTEAN builds a global prototype per class on the server and uses it both
|
||||
for local prototype alignment and for nearest-prototype inference. Its reference aggregation gives
|
||||
every contributing client equal weight, even though the local prototypes may be built from very
|
||||
different numbers of samples. This study asks whether, and under which realized non-IID conditions,
|
||||
weighting contributions by class sample support changes the global prototype and rare-class
|
||||
detection behavior.
|
||||
Under the original equal-weight prototype aggregation, a prototype built from a handful of samples
|
||||
contributes as much as one built from thousands. This thesis investigates whether — and under which
|
||||
realized non-IID conditions — weighting contributions by class-specific sample support changes the
|
||||
global prototype and rare-class detection behavior.
|
||||
|
||||
## Proposed Aggregation
|
||||
The study deliberately isolates one mechanism so that any observed change can be attributed to the
|
||||
aggregation weighting rather than to a confounded set of architectural or training changes.
|
||||
|
||||
## 3. Original PROTEAN versus Proposed Modification
|
||||
|
||||
**A. Local prototype (unchanged).** For class `j` with `N_ij > 0` samples on client `i`, the local
|
||||
prototype is the mean embedding of that client's class-`j` samples:
|
||||
|
||||
$$
|
||||
C_{ij} = \frac{1}{N_{ij}} \sum_{k=1}^{N_{ij}} z_{ijk}
|
||||
$$
|
||||
|
||||
where `z_ijk` is the embedding of the k-th sample of class `j` on client `i`.
|
||||
|
||||
**B. Original server-side prototype aggregation (reference).** PROTEAN Eq. (2) averages the `M`
|
||||
contributing clients equally (equal contribution, no `N_ij` weight):
|
||||
|
||||
$$
|
||||
\bar{C}_j = \frac{1}{M} \sum_{i=1}^{M} C_{ij}
|
||||
$$
|
||||
|
||||
**C. Proposed beta-controlled aggregation.** Only the prototype aggregation is modified:
|
||||
|
||||
$$
|
||||
\bar{C}_j(\beta) = \frac{\sum_{i \in S_j} N_{ij}^{\beta}\, C_{ij}}{\sum_{i \in S_j} N_{ij}^{\beta}}, \qquad \beta \in \{0,\ 0.25,\ 0.50,\ 0.75,\ 1.00\}
|
||||
$$
|
||||
|
||||
**D. Normalized contribution weight.** Client `i`'s normalized weight for class `j` is
|
||||
|
||||
$$
|
||||
w_{ij}(\beta) = \frac{N_{ij}^{\beta}}{\sum_{k \in S_j} N_{kj}^{\beta}}
|
||||
$$
|
||||
|
||||
**E. Symbol meanings.**
|
||||
|
||||
| Symbol | Meaning |
|
||||
|---|---|
|
||||
| `N_ij` | number of local **training** samples of class `j` on client `i` (class-specific sample support, not prototype quality) |
|
||||
| `C_ij` | client `i`'s local class-`j` prototype (mean embedding) |
|
||||
| `S_j` | participant set: clients with `N_ij > 0` and a valid local prototype for class `j` |
|
||||
| `M` | number of participants in the reference Eq. (2) (equal average) |
|
||||
| `beta` | scalar weighting exponent in `[0, 1]`, one value for all classes and clients |
|
||||
| `w_ij` | normalized contribution weight (sums to 1 over `S_j`) |
|
||||
| global prototype | `Cbar_j(beta)`; used for alignment and nearest-prototype inference |
|
||||
|
||||
**F. Exact difference between original and modified PROTEAN.**
|
||||
|
||||
- With `beta = 0`, `N_ij^0 = 1`, so the proposed aggregator becomes the **equal mean over `S_j`**.
|
||||
- With `beta = 1`, the contribution is proportional to class support (equivalent to the mean of
|
||||
all included local class embeddings when the assumptions hold).
|
||||
- `beta = 0` is equal weighting over `S_j`; it is equivalent to the original formulation **only
|
||||
if** the participant and missing-class rules are also equivalent. That equivalence is not
|
||||
verified against official PROTEAN code.
|
||||
- Model aggregation, local prototype formation, architecture, loss, and inference are **not**
|
||||
modified.
|
||||
|
||||
**Novelty boundary.** Sample-count prototype weighting is not new by itself; related work such as
|
||||
**FedProto** (Tan et al., 2022) already involves class sample counts in prototype-based federated
|
||||
learning. The contribution here is the **controlled beta parameterization inside PROTEAN's
|
||||
server-side prototype aggregation**, its condition-wise validation-based selection, the paired
|
||||
multi-seed multi-condition evaluation, and the resulting behavioral findings — not the invention of
|
||||
support weighting.
|
||||
|
||||
**Analysis chain (not a causal proof by itself).**
|
||||
|
||||
```
|
||||
Cbar_j(beta) = ( sum_{i in S_j} N_ij^beta * C_ij ) / ( sum_{i in S_j} N_ij^beta )
|
||||
S_j = { i | N_ij > 0 and a valid local prototype C_ij exists }
|
||||
beta in {0.00, 0.25, 0.50, 0.75, 1.00}
|
||||
support imbalance -> contribution weight -> global-prototype displacement
|
||||
-> rare-class detection -> per-client effect
|
||||
```
|
||||
|
||||
- `beta = 0`: equal weighting over the contributors `S_j`.
|
||||
- `beta = 1`: contribution proportional to class-specific support.
|
||||
- `0 < beta < 1`: intermediate weighting.
|
||||
## 4. Research Questions and Objectives
|
||||
|
||||
`S_j` (participant set and denominator rule) is a **documented reproduction assumption**: no verified
|
||||
official PROTEAN code is available. `beta = 0` is semantically the equal mean over `S_j`, but is not
|
||||
guaranteed to be bitwise identical to the frozen `original_protean` baseline because of
|
||||
floating-point operation ordering.
|
||||
| ID | Research question |
|
||||
|---|---|
|
||||
| RQ1 | How are class-support imbalance, differences between local prototypes, and rare-class detection performance related in PROTEAN? |
|
||||
| RQ2 | How does changing `beta` affect the global prototype and rare-class detection performance under different non-IID conditions? |
|
||||
| RQ3 | Does the validation-selected `beta` change across non-IID levels, and what is its effect on overall and per-client performance? |
|
||||
|
||||
## Experimental Setup
|
||||
Objectives: (1) relate support imbalance to prototype behavior and rare-class detection (RQ1);
|
||||
(2) characterize the effect of `beta` on the global prototype and rare-class detection across
|
||||
conditions (RQ2); (3) select `beta` by validation at different imbalance levels and quantify
|
||||
overall and per-client consequences (RQ3).
|
||||
|
||||
## 5. Experimental Design
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| Base method | PROTEAN federated intrusion detection |
|
||||
| Modified component | server-side prototype aggregation only |
|
||||
| Mechanism | `w_ij = N_ij^beta`, one `beta` for all classes/clients |
|
||||
| Datasets | X-IIoTID (`class2`, benign `Normal`, K=9); 5G-NIDD (`Attack Type`, benign `Benign`, K=9) |
|
||||
| Non-IID conditions | symmetric Dirichlet `alpha` = 0.75, 0.50, 0.25 |
|
||||
| Clients | 10 |
|
||||
| Non-IID conditions | symmetric Dirichlet `alpha` = 0.25, 0.50, 0.75 |
|
||||
| Seeds | 0, 1, 2 (project-defined) |
|
||||
| Clients / rounds / local epochs | 10 / 10 / 3 |
|
||||
| Batch size / optimizer / learning rate | 256 / Adam / 1e-3 |
|
||||
| FedProx `mu` / prototype-alignment `lambda` | 0.1 / 1.0 |
|
||||
| Compared configurations | `original_protean` + `support_weighted` at beta 0, 0.25, 0.50, 0.75, 1.00 |
|
||||
| Beta candidates | 0, 0.25, 0.50, 0.75, 1.00 |
|
||||
| Baseline | `original_protean` (equal-weight prototype aggregation) |
|
||||
| Rounds / local epochs / batch | 10 / 3 / 256 |
|
||||
| Optimizer / lr / mu / lambda | Adam / 1e-3 / 0.1 / 1.0 |
|
||||
| Splits | stratified train / validation / test = 70 / 10 / 20 |
|
||||
| Selection | pooled validation per-client-mean rare-class macro-F1 across seeds |
|
||||
| Test | read only after the selected beta is fixed |
|
||||
| Selection metric | pooled **validation** per-client-mean rare-class macro-F1 across seeds |
|
||||
| Test usage | read only after the selected `beta` is fixed |
|
||||
|
||||
Each `(dataset, alpha, seed)` uses one deterministic frozen partition; all compared configurations
|
||||
within it share the same manifest, split, class mapping, and initialization.
|
||||
Expected experimental matrix: `2 datasets × 3 alpha × 3 seeds × (1 baseline + 5 betas) = 108 runs`.
|
||||
Within each `(dataset, alpha, seed)` all six configurations share one manifest, split, class mapping,
|
||||
and round-0 initialization; only the aggregation mode changes.
|
||||
|
||||
## Experiment Status
|
||||
## 6. Completed Research Phases
|
||||
|
||||
- **Complete: 108 full runs** = 2 datasets × 3 alpha conditions × 3 seeds ×
|
||||
(1 original PROTEAN + 5 beta settings).
|
||||
- All conditions are trained, audited, and synthesized.
|
||||
- Smoke runs are excluded and are not scientific experiments.
|
||||
- **No further training planned** unless new academic feedback requires it.
|
||||
Documented phases, in order. Session numbers are used only where the repository documents them.
|
||||
|
||||
## Final Scientific Position
|
||||
| Phase / session | Objective | Key outputs | Validation |
|
||||
|---|---|---|---|
|
||||
| Planning and architecture | freeze scope, protocol, and project structure | `AGENTS.md`, `planningawal/PROTEAN_Revisi_Lurus.md`, `docs/research/IMPLEMENTATION_PLAN.md` | checkpoint commit |
|
||||
| Baseline reproduction | implement reproducible PROTEAN baseline pipeline | `docs/baseline/BASELINE_IMPLEMENTATION.md`, `src/protean_research/` | `docs/baseline/BASELINE_VALIDATION.md` |
|
||||
| Baseline validation + regression tests | verify baseline behavior and invariants | `tests/`, `docs/baseline/PROTEAN_BASELINE_INVESTIGATION.md` | unit/integration tests |
|
||||
| Support-weighted aggregation | implement beta-controlled aggregator | `docs/research/BETA_AGGREGATION_IMPLEMENTATION.md`, `src/protean_research/aggregation/` | aggregation invariance tests |
|
||||
| Session 5 — one paired condition | first full paired beta run (single seed) | `docs/research/PAIRED_BETA_VALIDATION.md` | `val_selection.json` |
|
||||
| Session 6 — multi-seed (X-IIoTID a0.50) | pooled validation across seeds `{0,1,2}` | `docs/research/MULTISEED_BETA_VALIDATION.md`, `SESSION6_AUDIT.md` | audit PASS with minor corrections |
|
||||
| Session 7 — X-IIoTID a0.25 | third X-IIoTID alpha condition | `docs/research/ALPHA025_BETA_VALIDATION.md`, `SESSION7_AUDIT.md` | audit PASS with minor corrections |
|
||||
| Session 8 — X-IIoTID a0.75 | complete X-IIoTID three-alpha study | `docs/research/ALPHA075_BETA_VALIDATION.md`, `SESSION8_AUDIT.md` | audit PASS |
|
||||
| Session 9 — 5G-NIDD a0.50 | first 5G-NIDD condition | `docs/research/5GNIDD_ALPHA050_BETA_VALIDATION.md`, `SESSION9_AUDIT.md` | audit PASS |
|
||||
| Session 10 — 5G-NIDD a0.75 | second 5G-NIDD condition | `docs/research/5GNIDD_ALPHA075_BETA_VALIDATION.md`, `SESSION10_AUDIT.md` | audit PASS |
|
||||
| Session 11 — 5G-NIDD a0.25 | final condition; completes the grid | `docs/research/5GNIDD_ALPHA025_BETA_VALIDATION.md`, `SESSION11_AUDIT.md` | audit PASS |
|
||||
| Session 12 — synthesis | consolidate all six conditions | `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md`, `CROSS_DATASET_BETA_ANALYSIS.md`, `docs/thesis/*`, `artifacts/analysis/final__*`, `artifacts/figures/final/*` | independent recomputation, zero mismatches |
|
||||
| Proposal phase | write the thesis proposal and audit aids | `docs/proposal/*` | committed `e27c5a8` |
|
||||
| Handoff / README | keep repository state current | `docs/handoff/SESSION_HANDOFF.md`, `README.md` | this document |
|
||||
|
||||
High-level, evidence-safe conclusions only (full tables live in the synthesis documents):
|
||||
## 7. Experiment Coverage and Reproducibility
|
||||
|
||||
- Realized support imbalance, missing client-class cells, and client-size ratio vary by condition;
|
||||
Dirichlet `alpha` is a design knob, not the realized heterogeneity itself.
|
||||
- Increasing `beta` changes the normalized contribution weights and therefore the position of each
|
||||
global prototype; for a fixed snapshot, displacement grows with `beta`.
|
||||
- The rare-class detection response is **condition-dependent and non-monotone** in `beta`; the direct
|
||||
aggregation effect and the end-to-end training effect are distinct.
|
||||
- **No universal beta exists.** The validation-selected beta differs across datasets and non-IID
|
||||
conditions and is not monotone in `alpha`.
|
||||
- Stronger support weighting can **help, have limited effect, or hurt** rare-class behavior,
|
||||
depending on the condition, and can redistribute benefit across clients within one condition.
|
||||
**Verified run inventory.** Counted from `results/` metadata, not from filenames alone:
|
||||
|
||||
Not supported by the evidence (must not be claimed): support weighting always improves rare-class
|
||||
detection; the selected beta universally beats `original_protean`; `alpha` causally determines beta;
|
||||
`N_ij` measures prototype quality; larger prototype displacement means better detection; a global
|
||||
optimum exists; results generalize beyond the two evaluated datasets.
|
||||
- non-smoke run directories: **108**
|
||||
- distinct `(dataset, alpha, seed)` cells: **18** (2 × 3 × 3)
|
||||
- configurations per cell: **1 `original_protean` + 5 `support_weighted`** (beta 0/0.25/0.50/0.75/1.00)
|
||||
- paired comparison: within every cell, all six runs share **one manifest content hash** (verified);
|
||||
`original_protean` and `support_weighted beta=0` additionally share a byte-identical round-0 snapshot
|
||||
- failed runs: **none**; one unrelated `smoke` directory is excluded
|
||||
|
||||
## Repository Structure
|
||||
**Known artifact gap.** One run, `xiiotid__alpha0.5__seed0__original_protean`, lacks its entire
|
||||
validation-side artifact set (`val_global_metrics.json`, `val_confusion_matrix.json`,
|
||||
`val_per_class.csv`, `val_per_client.csv`, `val_selection.json`). Its test artifacts and history are
|
||||
present, and it is not a selection run, so all 108 test evaluations and all six selections are
|
||||
unaffected. Every other run has the complete test and validation artifact set. This is recorded as
|
||||
an evidence gap rather than silently omitted.
|
||||
|
||||
**Reproducibility controls.**
|
||||
|
||||
- **Frozen protocol:** `docs/research/EXPERIMENT_PROTOCOL.md` (machine-readable
|
||||
`configs/experiments/experiment_protocol.yaml`). Values are either FROZEN from the paper or
|
||||
explicit PROJECT DECISIONs.
|
||||
- **Frozen manifests:** 18 deterministic Dirichlet partition manifests under `artifacts/partitions/`;
|
||||
generated once and reused. Regenerating a fresh split for a run is a protocol violation.
|
||||
- **Raw data:** external and read-only (`~/datasets/…`); never modified, copied, or committed.
|
||||
SHA-256 hashes are recorded in the audit documents.
|
||||
- **Validation/test separation:** `beta` is selected from validation only; every run records
|
||||
`test_used_for_selection = false`.
|
||||
- **Seeds** control split, Dirichlet partition, model initialization, and batch order.
|
||||
|
||||
## 8. Evaluation Metrics
|
||||
|
||||
All metrics below are implemented in `src/protean_research/evaluation/metrics.py` and persisted per
|
||||
run. No scikit-learn `classification_report` is used; metrics are pure NumPy.
|
||||
|
||||
| Metric | Definition | Level | Split | Where stored |
|
||||
|---|---|---|---|---|
|
||||
| Accuracy | `trace(CM) / sum(CM)` | global | val, test | `{split}_global_metrics.json` |
|
||||
| Macro-F1 | unweighted mean of per-class F1 over classes with support > 0 | global | val, test | `{split}_global_metrics.json` |
|
||||
| Weighted-F1 | support-weighted mean of per-class F1 | global | val, test | `{split}_global_metrics.json` |
|
||||
| Per-class precision/recall/F1/support | one-vs-rest from the confusion matrix | per class (global) | val, test | `{split}_per_class.csv` |
|
||||
| Rare-class macro-F1 | mean per-class F1 over the **evaluated rare classes** (union of client rare classes with support) | global | val, test | `{split}_global_metrics.json` |
|
||||
| Rare-class recall | mean recall over the evaluated rare classes | global | val, test | `{split}_global_metrics.json` |
|
||||
| Rare support | number of eval samples in evaluated rare classes | global | val, test | `{split}_global_metrics.json` |
|
||||
| Per-client accuracy/macro-F1 | standard metrics on each client's own eval partition | per client | val, test | `per_client.csv`, `val_per_client.csv` |
|
||||
| Per-client rare-class macro-F1 / recall | rare metrics using that client's own rare class ids | per client | val, test | `per_client.csv`, `val_per_client.csv` |
|
||||
| Selection score | mean over eligible clients of per-client validation rare-class macro-F1, pooled across seeds | per condition | validation | `val_selection.json`, aggregation artifacts |
|
||||
| Prototype displacement | `‖Cbar_j(beta) − Cbar_j(0)‖` and relative displacement, fixed round-0 snapshot | per class | n/a | `*__prototype_effect*.csv` |
|
||||
| Normalized weight | `N_ij^beta / Σ_k N_kj^beta` | per client-class | n/a | `*__prototype_weights.csv` |
|
||||
| Support CV / missing cells / client-size ratio | realized non-IID statistics from `N_ij` | per class / partition | training | `characterization.csv`, `*partition_heterogeneity*` |
|
||||
|
||||
**Averaging-level distinctions that must not be interchanged.**
|
||||
|
||||
- **Global rare-class macro-F1** pools all rare-class evaluation samples; **per-client mean
|
||||
rare-class macro-F1** averages each client's own score. The **selection metric is the per-client
|
||||
mean, pooled across seeds**.
|
||||
- **Validation** metrics choose `beta`; **test** metrics are reported only after selection.
|
||||
- **Per-seed** values and **pooled** values are different quantities; the pooled score averages
|
||||
per-seed per-client-mean scores.
|
||||
|
||||
**Not computed by this repository:** micro-F1, ROC/AUC, statistical significance tests, and
|
||||
persisted per-class unseen-class recall (unseen-class evaluation is defined in code but only the
|
||||
unseen flags and union counts are saved).
|
||||
|
||||
## 9. Quantitative Experimental Results
|
||||
|
||||
All test values are means over seeds `{0,1,2}` for the selected `beta`, and the corresponding
|
||||
`original_protean` test means. Values are from `artifacts/analysis/final__six_condition_master.csv`
|
||||
and `final__selected_vs_original.csv`, and were independently re-checked against the per-run
|
||||
`test_global_metrics.json`.
|
||||
|
||||
### 9.1 Pooled validation selection (validation-only)
|
||||
|
||||
Pooled validation per-client-mean rare-class macro-F1 (selected in bold):
|
||||
|
||||
| beta | X-IIoTID a0.25 | X-IIoTID a0.50 | X-IIoTID a0.75 | 5G-NIDD a0.25 | 5G-NIDD a0.50 | 5G-NIDD a0.75 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | **0.5917** | 0.5444 | 0.4717 | **0.7615** | 0.6872 | 0.7781 |
|
||||
| 0.25 | 0.5887 | 0.5554 | 0.4744 | 0.7493 | 0.6451 | 0.7905 |
|
||||
| 0.50 | 0.5859 | 0.5673 | **0.4767** | 0.7257 | 0.6747 | **0.8091** |
|
||||
| 0.75 | 0.5795 | **0.5794** | 0.4725 | 0.7222 | 0.6884 | 0.7915 |
|
||||
| 1.00 | 0.5321 | 0.5735 | 0.4686 | 0.7376 | **0.6924** | 0.8043 |
|
||||
|
||||
### 9.2 Selected beta versus original PROTEAN (test)
|
||||
|
||||
Test rare-class macro-F1 (mean over 3 seeds), selected `beta` vs `original_protean`:
|
||||
|
||||
| Dataset | alpha | selected beta | test rare-F1 (selected) | test rare-F1 (original) | delta | test macro-F1 (selected) | test accuracy (selected) |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| X-IIoTID | 0.25 | 0.00 | 0.6321 | 0.6328 | −0.0007 | 0.6813 | 0.8740 |
|
||||
| X-IIoTID | 0.50 | 0.75 | 0.6655 | 0.6425 | **+0.0230** | 0.7312 | 0.9064 |
|
||||
| X-IIoTID | 0.75 | 0.50 | 0.6052 | 0.6039 | +0.0014 | 0.7576 | 0.9281 |
|
||||
| 5G-NIDD | 0.25 | 0.00 | 0.8412 | 0.8423 | −0.0010 | 0.8166 | 0.7302 |
|
||||
| 5G-NIDD | 0.50 | 1.00 | 0.8720 | 0.8805 | −0.0086 | 0.8386 | 0.7433 |
|
||||
| 5G-NIDD | 0.75 | 0.50 | 0.8967 | 0.8799 | +0.0168 | 0.8531 | 0.7523 |
|
||||
|
||||
Reading of the table:
|
||||
|
||||
- The six conditions split into **three mutually exclusive categories that total six**:
|
||||
**three positive deltas** (X-IIoTID a0.50, X-IIoTID a0.75, 5G-NIDD a0.75), **one negative delta**
|
||||
(5G-NIDD a0.50), and **two effectively unchanged** (both `alpha = 0.25` conditions, where the
|
||||
selected `beta` is 0, so the selected run is the equal-weight run and the difference is
|
||||
floating-point divergence only).
|
||||
- The **largest positive delta (+0.0230, X-IIoTID a0.50) is still smaller than that condition's
|
||||
between-seed sample std (0.0267)**. No statistical superiority is claimed.
|
||||
- **No single beta dominates** across conditions, and the selection depends on both dataset and
|
||||
condition.
|
||||
|
||||
Cross-dataset metric levels are not comparable: 5G-NIDD rare classes are low-volume flood/scan
|
||||
types, X-IIoTID rare classes are low-volume behavioural categories, and their rare support differs.
|
||||
|
||||
### 9.3 Selected betas
|
||||
|
||||
| Dataset | alpha=0.25 | alpha=0.50 | alpha=0.75 |
|
||||
|---|---|---|---|
|
||||
| X-IIoTID | 0.00 | 0.75 | 0.50 |
|
||||
| 5G-NIDD | 0.00 | 1.00 | 0.50 |
|
||||
|
||||
## 10. Beta Selection by Dataset and Alpha
|
||||
|
||||
Selection is by pooled validation per-client-mean rare-class macro-F1 (validation only); the test
|
||||
set is never used. `artifacts/analysis/final__selection_strength.csv` quantifies separation:
|
||||
|
||||
| Dataset | alpha | selected beta | best | 2nd best | abs gap | top-2 gap / selected std | per-seed diagnostic betas |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| X-IIoTID | 0.25 | 0.00 | 0.5917 | 0.5887 (0.25) | 0.0030 | 0.034 | 0.75 / 0.50 / 0.00 |
|
||||
| X-IIoTID | 0.50 | 0.75 | 0.5794 | 0.5735 (1.00) | 0.0059 | 0.078 | 0.50 / 1.00 / 0.75 |
|
||||
| X-IIoTID | 0.75 | 0.50 | 0.4767 | 0.4744 (0.25) | 0.0024 | 0.051 | 0.50 / 0.50 / 0.00 |
|
||||
| 5G-NIDD | 0.25 | 0.00 | 0.7615 | 0.7493 (0.25) | 0.0122 | 0.556 | 0.25 / 0.00 / 0.00 |
|
||||
| 5G-NIDD | 0.50 | 1.00 | 0.6924 | 0.6884 (0.75) | 0.0040 | 0.029 | 1.00 / 0.00 / 1.00 |
|
||||
| 5G-NIDD | 0.75 | 0.50 | 0.8091 | 0.8043 (1.00) | 0.0048 | 0.081 | 0.50 / 1.00 / 1.00 |
|
||||
|
||||
In **all six conditions the top-two gap is smaller than the selected beta's across-seed sample std**
|
||||
(ratios 0.03–0.56), so each selection is a *small numerical separation relative to seed variability*.
|
||||
No tie-break was ever exercised. Per-seed diagnostic betas are shown only for stability; they are not
|
||||
the selection rule.
|
||||
|
||||
## 11. Prototype-Level Analysis
|
||||
|
||||
Direct aggregation effect, measured on fixed round-0 snapshots (`C_ij`, `N_ij` held constant, no
|
||||
retraining). `artifacts/analysis/final__prototype_displacement_by_beta.csv`.
|
||||
|
||||
Mean `‖Cbar_j(beta) − Cbar_j(0)‖` across classes:
|
||||
|
||||
| Dataset | alpha | beta=0.25 | beta=0.50 | beta=0.75 | beta=1.00 |
|
||||
|---|---|---|---|---|---|
|
||||
| X-IIoTID | 0.25 | 1.476 | 2.403 | 3.003 | **3.422** |
|
||||
| X-IIoTID | 0.50 | 0.826 | 1.374 | 1.780 | **2.099** |
|
||||
| X-IIoTID | 0.75 | 0.477 | 0.829 | 1.097 | **1.305** |
|
||||
| 5G-NIDD | 0.25 | 1.295 | 2.055 | 2.559 | **2.919** |
|
||||
| 5G-NIDD | 0.50 | 0.953 | 1.567 | 2.003 | **2.340** |
|
||||
| 5G-NIDD | 0.75 | 0.497 | 0.875 | 1.163 | **1.385** |
|
||||
|
||||
- Displacement is **monotone increasing in `beta`** for a fixed snapshot in all six conditions
|
||||
(mechanically expected: larger `beta` shifts weight toward high-`N_ij` clients).
|
||||
- Displacement at `beta = 1` **decreases as alpha rises**, tracking realized support imbalance.
|
||||
- At matched alpha, X-IIoTID has larger displacement at a0.25 (3.422 vs 2.919) while 5G-NIDD is
|
||||
slightly larger at a0.50 (2.340 vs 2.099) and a0.75 (1.385 vs 1.305); per-class ordering is
|
||||
seed- and dataset-specific, and rare-class identities differ.
|
||||
- **Displacement is a representation-aggregation metric, not detection quality.** No
|
||||
displacement-to-accuracy inference is made.
|
||||
|
||||
Normalized contribution weights are stored per client-class in
|
||||
`artifacts/analysis/*__prototype_weights.csv`; per-class displacement is in
|
||||
`final__prototype_displacement_by_class.csv`.
|
||||
|
||||
## 12. Rare-Class Detection Results
|
||||
|
||||
Rare classes are defined from each client's **training** support (the two attack classes with the
|
||||
smallest positive `N_ij`; one if only one; the client is excluded if none). Unseen classes
|
||||
(`N_ij = 0`) are treated separately and never merged with rare classes.
|
||||
|
||||
Key observations from the test results (Section 9.2) and the synthesis:
|
||||
|
||||
- Selected rare-class macro-F1 deltas are **mixed in sign and small**; the largest positive
|
||||
difference is within seed noise.
|
||||
- Selected rare-class recall is **slightly lower than original in most conditions** while macro-F1
|
||||
may be higher: weighting can trade a little rare-class recall for precision.
|
||||
- At both `alpha = 0.25` conditions the selected `beta` is 0, so those conditions cannot demonstrate
|
||||
any support-weighting benefit.
|
||||
- Full rare-class curves and per-condition reports: `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md`,
|
||||
the per-condition `*_BETA_VALIDATION.md` reports, and `final__selected_vs_original.csv`.
|
||||
|
||||
## 13. Per-Client Analysis
|
||||
|
||||
Per-client effects are computed on each client's own evaluation partition relative to `beta = 0`
|
||||
(`artifacts/analysis/final__client_effect_summary.csv`; detail in the per-condition
|
||||
`*__multiseed__client_validation_*.csv`).
|
||||
|
||||
Improve / degrade / unchanged clients at the **selected** beta (validation, vs `beta=0`), by seed:
|
||||
|
||||
| Condition (selected beta) | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| X-IIoTID a0.25 (0.00) | 0 / 0 / 10 | 0 / 0 / 10 | 0 / 0 / 10 |
|
||||
| X-IIoTID a0.50 (0.75) | 7 / 3 / 0 | 8 / 1 / 1 | 8 / 1 / 1 |
|
||||
| X-IIoTID a0.75 (0.50) | 6 / 4 / 0 | 9 / 0 / 1 | 6 / 4 / 0 |
|
||||
| 5G-NIDD a0.25 (0.00) | 0 / 0 / 10 | 0 / 0 / 10 | 0 / 0 / 10 |
|
||||
| 5G-NIDD a0.50 (1.00) | 6 / 3 / 1 | 3 / 5 / 2 | 7 / 2 / 1 |
|
||||
| 5G-NIDD a0.75 (0.50) | 4 / 6 / 0 | 6 / 3 / 1 | 4 / 2 / 4 |
|
||||
|
||||
- Weighting **redistributes benefit across clients**: within a condition some clients improve while
|
||||
others degrade.
|
||||
- At `beta = 1`, mean per-client delta is most negative at both `alpha = 0.25` conditions
|
||||
(e.g. X-IIoTID a0.25 ≈ −0.060, 5G-NIDD a0.25 ≈ −0.024), consistent with those conditions
|
||||
selecting `beta = 0`.
|
||||
- `N_ij` versus per-client delta is **non-monotonic**: the worst-degrading client is not always the
|
||||
lowest-support client.
|
||||
- Client identities are per-seed realizations and are **not pooled** across seeds.
|
||||
|
||||
## 14. Cross-Alpha and Cross-Dataset Findings
|
||||
|
||||
- **Matched-alpha selections agree at alpha=0.25 (both 0.00) and alpha=0.75 (both 0.50)** and differ
|
||||
at alpha=0.50 (X-IIoTID 0.75 vs 5G-NIDD 1.00). Across the six conditions the selected beta takes
|
||||
values `{0.00, 0.50, 1.00}` — **not universal, not monotone in alpha, not a function of alpha
|
||||
alone**.
|
||||
- **Realized heterogeneity** (support CV, missing client-class cells, client-size ratio) decreases as
|
||||
alpha rises within each dataset, but not strictly per seed. Dirichlet alpha is a design knob, not
|
||||
the realized heterogeneity itself.
|
||||
- Displacement at `beta = 1` falls with alpha on both datasets.
|
||||
- **Rare-set composition differs by dataset**, so absolute metric levels are **not comparable across
|
||||
datasets**; only within-condition differences are meaningful.
|
||||
- **No cross-dataset causality, superiority, or dataset-independent beta law is claimed.**
|
||||
|
||||
Reports: `docs/research/CROSS_DATASET_BETA_ANALYSIS.md`, `XIIOTID_CROSS_ALPHA_ANALYSIS.md`,
|
||||
`5GNIDD_CROSS_ALPHA_ANALYSIS.md`.
|
||||
|
||||
## 15. Confusion Matrix and Classification Breakdown
|
||||
|
||||
**Confusion matrices do exist in the saved artifacts.** They are written per run by the evaluation
|
||||
code (`confusion_matrix` in `src/protean_research/evaluation/metrics.py`) and saved as:
|
||||
|
||||
- `results/<run>/test_confusion_matrix.json` — **108 / 108 runs**
|
||||
- `results/<run>/val_confusion_matrix.json` — **107 / 108 runs** (missing only for
|
||||
`xiiotid__alpha0.5__seed0__original_protean`, which also lacks the other validation artifacts)
|
||||
|
||||
**Properties of the stored matrices.**
|
||||
|
||||
| Property | Value |
|
||||
|---|---|
|
||||
| Shape | 9 × 9 (K = 9 classes) |
|
||||
| Values | **raw integer counts** (not normalized) |
|
||||
| Scope | **global** over the evaluation split, not per-client |
|
||||
| Restriction | full multiclass; **not** restricted to rare classes |
|
||||
| Split | separate `test_` and `val_` matrices |
|
||||
| Class order | X-IIoTID: `C&C, Exfiltration, Exploitation, Lateral_movement, Normal, RDOS, Reconnaissance, Tampering, Weaponization`; 5G-NIDD: `Benign, HTTPFlood, ICMPFlood, SYNFlood, SYNScan, SlowrateDoS, TCPConnectScan, UDPFlood, UDPScan` (also in each run's `metadata.json`) |
|
||||
| Aggregation | each client's eval partition concatenated; no seed/client mixing |
|
||||
|
||||
Representative interpretation — 5G-NIDD, `alpha=0.75`, `seed=0`, **test**, original vs selected
|
||||
(`beta=0.50`) — using actual counts:
|
||||
|
||||
| Class | eval support | original recall | selected-beta recall |
|
||||
|---|---|---|---|
|
||||
| Benign | 95,547 | 0.3795 | 0.3825 |
|
||||
| HTTPFlood | 28,162 | 0.9729 | 0.9923 |
|
||||
| ICMPFlood | 231 | 1.0000 | 1.0000 |
|
||||
| SYNFlood | 1,944 | 0.8570 | 0.8570 |
|
||||
| SYNScan | 4,009 | 0.9968 | 0.9968 |
|
||||
| SlowrateDoS | 14,625 | 0.9217 | 0.9054 |
|
||||
| TCPConnectScan | 4,010 | 0.9948 | 0.9948 |
|
||||
| UDPFlood | 91,468 | 1.0000 | 1.0000 |
|
||||
| UDPScan | 3,181 | 0.9975 | 0.9981 |
|
||||
|
||||
Largest off-diagonal cells (true -> predicted):
|
||||
|
||||
| true | predicted | original count | selected-beta count |
|
||||
|---|---|---|---|
|
||||
| Benign | UDPFlood | 56,899 | 57,054 |
|
||||
| Benign | TCPConnectScan | 2,015 | 1,495 |
|
||||
| SlowrateDoS | HTTPFlood | 1,079 | 1,308 |
|
||||
| HTTPFlood | SlowrateDoS | 759 | 212 |
|
||||
| SYNFlood | TCPConnectScan | 278 | 278 |
|
||||
|
||||
What these counts support:
|
||||
|
||||
- The dominant error mode is **Benign misclassified as UDPFlood** (about 57k samples), which
|
||||
dominates the off-diagonal mass. It is essentially unchanged by `beta`, so `beta` does not fix
|
||||
this confusion. This is visible in low Benign recall (≈0.38) with very high UDPFlood recall
|
||||
(1.0000), i.e. UDPFlood is an attractor for Benign.
|
||||
- The change from original to selected `beta` shifts a few specific off-diagonal cells
|
||||
(`Benign → TCPConnectScan` decreases; `SlowrateDoS → HTTPFlood` increases), consistent with the
|
||||
small, mixed-sign aggregate deltas, **not** with a systematic improvement.
|
||||
- Diagonal versus off-diagonal entries must be read together: high diagonal on the rare/scan classes
|
||||
coexists with the large Benign confusion, which is why accuracy is moderate while macro-F1 is
|
||||
higher.
|
||||
|
||||
Reconstruction note: because only aggregate per-class metrics (precision/recall/F1/support) are
|
||||
persisted, a confusion matrix cannot be uniquely reconstructed from them; the raw `y_true`/`y_pred`
|
||||
arrays are not saved. Reconstruction is unnecessary here because the matrices themselves are already
|
||||
persisted as JSON.
|
||||
|
||||
Representative confusion-matrix files:
|
||||
|
||||
- `results/5gnidd__alpha0.75__seed0__original_protean/test_confusion_matrix.json`
|
||||
- `results/5gnidd__alpha0.75__seed0__support_weighted__beta0.5/test_confusion_matrix.json`
|
||||
- `results/xiiotid__alpha0.25__seed0__original_protean/test_confusion_matrix.json`
|
||||
|
||||
## 16. Final Figures and Visualizations
|
||||
|
||||
All figures are under `artifacts/figures/final/` and load from the `final__*` analysis tables. They
|
||||
were generated with matplotlib; no training was involved.
|
||||
|
||||
**Fig A — pooled validation rare-class macro-F1 vs beta** (per dataset). Axes: `beta` (x) vs pooled
|
||||
validation per-client-mean rare-class macro-F1 (y); one line per alpha. Evidence: the validation
|
||||
curve is non-monotone and the selected `beta` differs by condition.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
**Fig B — selected beta by condition.** Bar chart of the six validation-selected betas; shows that
|
||||
selections take `{0.00, 0.50, 1.00}` and are not universal.
|
||||
|
||||
|
||||
|
||||
**Fig C — realized support CV.** Condition-level mean class-support CV; shows realized imbalance
|
||||
decreases as alpha rises.
|
||||
|
||||
|
||||
|
||||
**Fig D — prototype displacement at beta=0 vs beta=1.** Fixed-snapshot mean displacement per class;
|
||||
shows displacement grows with `beta` and larger where support imbalance is larger.
|
||||
|
||||
|
||||
|
||||
**Fig E — support CV vs displacement.** Scatter of realized support CV (x) vs beta=1 displacement
|
||||
(y) across conditions; a descriptive co-variation, not a causal relation.
|
||||
|
||||
|
||||
|
||||
**Fig F — selected vs original test rare-class macro-F1.** Paired comparison per condition; shows
|
||||
small, mixed-sign differences.
|
||||
|
||||
|
||||
|
||||
**Fig G — per-client effect at the selected beta.** Per-client validation delta relative to
|
||||
`beta = 0`; shows within-condition heterogeneity across clients.
|
||||
|
||||
|
||||
|
||||
## 17. Main Scientific Findings
|
||||
|
||||
1. **Changing `beta` changes the global prototype.** Contribution weights become proportional to
|
||||
support, and for a fixed snapshot mean prototype displacement grows monotonically with `beta` in
|
||||
every condition.
|
||||
2. **The rare-class response is condition-dependent and non-monotone in `beta`.** The direct
|
||||
aggregation effect (geometry) and the end-to-end training effect (detection) are distinct.
|
||||
3. **No universal beta was found.** The validation-selected `beta` differs across datasets and
|
||||
conditions and is not monotone in `alpha`. Every selection is weak relative to seed variability.
|
||||
4. **Selected-vs-original differences are small and mixed in sign.** No statistical superiority is
|
||||
established; the largest positive difference is within seed noise.
|
||||
5. **Weighting redistributes benefit across clients.** Within a condition some clients improve while
|
||||
others degrade; the effect is not simply "more support is better".
|
||||
6. **Where weighting does not help:** both `alpha = 0.25` conditions select `beta = 0` (equal
|
||||
weighting), and at `beta = 1` the per-client effect is most damaging there.
|
||||
7. **Realized support imbalance and prototype displacement co-vary** descriptively; displacement is
|
||||
not a proxy for detection quality.
|
||||
8. **The dominant classification error (Benign → UDPFlood) is not addressed by `beta`.**
|
||||
|
||||
## 18. Limitations and Novelty Boundary
|
||||
|
||||
**Limitations.**
|
||||
|
||||
- The `S_j` participant/denominator rule is a **documented reproduction assumption** (no verified
|
||||
official PROTEAN code). It is material in up to 19/27 class-condition cells; all results are
|
||||
conditional on it.
|
||||
- `beta = 0` is only arithmetically the equal mean over `S_j`; it is **not bitwise identical** to
|
||||
`original_protean` (floating-point divergence; equivalence class B in every condition).
|
||||
- Two datasets, three alpha values, three project-defined seeds, 10 clients, one architecture.
|
||||
- The `beta` grid is discrete; intermediate values were not tested.
|
||||
- Three seeds are underpowered for inferential testing; no significance test is claimed.
|
||||
- Prototype displacement is a representation metric, not detection quality.
|
||||
- One validation artifact set is missing (Section 7); rare-class and unseen-class results cannot be
|
||||
merged, and cross-dataset absolute metric levels are not comparable.
|
||||
|
||||
**Novelty boundary.** Support-weighted prototype aggregation is not new (e.g. FedProto). The
|
||||
contribution is the controlled `beta` parameterization inside PROTEAN, the validation-only
|
||||
condition-wise selection, the paired multi-seed design, and the behavioral findings. No claim is
|
||||
made that the method universally improves PROTEAN.
|
||||
|
||||
## 19. Repository Structure
|
||||
|
||||
```
|
||||
src/protean_research/ data, models, aggregation, federated, evaluation, utils
|
||||
scripts/ training, partition, and analysis scripts
|
||||
configs/ datasets, experiments, model configuration
|
||||
docs/proposal/ thesis proposal and its internal audit aids
|
||||
docs/research/ frozen protocol, per-condition reports, cross-alpha / cross-dataset synthesis
|
||||
src/protean_research/ data loading/preprocessing/partitioning, models, aggregation, federated
|
||||
round logic, evaluation metrics, analysis, utils
|
||||
scripts/ training runner, partition generation, and read-only analysis scripts
|
||||
configs/ dataset, experiment protocol, and model configuration (YAML)
|
||||
docs/proposal/ thesis proposal and internal audit aids
|
||||
docs/research/ frozen protocol, per-condition reports, cross-alpha/cross-dataset synthesis,
|
||||
session audits
|
||||
docs/baseline/ PROTEAN baseline investigation, implementation, and validation
|
||||
docs/thesis/ Chapter 4 blueprint and claims register
|
||||
docs/baseline/ PROTEAN baseline investigation and validation
|
||||
docs/handoff/ session handoff (start here for a new session)
|
||||
planningawal/ immutable research specification (do not modify)
|
||||
artifacts/partitions/ deterministic frozen Dirichlet partition manifests (18)
|
||||
artifacts/analysis/ committed machine-readable analysis outputs
|
||||
artifacts/figures/final/ thesis-ready figures
|
||||
artifacts/partitions/ deterministic frozen Dirichlet partition manifests
|
||||
results/ local run outputs (git-ignored, intentionally not committed)
|
||||
results/ local run outputs (git-ignored; 108 non-smoke run dirs + 1 smoke)
|
||||
```
|
||||
|
||||
## Key Scripts
|
||||
|
||||
| Script | Role |
|
||||
|---|---|
|
||||
| `scripts/run_baseline.py` | full baseline / support-weighted training runner (frozen; grid complete) |
|
||||
| `scripts/generate_partitions.py` | deterministic frozen Dirichlet manifests |
|
||||
| `scripts/analyze_multiseed_beta.py` | pooled validation beta selection |
|
||||
| `scripts/analyze_partition_heterogeneity.py` | realized non-IID characterization |
|
||||
| `scripts/analyze_prototype_effect.py` | fixed-snapshot prototype displacement |
|
||||
| `scripts/analyze_cross_alpha.py` | cross-alpha synthesis |
|
||||
| `scripts/verify_beta0_equivalence.py` | beta=0 vs original semantic/numerical audit |
|
||||
| `scripts/validate_sj_participant_rule.py` | `S_j` reproduction-assumption analysis |
|
||||
|
||||
## Important Project Documents
|
||||
|
||||
**Proposal**
|
||||
|
||||
- `docs/proposal/THESIS_PROPOSAL_DRAFT.md` — final Markdown proposal source for the current scope
|
||||
- `docs/proposal/PROPOSAL_SOURCE_MAP.md` — claim-to-evidence map (internal)
|
||||
- `docs/proposal/LITERATURE_MATRIX.md` — literature matrix (internal)
|
||||
- `docs/proposal/REFERENCES_VERIFIED.md` — bibliography verification record (internal)
|
||||
|
||||
**Research synthesis**
|
||||
|
||||
- `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md` — authoritative six-condition synthesis
|
||||
- `docs/research/CROSS_DATASET_BETA_ANALYSIS.md` — cross-dataset comparison
|
||||
- `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md` — X-IIoTID cross-alpha analysis
|
||||
- `docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md` — 5G-NIDD cross-alpha analysis
|
||||
- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen reproduction protocol
|
||||
|
||||
**Thesis support**
|
||||
|
||||
- `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md` — Chapter 4 blueprint
|
||||
- `docs/thesis/CLAIMS_REGISTER.md` — claim / evidence / status register
|
||||
|
||||
**Handoff and planning**
|
||||
|
||||
- `docs/handoff/SESSION_HANDOFF.md` — current state and next steps (start here)
|
||||
- `planningawal/PROTEAN_Revisi_Lurus.md` — immutable research specification
|
||||
|
||||
> `planningawal/` is **immutable**. Read it, cite it, but never edit, rename, move, or add files to
|
||||
> it.
|
||||
|
||||
## Testing and Reproducibility
|
||||
|
||||
```bash
|
||||
# unit + integration tests (expected: 75 passed)
|
||||
.venv/bin/python -m pytest
|
||||
```
|
||||
|
||||
- **Paired comparison:** within each `(dataset, alpha, seed)`, every compared configuration reuses
|
||||
the same partition, split, class mapping, and model initialization.
|
||||
- **Frozen manifests:** partitions are generated once and reused; regenerating a fresh Dirichlet
|
||||
split is a protocol violation.
|
||||
- **Immutable raw data:** datasets live outside the repository and are never modified or committed.
|
||||
- **Validation-only selection:** beta is chosen from validation data only; test data is never used
|
||||
for any model-selection decision.
|
||||
- **Machine-readable outputs:** analysis is stored as CSV/JSON under `artifacts/analysis/`, and the
|
||||
final figures load from those artifacts.
|
||||
|
||||
## Frozen / Do Not Modify
|
||||
|
||||
Treat the following as finished unless there is an explicit, documented reason:
|
||||
|
||||
- `planningawal/` — immutable research source.
|
||||
- Raw datasets under `~/datasets/` — read-only, external, never copied into the repository.
|
||||
- Completed scientific results and the final synthesis documents.
|
||||
- Frozen experiment artifacts: partition manifests, analysis outputs, and final figures.
|
||||
- `src/`, model, and aggregation code used for the 108-run grid.
|
||||
- The finalized proposal Markdown.
|
||||
|
||||
## Limitations
|
||||
|
||||
- The `S_j` participant/denominator rule remains a reproduction assumption and is materially
|
||||
exercised in a substantial fraction of class-condition cells.
|
||||
- Two datasets, three alpha values, three project-defined seeds, 10 clients, one architecture.
|
||||
- The beta grid is discrete; intermediate values were not tested.
|
||||
- No inferential trend testing (three seeds are underpowered for it).
|
||||
- `beta = 0` is not bitwise identical to `original_protean` (floating-point ordering).
|
||||
- Prototype displacement is a representation metric, not detection quality.
|
||||
|
||||
## Quickstart
|
||||
## 20. Reproduction and Tests
|
||||
|
||||
```bash
|
||||
# unit + integration tests (expected: 75 passed)
|
||||
.venv/bin/python -m pytest
|
||||
|
||||
# read-only pooled validation analysis (no retraining required)
|
||||
# read-only pooled validation analysis (no retraining)
|
||||
.venv/bin/python scripts/analyze_multiseed_beta.py --dataset 5gnidd --alpha 0.25 --seeds 0 1 2
|
||||
|
||||
# read-only cross-alpha / cross-dataset synthesis
|
||||
# read-only realized-partition characterization
|
||||
.venv/bin/python scripts/analyze_partition_heterogeneity.py --dataset 5gnidd --alphas 0.25 0.5 0.75
|
||||
|
||||
# read-only cross-alpha synthesis
|
||||
.venv/bin/python scripts/analyze_cross_alpha.py --dataset 5gnidd --alphas 0.25 0.5 0.75
|
||||
|
||||
# beta=0 vs original equivalence audit (read-only)
|
||||
.venv/bin/python scripts/verify_beta0_equivalence.py --dataset 5gnidd --alpha 0.25 --seeds 0 1 2
|
||||
```
|
||||
|
||||
See `docs/handoff/SESSION_HANDOFF.md` before making any methodological change. The next work is
|
||||
academic writing and formatting, not new experiments.
|
||||
A full training run (not needed; the grid is complete) is:
|
||||
|
||||
```bash
|
||||
.venv/bin/python scripts/run_baseline.py --dataset 5gnidd --alpha 0.25 --seed 1 \
|
||||
--clients 10 --rounds 10 --local-epochs 3 \
|
||||
--aggregation-mode support_weighted --beta 1.0
|
||||
```
|
||||
|
||||
Runtime context: the 108 runs consumed ≈ 94.5 wall-clock hours on CPU (8 cores); per-run wall clock
|
||||
was ≈ 62–65 minutes for 5G-NIDD and ≈ 40 minutes for X-IIoTID (`*__multiseed__runtime.csv`). Do not
|
||||
rerun the grid.
|
||||
|
||||
**Repository safety.**
|
||||
|
||||
- `planningawal/` is immutable; raw datasets are external and read-only.
|
||||
- Do not modify or regenerate completed results, manifests, or figures.
|
||||
- Do not add an `equal_available` mode; it is arithmetically `support_weighted` at `beta = 0` under
|
||||
the shared `S_j`.
|
||||
- `beta` selection uses validation only; the test set is never used for selection.
|
||||
|
||||
## 21. Complete Artifact Index
|
||||
|
||||
Per-run outputs (`results/<dataset>__alpha<alpha>__seed<seed>__<config>/`):
|
||||
|
||||
| File | Content |
|
||||
|---|---|
|
||||
| `metadata.json` | run config, dataset/alpha/seed, mode, class names, hashes |
|
||||
| `history.json` | per-round global accuracy/macro-F1/rare-class metrics |
|
||||
| `characterization.csv` | per-class training support statistics (`N_ij`) |
|
||||
| `train_support_Nij.csv` | `N_ij` matrix (client × class) |
|
||||
| `rare_unseen.csv` | per client-class rare/unseen flags from training support |
|
||||
| `test_global_metrics.json`, `val_global_metrics.json` | global metrics per split |
|
||||
| `test_per_class.csv`, `val_per_class.csv` | per-class precision/recall/F1/support |
|
||||
| `per_client.csv`, `val_per_client.csv` | per-client metrics |
|
||||
| `test_confusion_matrix.json`, `val_confusion_matrix.json` | 9×9 raw-count confusion matrices |
|
||||
| `val_selection.json` | validation selection metric and `test_used_for_selection` flag |
|
||||
|
||||
For the one incomplete validation run, see Section 7.
|
||||
|
||||
Analysis outputs (`artifacts/analysis/`), by function:
|
||||
|
||||
| Category | Representative files |
|
||||
|---|---|
|
||||
| Final condition summary | `final__six_condition_master.csv/.json` |
|
||||
| Beta-selection curves | `final__pooled_validation_curves.csv`, `final__selection_strength.csv` |
|
||||
| Selected vs original | `final__selected_vs_original.csv` |
|
||||
| Realized heterogeneity | `final__realized_heterogeneity.csv`, `*__partition_heterogeneity_*.csv` |
|
||||
| Prototype displacement | `final__prototype_displacement_by_beta.csv`, `..._by_class.csv` |
|
||||
| Per-client effects | `final__client_effect_summary.csv`, `*__multiseed__client_validation_*.csv` |
|
||||
| Per-condition validation | `*__multiseed__validation_per_seed.csv`, `..._validation_pooled.csv` |
|
||||
| Per-condition test | `*__multiseed__test_descriptive.csv` |
|
||||
| Cross-alpha | `{xiiotid,5gnidd}__cross_alpha__*.csv/.json` |
|
||||
| Cross-dataset | `cross_dataset__matched_alpha_summary.csv/.json`, `cross_dataset__beta_curve.csv` |
|
||||
| beta=0 equivalence | `*__multiseed__beta0_equivalence.csv` |
|
||||
| Runtime | `*__multiseed__runtime.csv` |
|
||||
| Prototype snapshots & weights | `*__prototype_snapshot.json`, `*__prototype_weights.csv`, `*__prototype_effect*.csv` |
|
||||
|
||||
Partition manifests: `artifacts/partitions/<dataset>__alpha<alpha>__seed<seed>.json` (18 files).
|
||||
Figures: `artifacts/figures/final/figA–figG*.png` (8 files).
|
||||
|
||||
Key documents:
|
||||
|
||||
| Document | Purpose |
|
||||
|---|---|
|
||||
| `docs/research/EXPERIMENT_PROTOCOL.md` | frozen reproduction protocol |
|
||||
| `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md` | authoritative six-condition synthesis |
|
||||
| `docs/research/CROSS_DATASET_BETA_ANALYSIS.md` | cross-dataset comparison |
|
||||
| `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md` | X-IIoTID cross-alpha analysis |
|
||||
| `docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md` | 5G-NIDD cross-alpha analysis |
|
||||
| `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md` | Chapter 4 blueprint |
|
||||
| `docs/thesis/CLAIMS_REGISTER.md` | claim / evidence / status register |
|
||||
| `docs/proposal/THESIS_PROPOSAL_DRAFT.md` | final proposal Markdown source |
|
||||
| `docs/proposal/PROPOSAL_SOURCE_MAP.md` | claim-to-evidence map |
|
||||
| `docs/proposal/LITERATURE_MATRIX.md` | literature matrix |
|
||||
| `docs/proposal/REFERENCES_VERIFIED.md` | bibliography verification record |
|
||||
| `docs/handoff/SESSION_HANDOFF.md` | current state and next steps |
|
||||
| `planningawal/PROTEAN_Revisi_Lurus.md` | immutable research specification |
|
||||
|
||||
## 22. References
|
||||
|
||||
- Chennoufi, S., Han, Y., Blanc, G., De Cristofaro, E., & Kiennert, C. (2025). PROTEAN: Federated
|
||||
Intrusion Detection in Non-IID Environments through Prototype-Based Knowledge Sharing. ESORICS
|
||||
2025, pp. 103–125. DOI: 10.1007/978-3-032-07884-1_6. arXiv:2507.05524.
|
||||
- Tan, Y., Long, G., Liu, L., Zhou, T., Lu, Q., Jiang, J., & Zhang, C. (2022). FedProto: Federated
|
||||
Prototype Learning across Heterogeneous Clients. AAAI 36(8), 8432–8440.
|
||||
DOI: 10.1609/aaai.v36i8.20819. arXiv:2105.00243.
|
||||
|
||||
The complete, verified bibliography (including the FL foundations, non-IID, class-imbalance, and
|
||||
evaluation-metric references used in the proposal) is maintained in
|
||||
`docs/proposal/REFERENCES_VERIFIED.md` and the proposal bibliography. This README cites only the two
|
||||
works directly relevant to the mechanism and novelty boundary.
|
||||
Reference in new issue
Block a user