# PROTEAN Prototype Aggregation Study

Research code and audited artifacts for a master's thesis on **server-side prototype aggregation
in PROTEAN** (Chennoufi et al., 2025), a federated intrusion-detection framework, under non-IID
client data. The study evaluates a **beta-controlled, class-specific sample-support weighting** of
local class prototypes for **rare attack detection**.

**Research status.** The experimental grid is **complete** (108 full federated runs), the results
are synthesized and audited, and the thesis proposal Markdown is finalized. No further training is
planned unless new academic feedback explicitly requires it. The repository is in a documentation
and thesis-writing stage.

**Final title.** *Optimasi Agregasi Prototype pada PROTEAN melalui Pembobotan Berbasis Dukungan
Sampel Kelas untuk Deteksi Serangan Langka pada Data Non-IID.* Here "Optimasi" means empirical
selection of `beta` among the predefined candidate values, not a global mathematical optimum.

---

## 1. Research Overview

PROTEAN is a federated intrusion-detection method that shares **class prototypes** between clients
and the server in addition to model parameters. Each client `i` builds a local prototype `C_ij` for
class `j` from its own training samples; the server aggregates these into a global prototype
`Cbar_j` that is used both for local prototype alignment during training and as the classifier at
inference (nearest prototype in embedding space).

This thesis modifies **only the server-side prototype aggregation**. The original PROTEAN
aggregation gives every contributing client equal weight. The proposed aggregation weights each
client's prototype by `N_ij^beta`, where `N_ij` is that client's class-specific training support and
`beta` is a single exponent in `[0, 1]`. Everything else — local prototype formation, model
aggregation, architecture, loss, and inference rule — is unchanged relative to the reproduced
baseline.

`N_ij` is **class-specific sample support, not prototype quality**. A client with few samples is
not assumed to have a worse prototype.

## 2. Problem Statement and Motivation

Federated IDS data is non-IID: clients observe different traffic mixtures and different attack
types, so class support varies strongly across clients. Rare attack classes appear in little of
the data, so aggregate metrics can look acceptable while the model fails on exactly the classes
that matter for detection.

Under the original equal-weight prototype aggregation, a prototype built from a handful of samples
contributes as much as one built from thousands. This thesis investigates whether — and under which
realized non-IID conditions — weighting contributions by class-specific sample support changes the
global prototype and rare-class detection behavior.

The study deliberately isolates one mechanism so that any observed change can be attributed to the
aggregation weighting rather than to a confounded set of architectural or training changes.

## 3. Original PROTEAN versus Proposed Modification

**A. Local prototype (unchanged).** For class `j` with `N_ij > 0` samples on client `i`, the local
prototype is the mean embedding of that client's class-`j` samples:

$$
C_{ij} = \frac{1}{N_{ij}} \sum_{k=1}^{N_{ij}} z_{ijk}
$$

where `z_ijk` is the embedding of the k-th sample of class `j` on client `i`.

**B. Original server-side prototype aggregation (reference).** PROTEAN Eq. (2) averages the `M`
contributing clients equally (equal contribution, no `N_ij` weight):

$$
\bar{C}_j = \frac{1}{M} \sum_{i=1}^{M} C_{ij}
$$

**C. Proposed beta-controlled aggregation.** Only the prototype aggregation is modified:

$$
\bar{C}_j(\beta) = \frac{\sum_{i \in S_j} N_{ij}^{\beta}\, C_{ij}}{\sum_{i \in S_j} N_{ij}^{\beta}}, \qquad \beta \in \{0,\ 0.25,\ 0.50,\ 0.75,\ 1.00\}
$$

**D. Normalized contribution weight.** Client `i`'s normalized weight for class `j` is

$$
w_{ij}(\beta) = \frac{N_{ij}^{\beta}}{\sum_{k \in S_j} N_{kj}^{\beta}}
$$

**E. Symbol meanings.**

| Symbol | Meaning |
|---|---|
| `N_ij` | number of local **training** samples of class `j` on client `i` (class-specific sample support, not prototype quality) |
| `C_ij` | client `i`'s local class-`j` prototype (mean embedding) |
| `S_j` | participant set: clients with `N_ij > 0` and a valid local prototype for class `j` |
| `M` | number of participants in the reference Eq. (2) (equal average) |
| `beta` | scalar weighting exponent in `[0, 1]`, one value for all classes and clients |
| `w_ij` | normalized contribution weight (sums to 1 over `S_j`) |
| global prototype | `Cbar_j(beta)`; used for alignment and nearest-prototype inference |

**F. Exact difference between original and modified PROTEAN.**

- With `beta = 0`, `N_ij^0 = 1`, so the proposed aggregator becomes the **equal mean over `S_j`**.
- With `beta = 1`, the contribution is proportional to class support (equivalent to the mean of
  all included local class embeddings when the assumptions hold).
- `beta = 0` is equal weighting over `S_j`; it is equivalent to the original formulation **only
  if** the participant and missing-class rules are also equivalent. That equivalence is not
  verified against official PROTEAN code.
- Model aggregation, local prototype formation, architecture, loss, and inference are **not**
  modified.

**Novelty boundary.** Sample-count prototype weighting is not new by itself; related work such as
**FedProto** (Tan et al., 2022) already involves class sample counts in prototype-based federated
learning. The contribution here is the **controlled beta parameterization inside PROTEAN's
server-side prototype aggregation**, its condition-wise validation-based selection, the paired
multi-seed multi-condition evaluation, and the resulting behavioral findings — not the invention of
support weighting.

**Analysis chain (not a causal proof by itself).**

```
support imbalance -> contribution weight -> global-prototype displacement
                  -> rare-class detection -> per-client effect
```

## 4. Research Questions and Objectives

| ID | Research question |
|---|---|
| RQ1 | How are class-support imbalance, differences between local prototypes, and rare-class detection performance related in PROTEAN? |
| RQ2 | How does changing `beta` affect the global prototype and rare-class detection performance under different non-IID conditions? |
| RQ3 | Does the validation-selected `beta` change across non-IID levels, and what is its effect on overall and per-client performance? |

Objectives: (1) relate support imbalance to prototype behavior and rare-class detection (RQ1);
(2) characterize the effect of `beta` on the global prototype and rare-class detection across
conditions (RQ2); (3) select `beta` by validation at different imbalance levels and quantify
overall and per-client consequences (RQ3).

## 5. Experimental Design

| Item | Value |
|---|---|
| Base method | PROTEAN federated intrusion detection |
| Modified component | server-side prototype aggregation only |
| Mechanism | `w_ij = N_ij^beta`, one `beta` for all classes/clients |
| Datasets | X-IIoTID (`class2`, benign `Normal`, K=9); 5G-NIDD (`Attack Type`, benign `Benign`, K=9) |
| Clients | 10 |
| Non-IID conditions | symmetric Dirichlet `alpha` = 0.25, 0.50, 0.75 |
| Seeds | 0, 1, 2 (project-defined) |
| Beta candidates | 0, 0.25, 0.50, 0.75, 1.00 |
| Baseline | `original_protean` (equal-weight prototype aggregation) |
| Rounds / local epochs / batch | 10 / 3 / 256 |
| Optimizer / lr / mu / lambda | Adam / 1e-3 / 0.1 / 1.0 |
| Splits | stratified train / validation / test = 70 / 10 / 20 |
| Selection metric | pooled **validation** per-client-mean rare-class macro-F1 across seeds |
| Test usage | read only after the selected `beta` is fixed |

Expected experimental matrix: `2 datasets × 3 alpha × 3 seeds × (1 baseline + 5 betas) = 108 runs`.
Within each `(dataset, alpha, seed)` all six configurations share one manifest, split, class mapping,
and round-0 initialization; only the aggregation mode changes.

## 6. Completed Research Phases

Documented phases, in order. Session numbers are used only where the repository documents them.

| Phase / session | Objective | Key outputs | Validation |
|---|---|---|---|
| Planning and architecture | freeze scope, protocol, and project structure | `AGENTS.md`, `planningawal/PROTEAN_Revisi_Lurus.md`, `docs/research/IMPLEMENTATION_PLAN.md` | checkpoint commit |
| Baseline reproduction | implement reproducible PROTEAN baseline pipeline | `docs/baseline/BASELINE_IMPLEMENTATION.md`, `src/protean_research/` | `docs/baseline/BASELINE_VALIDATION.md` |
| Baseline validation + regression tests | verify baseline behavior and invariants | `tests/`, `docs/baseline/PROTEAN_BASELINE_INVESTIGATION.md` | unit/integration tests |
| Support-weighted aggregation | implement beta-controlled aggregator | `docs/research/BETA_AGGREGATION_IMPLEMENTATION.md`, `src/protean_research/aggregation/` | aggregation invariance tests |
| Session 5 — one paired condition | first full paired beta run (single seed) | `docs/research/PAIRED_BETA_VALIDATION.md` | `val_selection.json` |
| Session 6 — multi-seed (X-IIoTID a0.50) | pooled validation across seeds `{0,1,2}` | `docs/research/MULTISEED_BETA_VALIDATION.md`, `SESSION6_AUDIT.md` | audit PASS with minor corrections |
| Session 7 — X-IIoTID a0.25 | third X-IIoTID alpha condition | `docs/research/ALPHA025_BETA_VALIDATION.md`, `SESSION7_AUDIT.md` | audit PASS with minor corrections |
| Session 8 — X-IIoTID a0.75 | complete X-IIoTID three-alpha study | `docs/research/ALPHA075_BETA_VALIDATION.md`, `SESSION8_AUDIT.md` | audit PASS |
| Session 9 — 5G-NIDD a0.50 | first 5G-NIDD condition | `docs/research/5GNIDD_ALPHA050_BETA_VALIDATION.md`, `SESSION9_AUDIT.md` | audit PASS |
| Session 10 — 5G-NIDD a0.75 | second 5G-NIDD condition | `docs/research/5GNIDD_ALPHA075_BETA_VALIDATION.md`, `SESSION10_AUDIT.md` | audit PASS |
| Session 11 — 5G-NIDD a0.25 | final condition; completes the grid | `docs/research/5GNIDD_ALPHA025_BETA_VALIDATION.md`, `SESSION11_AUDIT.md` | audit PASS |
| Session 12 — synthesis | consolidate all six conditions | `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md`, `CROSS_DATASET_BETA_ANALYSIS.md`, `docs/thesis/*`, `artifacts/analysis/final__*`, `artifacts/figures/final/*` | independent recomputation, zero mismatches |
| Proposal phase | write the thesis proposal and audit aids | `docs/proposal/*` | committed `e27c5a8` |
| Handoff / README | keep repository state current | `docs/handoff/SESSION_HANDOFF.md`, `README.md` | this document |

## 7. Experiment Coverage and Reproducibility

**Verified run inventory.** Counted from `results/` metadata, not from filenames alone:

- non-smoke run directories: **108**
- distinct `(dataset, alpha, seed)` cells: **18** (2 × 3 × 3)
- configurations per cell: **1 `original_protean` + 5 `support_weighted`** (beta 0/0.25/0.50/0.75/1.00)
- paired comparison: within every cell, all six runs share **one manifest content hash** (verified);
  `original_protean` and `support_weighted beta=0` additionally share a byte-identical round-0 snapshot
- failed runs: **none**; one unrelated `smoke` directory is excluded

**Known artifact gap.** One run, `xiiotid__alpha0.5__seed0__original_protean`, lacks its entire
validation-side artifact set (`val_global_metrics.json`, `val_confusion_matrix.json`,
`val_per_class.csv`, `val_per_client.csv`, `val_selection.json`). Its test artifacts and history are
present, and it is not a selection run, so all 108 test evaluations and all six selections are
unaffected. Every other run has the complete test and validation artifact set. This is recorded as
an evidence gap rather than silently omitted.

**Reproducibility controls.**

- **Frozen protocol:** `docs/research/EXPERIMENT_PROTOCOL.md` (machine-readable
  `configs/experiments/experiment_protocol.yaml`). Values are either FROZEN from the paper or
  explicit PROJECT DECISIONs.
- **Frozen manifests:** 18 deterministic Dirichlet partition manifests under `artifacts/partitions/`;
  generated once and reused. Regenerating a fresh split for a run is a protocol violation.
- **Raw data:** external and read-only (`~/datasets/…`); never modified, copied, or committed.
  SHA-256 hashes are recorded in the audit documents.
- **Validation/test separation:** `beta` is selected from validation only; every run records
  `test_used_for_selection = false`.
- **Seeds** control split, Dirichlet partition, model initialization, and batch order.

## 8. Evaluation Metrics

All metrics below are implemented in `src/protean_research/evaluation/metrics.py` and persisted per
run. No scikit-learn `classification_report` is used; metrics are pure NumPy.

| Metric | Definition | Level | Split | Where stored |
|---|---|---|---|---|
| Accuracy | `trace(CM) / sum(CM)` | global | val, test | `{split}_global_metrics.json` |
| Macro-F1 | unweighted mean of per-class F1 over classes with support > 0 | global | val, test | `{split}_global_metrics.json` |
| Weighted-F1 | support-weighted mean of per-class F1 | global | val, test | `{split}_global_metrics.json` |
| Per-class precision/recall/F1/support | one-vs-rest from the confusion matrix | per class (global) | val, test | `{split}_per_class.csv` |
| Rare-class macro-F1 | mean per-class F1 over the **evaluated rare classes** (union of client rare classes with support) | global | val, test | `{split}_global_metrics.json` |
| Rare-class recall | mean recall over the evaluated rare classes | global | val, test | `{split}_global_metrics.json` |
| Rare support | number of eval samples in evaluated rare classes | global | val, test | `{split}_global_metrics.json` |
| Per-client accuracy/macro-F1 | standard metrics on each client's own eval partition | per client | val, test | `per_client.csv`, `val_per_client.csv` |
| Per-client rare-class macro-F1 / recall | rare metrics using that client's own rare class ids | per client | val, test | `per_client.csv`, `val_per_client.csv` |
| Selection score | mean over eligible clients of per-client validation rare-class macro-F1, pooled across seeds | per condition | validation | `val_selection.json`, aggregation artifacts |
| Prototype displacement | `‖Cbar_j(beta) − Cbar_j(0)‖` and relative displacement, fixed round-0 snapshot | per class | n/a | `*__prototype_effect*.csv` |
| Normalized weight | `N_ij^beta / Σ_k N_kj^beta` | per client-class | n/a | `*__prototype_weights.csv` |
| Support CV / missing cells / client-size ratio | realized non-IID statistics from `N_ij` | per class / partition | training | `characterization.csv`, `*partition_heterogeneity*` |

**Averaging-level distinctions that must not be interchanged.**

- **Global rare-class macro-F1** pools all rare-class evaluation samples; **per-client mean
  rare-class macro-F1** averages each client's own score. The **selection metric is the per-client
  mean, pooled across seeds**.
- **Validation** metrics choose `beta`; **test** metrics are reported only after selection.
- **Per-seed** values and **pooled** values are different quantities; the pooled score averages
  per-seed per-client-mean scores.

**Not computed by this repository:** micro-F1, ROC/AUC, statistical significance tests, and
persisted per-class unseen-class recall (unseen-class evaluation is defined in code but only the
unseen flags and union counts are saved).

## 9. Quantitative Experimental Results

All test values are means over seeds `{0,1,2}` for the selected `beta`, and the corresponding
`original_protean` test means. Values are from `artifacts/analysis/final__six_condition_master.csv`
and `final__selected_vs_original.csv`, and were independently re-checked against the per-run
`test_global_metrics.json`.

### 9.1 Pooled validation selection (validation-only)

Pooled validation per-client-mean rare-class macro-F1 (selected in bold):

| beta | X-IIoTID a0.25 | X-IIoTID a0.50 | X-IIoTID a0.75 | 5G-NIDD a0.25 | 5G-NIDD a0.50 | 5G-NIDD a0.75 |
|---|---|---|---|---|---|---|
| 0.00 | **0.5917** | 0.5444 | 0.4717 | **0.7615** | 0.6872 | 0.7781 |
| 0.25 | 0.5887 | 0.5554 | 0.4744 | 0.7493 | 0.6451 | 0.7905 |
| 0.50 | 0.5859 | 0.5673 | **0.4767** | 0.7257 | 0.6747 | **0.8091** |
| 0.75 | 0.5795 | **0.5794** | 0.4725 | 0.7222 | 0.6884 | 0.7915 |
| 1.00 | 0.5321 | 0.5735 | 0.4686 | 0.7376 | **0.6924** | 0.8043 |

### 9.2 Selected beta versus original PROTEAN (test)

Test rare-class macro-F1 (mean over 3 seeds), selected `beta` vs `original_protean`:

| Dataset | alpha | selected beta | test rare-F1 (selected) | test rare-F1 (original) | delta | test macro-F1 (selected) | test accuracy (selected) |
|---|---|---|---|---|---|---|---|
| X-IIoTID | 0.25 | 0.00 | 0.6321 | 0.6328 | −0.0007 | 0.6813 | 0.8740 |
| X-IIoTID | 0.50 | 0.75 | 0.6655 | 0.6425 | **+0.0230** | 0.7312 | 0.9064 |
| X-IIoTID | 0.75 | 0.50 | 0.6052 | 0.6039 | +0.0014 | 0.7576 | 0.9281 |
| 5G-NIDD | 0.25 | 0.00 | 0.8412 | 0.8423 | −0.0010 | 0.8166 | 0.7302 |
| 5G-NIDD | 0.50 | 1.00 | 0.8720 | 0.8805 | −0.0086 | 0.8386 | 0.7433 |
| 5G-NIDD | 0.75 | 0.50 | 0.8967 | 0.8799 | +0.0168 | 0.8531 | 0.7523 |

Reading of the table:

- The six conditions split into **three mutually exclusive categories that total six**:
  **three positive deltas** (X-IIoTID a0.50, X-IIoTID a0.75, 5G-NIDD a0.75), **one negative delta**
  (5G-NIDD a0.50), and **two effectively unchanged** (both `alpha = 0.25` conditions, where the
  selected `beta` is 0, so the selected run is the equal-weight run and the difference is
  floating-point divergence only).
- The **largest positive delta (+0.0230, X-IIoTID a0.50) is still smaller than that condition's
  between-seed sample std (0.0267)**. No statistical superiority is claimed.
- **No single beta dominates** across conditions, and the selection depends on both dataset and
  condition.

Cross-dataset metric levels are not comparable: 5G-NIDD rare classes are low-volume flood/scan
types, X-IIoTID rare classes are low-volume behavioural categories, and their rare support differs.

### 9.3 Selected betas

| Dataset | alpha=0.25 | alpha=0.50 | alpha=0.75 |
|---|---|---|---|
| X-IIoTID | 0.00 | 0.75 | 0.50 |
| 5G-NIDD | 0.00 | 1.00 | 0.50 |

## 10. Beta Selection by Dataset and Alpha

Selection is by pooled validation per-client-mean rare-class macro-F1 (validation only); the test
set is never used. `artifacts/analysis/final__selection_strength.csv` quantifies separation:

| Dataset | alpha | selected beta | best | 2nd best | abs gap | top-2 gap / selected std | per-seed diagnostic betas |
|---|---|---|---|---|---|---|---|
| X-IIoTID | 0.25 | 0.00 | 0.5917 | 0.5887 (0.25) | 0.0030 | 0.034 | 0.75 / 0.50 / 0.00 |
| X-IIoTID | 0.50 | 0.75 | 0.5794 | 0.5735 (1.00) | 0.0059 | 0.078 | 0.50 / 1.00 / 0.75 |
| X-IIoTID | 0.75 | 0.50 | 0.4767 | 0.4744 (0.25) | 0.0024 | 0.051 | 0.50 / 0.50 / 0.00 |
| 5G-NIDD | 0.25 | 0.00 | 0.7615 | 0.7493 (0.25) | 0.0122 | 0.556 | 0.25 / 0.00 / 0.00 |
| 5G-NIDD | 0.50 | 1.00 | 0.6924 | 0.6884 (0.75) | 0.0040 | 0.029 | 1.00 / 0.00 / 1.00 |
| 5G-NIDD | 0.75 | 0.50 | 0.8091 | 0.8043 (1.00) | 0.0048 | 0.081 | 0.50 / 1.00 / 1.00 |

In **all six conditions the top-two gap is smaller than the selected beta's across-seed sample std**
(ratios 0.03–0.56), so each selection is a *small numerical separation relative to seed variability*.
No tie-break was ever exercised. Per-seed diagnostic betas are shown only for stability; they are not
the selection rule.

## 11. Prototype-Level Analysis

Direct aggregation effect, measured on fixed round-0 snapshots (`C_ij`, `N_ij` held constant, no
retraining). `artifacts/analysis/final__prototype_displacement_by_beta.csv`.

Mean `‖Cbar_j(beta) − Cbar_j(0)‖` across classes:

| Dataset | alpha | beta=0.25 | beta=0.50 | beta=0.75 | beta=1.00 |
|---|---|---|---|---|---|
| X-IIoTID | 0.25 | 1.476 | 2.403 | 3.003 | **3.422** |
| X-IIoTID | 0.50 | 0.826 | 1.374 | 1.780 | **2.099** |
| X-IIoTID | 0.75 | 0.477 | 0.829 | 1.097 | **1.305** |
| 5G-NIDD | 0.25 | 1.295 | 2.055 | 2.559 | **2.919** |
| 5G-NIDD | 0.50 | 0.953 | 1.567 | 2.003 | **2.340** |
| 5G-NIDD | 0.75 | 0.497 | 0.875 | 1.163 | **1.385** |

- Displacement is **monotone increasing in `beta`** for a fixed snapshot in all six conditions
  (mechanically expected: larger `beta` shifts weight toward high-`N_ij` clients).
- Displacement at `beta = 1` **decreases as alpha rises**, tracking realized support imbalance.
- At matched alpha, X-IIoTID has larger displacement at a0.25 (3.422 vs 2.919) while 5G-NIDD is
  slightly larger at a0.50 (2.340 vs 2.099) and a0.75 (1.385 vs 1.305); per-class ordering is
  seed- and dataset-specific, and rare-class identities differ.
- **Displacement is a representation-aggregation metric, not detection quality.** No
  displacement-to-accuracy inference is made.

Normalized contribution weights are stored per client-class in
`artifacts/analysis/*__prototype_weights.csv`; per-class displacement is in
`final__prototype_displacement_by_class.csv`.

## 12. Rare-Class Detection Results

Rare classes are defined from each client's **training** support (the two attack classes with the
smallest positive `N_ij`; one if only one; the client is excluded if none). Unseen classes
(`N_ij = 0`) are treated separately and never merged with rare classes.

Key observations from the test results (Section 9.2) and the synthesis:

- Selected rare-class macro-F1 deltas are **mixed in sign and small**; the largest positive
  difference is within seed noise.
- Selected rare-class recall is **slightly lower than original in most conditions** while macro-F1
  may be higher: weighting can trade a little rare-class recall for precision.
- At both `alpha = 0.25` conditions the selected `beta` is 0, so those conditions cannot demonstrate
  any support-weighting benefit.
- Full rare-class curves and per-condition reports: `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md`,
  the per-condition `*_BETA_VALIDATION.md` reports, and `final__selected_vs_original.csv`.

## 13. Per-Client Analysis

Per-client effects are computed on each client's own evaluation partition relative to `beta = 0`
(`artifacts/analysis/final__client_effect_summary.csv`; detail in the per-condition
`*__multiseed__client_validation_*.csv`).

Improve / degrade / unchanged clients at the **selected** beta (validation, vs `beta=0`), by seed:

| Condition (selected beta) | seed 0 | seed 1 | seed 2 |
|---|---|---|---|
| X-IIoTID a0.25 (0.00) | 0 / 0 / 10 | 0 / 0 / 10 | 0 / 0 / 10 |
| X-IIoTID a0.50 (0.75) | 7 / 3 / 0 | 8 / 1 / 1 | 8 / 1 / 1 |
| X-IIoTID a0.75 (0.50) | 6 / 4 / 0 | 9 / 0 / 1 | 6 / 4 / 0 |
| 5G-NIDD a0.25 (0.00) | 0 / 0 / 10 | 0 / 0 / 10 | 0 / 0 / 10 |
| 5G-NIDD a0.50 (1.00) | 6 / 3 / 1 | 3 / 5 / 2 | 7 / 2 / 1 |
| 5G-NIDD a0.75 (0.50) | 4 / 6 / 0 | 6 / 3 / 1 | 4 / 2 / 4 |

- Weighting **redistributes benefit across clients**: within a condition some clients improve while
  others degrade.
- At `beta = 1`, mean per-client delta is most negative at both `alpha = 0.25` conditions
  (e.g. X-IIoTID a0.25 ≈ −0.060, 5G-NIDD a0.25 ≈ −0.024), consistent with those conditions
  selecting `beta = 0`.
- `N_ij` versus per-client delta is **non-monotonic**: the worst-degrading client is not always the
  lowest-support client.
- Client identities are per-seed realizations and are **not pooled** across seeds.

## 14. Cross-Alpha and Cross-Dataset Findings

- **Matched-alpha selections agree at alpha=0.25 (both 0.00) and alpha=0.75 (both 0.50)** and differ
  at alpha=0.50 (X-IIoTID 0.75 vs 5G-NIDD 1.00). Across the six conditions the selected beta takes
  values `{0.00, 0.50, 1.00}` — **not universal, not monotone in alpha, not a function of alpha
  alone**.
- **Realized heterogeneity** (support CV, missing client-class cells, client-size ratio) decreases as
  alpha rises within each dataset, but not strictly per seed. Dirichlet alpha is a design knob, not
  the realized heterogeneity itself.
- Displacement at `beta = 1` falls with alpha on both datasets.
- **Rare-set composition differs by dataset**, so absolute metric levels are **not comparable across
  datasets**; only within-condition differences are meaningful.
- **No cross-dataset causality, superiority, or dataset-independent beta law is claimed.**

Reports: `docs/research/CROSS_DATASET_BETA_ANALYSIS.md`, `XIIOTID_CROSS_ALPHA_ANALYSIS.md`,
`5GNIDD_CROSS_ALPHA_ANALYSIS.md`.

## 15. Confusion Matrix and Classification Breakdown

**Confusion matrices do exist in the saved artifacts.** They are written per run by the evaluation
code (`confusion_matrix` in `src/protean_research/evaluation/metrics.py`) and saved as:

- `results/<run>/test_confusion_matrix.json` — **108 / 108 runs**
- `results/<run>/val_confusion_matrix.json` — **107 / 108 runs** (missing only for
  `xiiotid__alpha0.5__seed0__original_protean`, which also lacks the other validation artifacts)

**Properties of the stored matrices.**

| Property | Value |
|---|---|
| Shape | 9 × 9 (K = 9 classes) |
| Values | **raw integer counts** (not normalized) |
| Scope | **global** over the evaluation split, not per-client |
| Restriction | full multiclass; **not** restricted to rare classes |
| Split | separate `test_` and `val_` matrices |
| Class order | X-IIoTID: `C&C, Exfiltration, Exploitation, Lateral_movement, Normal, RDOS, Reconnaissance, Tampering, Weaponization`; 5G-NIDD: `Benign, HTTPFlood, ICMPFlood, SYNFlood, SYNScan, SlowrateDoS, TCPConnectScan, UDPFlood, UDPScan` (also in each run's `metadata.json`) |
| Aggregation | each client's eval partition concatenated; no seed/client mixing |

Representative interpretation — 5G-NIDD, `alpha=0.75`, `seed=0`, **test**, original vs selected
(`beta=0.50`) — using actual counts:

| Class | eval support | original recall | selected-beta recall |
|---|---|---|---|
| Benign | 95,547 | 0.3795 | 0.3825 |
| HTTPFlood | 28,162 | 0.9729 | 0.9923 |
| ICMPFlood | 231 | 1.0000 | 1.0000 |
| SYNFlood | 1,944 | 0.8570 | 0.8570 |
| SYNScan | 4,009 | 0.9968 | 0.9968 |
| SlowrateDoS | 14,625 | 0.9217 | 0.9054 |
| TCPConnectScan | 4,010 | 0.9948 | 0.9948 |
| UDPFlood | 91,468 | 1.0000 | 1.0000 |
| UDPScan | 3,181 | 0.9975 | 0.9981 |

Largest off-diagonal cells (true -> predicted):

| true | predicted | original count | selected-beta count |
|---|---|---|---|
| Benign | UDPFlood | 56,899 | 57,054 |
| Benign | TCPConnectScan | 2,015 | 1,495 |
| SlowrateDoS | HTTPFlood | 1,079 | 1,308 |
| HTTPFlood | SlowrateDoS | 759 | 212 |
| SYNFlood | TCPConnectScan | 278 | 278 |

What these counts support:

- The dominant error mode is **Benign misclassified as UDPFlood** (about 57k samples), which
  dominates the off-diagonal mass. It is essentially unchanged by `beta`, so `beta` does not fix
  this confusion. This is visible in low Benign recall (≈0.38) with very high UDPFlood recall
  (1.0000), i.e. UDPFlood is an attractor for Benign.
- The change from original to selected `beta` shifts a few specific off-diagonal cells
  (`Benign → TCPConnectScan` decreases; `SlowrateDoS → HTTPFlood` increases), consistent with the
  small, mixed-sign aggregate deltas, **not** with a systematic improvement.
- Diagonal versus off-diagonal entries must be read together: high diagonal on the rare/scan classes
  coexists with the large Benign confusion, which is why accuracy is moderate while macro-F1 is
  higher.

Reconstruction note: because only aggregate per-class metrics (precision/recall/F1/support) are
persisted, a confusion matrix cannot be uniquely reconstructed from them; the raw `y_true`/`y_pred`
arrays are not saved. Reconstruction is unnecessary here because the matrices themselves are already
persisted as JSON.

Representative confusion-matrix files:

- `results/5gnidd__alpha0.75__seed0__original_protean/test_confusion_matrix.json`
- `results/5gnidd__alpha0.75__seed0__support_weighted__beta0.5/test_confusion_matrix.json`
- `results/xiiotid__alpha0.25__seed0__original_protean/test_confusion_matrix.json`

## 16. Final Figures and Visualizations

All figures are under `artifacts/figures/final/` and load from the `final__*` analysis tables. They
were generated with matplotlib; no training was involved.

**Fig A — pooled validation rare-class macro-F1 vs beta** (per dataset). Axes: `beta` (x) vs pooled
validation per-client-mean rare-class macro-F1 (y); one line per alpha. Evidence: the validation
curve is non-monotone and the selected `beta` differs by condition.

![Pooled validation beta curve, 5G-NIDD](artifacts/figures/final/figA_5gnidd_pooled_validation_beta_curve.png)

![Pooled validation beta curve, X-IIoTID](artifacts/figures/final/figA_xiiotid_pooled_validation_beta_curve.png)

**Fig B — selected beta by condition.** Bar chart of the six validation-selected betas; shows that
selections take `{0.00, 0.50, 1.00}` and are not universal.

![Selected beta by condition](artifacts/figures/final/figB_selected_beta_by_condition.png)

**Fig C — realized support CV.** Condition-level mean class-support CV; shows realized imbalance
decreases as alpha rises.

![Realized support CV](artifacts/figures/final/figC_realized_support_cv.png)

**Fig D — prototype displacement at beta=0 vs beta=1.** Fixed-snapshot mean displacement per class;
shows displacement grows with `beta` and larger where support imbalance is larger.

![Prototype displacement beta0 vs beta1](artifacts/figures/final/figD_prototype_displacement_beta0_vs_beta1.png)

**Fig E — support CV vs displacement.** Scatter of realized support CV (x) vs beta=1 displacement
(y) across conditions; a descriptive co-variation, not a causal relation.

![Support CV vs displacement](artifacts/figures/final/figE_support_cv_vs_displacement.png)

**Fig F — selected vs original test rare-class macro-F1.** Paired comparison per condition; shows
small, mixed-sign differences.

![Selected vs original test rare macro-F1](artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.png)

**Fig G — per-client effect at the selected beta.** Per-client validation delta relative to
`beta = 0`; shows within-condition heterogeneity across clients.

![Client effect at selected beta](artifacts/figures/final/figG_client_effect_at_selected_beta.png)

## 17. Main Scientific Findings

1. **Changing `beta` changes the global prototype.** Contribution weights become proportional to
   support, and for a fixed snapshot mean prototype displacement grows monotonically with `beta` in
   every condition.
2. **The rare-class response is condition-dependent and non-monotone in `beta`.** The direct
   aggregation effect (geometry) and the end-to-end training effect (detection) are distinct.
3. **No universal beta was found.** The validation-selected `beta` differs across datasets and
   conditions and is not monotone in `alpha`. Every selection is weak relative to seed variability.
4. **Selected-vs-original differences are small and mixed in sign.** No statistical superiority is
   established; the largest positive difference is within seed noise.
5. **Weighting redistributes benefit across clients.** Within a condition some clients improve while
   others degrade; the effect is not simply "more support is better".
6. **Where weighting does not help:** both `alpha = 0.25` conditions select `beta = 0` (equal
   weighting), and at `beta = 1` the per-client effect is most damaging there.
7. **Realized support imbalance and prototype displacement co-vary** descriptively; displacement is
   not a proxy for detection quality.
8. **The dominant classification error (Benign → UDPFlood) is not addressed by `beta`.**

## 18. Limitations and Novelty Boundary

**Limitations.**

- The `S_j` participant/denominator rule is a **documented reproduction assumption** (no verified
  official PROTEAN code). It is material in up to 19/27 class-condition cells; all results are
  conditional on it.
- `beta = 0` is only arithmetically the equal mean over `S_j`; it is **not bitwise identical** to
  `original_protean` (floating-point divergence; equivalence class B in every condition).
- Two datasets, three alpha values, three project-defined seeds, 10 clients, one architecture.
- The `beta` grid is discrete; intermediate values were not tested.
- Three seeds are underpowered for inferential testing; no significance test is claimed.
- Prototype displacement is a representation metric, not detection quality.
- One validation artifact set is missing (Section 7); rare-class and unseen-class results cannot be
  merged, and cross-dataset absolute metric levels are not comparable.

**Novelty boundary.** Support-weighted prototype aggregation is not new (e.g. FedProto). The
contribution is the controlled `beta` parameterization inside PROTEAN, the validation-only
condition-wise selection, the paired multi-seed design, and the behavioral findings. No claim is
made that the method universally improves PROTEAN.

## 19. Repository Structure

```
src/protean_research/   data loading/preprocessing/partitioning, models, aggregation, federated
                        round logic, evaluation metrics, analysis, utils
scripts/                training runner, partition generation, and read-only analysis scripts
configs/                dataset, experiment protocol, and model configuration (YAML)
docs/proposal/          thesis proposal and internal audit aids
docs/research/          frozen protocol, per-condition reports, cross-alpha/cross-dataset synthesis,
                        session audits
docs/baseline/          PROTEAN baseline investigation, implementation, and validation
docs/thesis/            Chapter 4 blueprint and claims register
docs/handoff/           session handoff (start here for a new session)
planningawal/           immutable research specification (do not modify)
artifacts/partitions/   deterministic frozen Dirichlet partition manifests (18)
artifacts/analysis/     committed machine-readable analysis outputs
artifacts/figures/final/ thesis-ready figures
results/                local run outputs (git-ignored; 108 non-smoke run dirs + 1 smoke)
```

## 20. Reproduction and Tests

```bash
# unit + integration tests (expected: 75 passed)
.venv/bin/python -m pytest

# read-only pooled validation analysis (no retraining)
.venv/bin/python scripts/analyze_multiseed_beta.py --dataset 5gnidd --alpha 0.25 --seeds 0 1 2

# read-only realized-partition characterization
.venv/bin/python scripts/analyze_partition_heterogeneity.py --dataset 5gnidd --alphas 0.25 0.5 0.75

# read-only cross-alpha synthesis
.venv/bin/python scripts/analyze_cross_alpha.py --dataset 5gnidd --alphas 0.25 0.5 0.75

# beta=0 vs original equivalence audit (read-only)
.venv/bin/python scripts/verify_beta0_equivalence.py --dataset 5gnidd --alpha 0.25 --seeds 0 1 2
```

A full training run (not needed; the grid is complete) is:

```bash
.venv/bin/python scripts/run_baseline.py --dataset 5gnidd --alpha 0.25 --seed 1 \
    --clients 10 --rounds 10 --local-epochs 3 \
    --aggregation-mode support_weighted --beta 1.0
```

Runtime context: the 108 runs consumed ≈ 94.5 wall-clock hours on CPU (8 cores); per-run wall clock
was ≈ 62–65 minutes for 5G-NIDD and ≈ 40 minutes for X-IIoTID (`*__multiseed__runtime.csv`). Do not
rerun the grid.

**Repository safety.**

- `planningawal/` is immutable; raw datasets are external and read-only.
- Do not modify or regenerate completed results, manifests, or figures.
- Do not add an `equal_available` mode; it is arithmetically `support_weighted` at `beta = 0` under
  the shared `S_j`.
- `beta` selection uses validation only; the test set is never used for selection.

## 21. Complete Artifact Index

Per-run outputs (`results/<dataset>__alpha<alpha>__seed<seed>__<config>/`):

| File | Content |
|---|---|
| `metadata.json` | run config, dataset/alpha/seed, mode, class names, hashes |
| `history.json` | per-round global accuracy/macro-F1/rare-class metrics |
| `characterization.csv` | per-class training support statistics (`N_ij`) |
| `train_support_Nij.csv` | `N_ij` matrix (client × class) |
| `rare_unseen.csv` | per client-class rare/unseen flags from training support |
| `test_global_metrics.json`, `val_global_metrics.json` | global metrics per split |
| `test_per_class.csv`, `val_per_class.csv` | per-class precision/recall/F1/support |
| `per_client.csv`, `val_per_client.csv` | per-client metrics |
| `test_confusion_matrix.json`, `val_confusion_matrix.json` | 9×9 raw-count confusion matrices |
| `val_selection.json` | validation selection metric and `test_used_for_selection` flag |

For the one incomplete validation run, see Section 7.

Analysis outputs (`artifacts/analysis/`), by function:

| Category | Representative files |
|---|---|
| Final condition summary | `final__six_condition_master.csv/.json` |
| Beta-selection curves | `final__pooled_validation_curves.csv`, `final__selection_strength.csv` |
| Selected vs original | `final__selected_vs_original.csv` |
| Realized heterogeneity | `final__realized_heterogeneity.csv`, `*__partition_heterogeneity_*.csv` |
| Prototype displacement | `final__prototype_displacement_by_beta.csv`, `..._by_class.csv` |
| Per-client effects | `final__client_effect_summary.csv`, `*__multiseed__client_validation_*.csv` |
| Per-condition validation | `*__multiseed__validation_per_seed.csv`, `..._validation_pooled.csv` |
| Per-condition test | `*__multiseed__test_descriptive.csv` |
| Cross-alpha | `{xiiotid,5gnidd}__cross_alpha__*.csv/.json` |
| Cross-dataset | `cross_dataset__matched_alpha_summary.csv/.json`, `cross_dataset__beta_curve.csv` |
| beta=0 equivalence | `*__multiseed__beta0_equivalence.csv` |
| Runtime | `*__multiseed__runtime.csv` |
| Prototype snapshots & weights | `*__prototype_snapshot.json`, `*__prototype_weights.csv`, `*__prototype_effect*.csv` |

Partition manifests: `artifacts/partitions/<dataset>__alpha<alpha>__seed<seed>.json` (18 files).
Figures: `artifacts/figures/final/figA–figG*.png` (8 files).

Key documents:

| Document | Purpose |
|---|---|
| `docs/research/EXPERIMENT_PROTOCOL.md` | frozen reproduction protocol |
| `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md` | authoritative six-condition synthesis |
| `docs/research/CROSS_DATASET_BETA_ANALYSIS.md` | cross-dataset comparison |
| `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md` | X-IIoTID cross-alpha analysis |
| `docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md` | 5G-NIDD cross-alpha analysis |
| `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md` | Chapter 4 blueprint |
| `docs/thesis/CLAIMS_REGISTER.md` | claim / evidence / status register |
| `docs/proposal/THESIS_PROPOSAL_DRAFT.md` | final proposal Markdown source |
| `docs/proposal/PROPOSAL_SOURCE_MAP.md` | claim-to-evidence map |
| `docs/proposal/LITERATURE_MATRIX.md` | literature matrix |
| `docs/proposal/REFERENCES_VERIFIED.md` | bibliography verification record |
| `docs/handoff/SESSION_HANDOFF.md` | current state and next steps |
| `planningawal/PROTEAN_Revisi_Lurus.md` | immutable research specification |

## 22. References

- Chennoufi, S., Han, Y., Blanc, G., De Cristofaro, E., & Kiennert, C. (2025). PROTEAN: Federated
  Intrusion Detection in Non-IID Environments through Prototype-Based Knowledge Sharing. ESORICS
  2025, pp. 103–125. DOI: 10.1007/978-3-032-07884-1_6. arXiv:2507.05524.
- Tan, Y., Long, G., Liu, L., Zhou, T., Lu, Q., Jiang, J., & Zhang, C. (2022). FedProto: Federated
  Prototype Learning across Heterogeneous Clients. AAAI 36(8), 8432–8440.
  DOI: 10.1609/aaai.v36i8.20819. arXiv:2105.00243.

The complete, verified bibliography (including the FL foundations, non-IID, class-imbalance, and
evaluation-metric references used in the proposal) is maintained in
`docs/proposal/REFERENCES_VERIFIED.md` and the proposal bibliography. This README cites only the two
works directly relevant to the mechanism and novelty boundary.