research: add paired original vs proposed PROTEAN analysis

2026-10-09 13:40:20 +07:00
1 parent 87f9b14ada
commit ad8b2d7671
@@ -0,0 +1,359 @@
+# Original PROTEAN vs Proposed Beta-Tuned Aggregation: Deep Paired Comparison
+
+Status: **descriptive post-hoc analysis of completed experiments**. No training, no partition
+regeneration, no modification of any result artifact. Every number is derived from the audited
+machine-readable outputs in `results/` and `artifacts/analysis/`, using the reproducible scripts
+`scripts/deep_comparison.py` (derived tables) and `scripts/make_comparison_figures.py`
+(Figures I–K).
+
+Primary comparison:
+
+```
+original_protean            (frozen reproduced baseline)
+        vs
+support_weighted(beta*)     (validation-selected beta per dataset-alpha condition)
+```
+
+This document does **not** substitute `support_weighted(beta=0)` for the baseline anywhere;
+`beta=0` appears only as an explicitly labelled control.
+
+---
+
+## 1. Research question and comparison scope
+
+The question is not whether the proposed weighting is universally better. It is: **where, on the
+completed grid, does the validation-selected `support_weighted` run measurably differ from
+`original_protean`, in which direction, for which clients and classes, and how large are those
+differences relative to seed variability?**
+
+Six dataset-alpha conditions (2 datasets × 3 Dirichlet alpha), three seeds each (18 paired cells, 30
+client-seed observations per condition). All comparisons are on the **test** split, using the same
+partition, split, class mapping, and initialization within each cell.
+
+## 2. Original PROTEAN versus proposed mechanism
+
+Only the **server-side global prototype aggregation** changes:
+
+$$
+\bar{C}_j(\beta) = \frac{\sum_{i \in S_j} N_{ij}^{\beta} C_{ij}}{\sum_{i \in S_j} N_{ij}^{\beta}},
+\qquad
+w_{ij}(\beta) = \frac{N_{ij}^{\beta}}{\sum_{k \in S_j} N_{kj}^{\beta}}
+$$
+
+- `original_protean` uses equal contributions over `S_j` (the reference Eq. 2 behavior under the
+  documented participant rule).
+- `support_weighted(beta*)` reweights each contributing client by its class-specific training support
+  `N_ij`, at the beta selected by pooled validation.
+
+**Unchanged:** local prototype construction, model aggregation, architecture, loss, and the
+nearest-prototype inference rule. **Changed:** only the prototype aggregation weights.
+`N_ij` is class-specific sample support, not prototype quality.
+
+Selected betas (source: `artifacts/analysis/final__selection_strength.csv`), verified:
+
+| Dataset | alpha=0.25 | alpha=0.50 | alpha=0.75 |
+|---|---|---|---|
+| X-IIoTID | 0.00 | 0.75 | 0.50 |
+| 5G-NIDD | 0.00 | 1.00 | 0.50 |
+
+## 3. Pairing and evaluation methodology
+
+- **Pairing verified** for all 18 cells: same dataset, alpha, seed, and **identical
+  `manifest_content_hash`** between the `original_protean` and `support_weighted` runs
+  (`scripts/deep_comparison.py` raises on any mismatch).
+- `Delta = proposed − original` everywhere. Percentage points (pp) are never conflated with relative
+  percent.
+- Per-client test metrics come from each run's `per_client.csv` (rare-class macro-F1 and recall are
+  computed on each client's own evaluation partition using that client's own rare-class ids;
+  all 10 clients are eligible in every condition). Per-class metrics come from `test_per_class.csv`;
+  confusion matrices from `test_confusion_matrix.json`.
+- **Effectively-unchanged band = ±0.02** for paired per-client deltas, chosen to reflect the observed
+  `original_protean` vs `support_weighted(beta=0)` residual divergence (the two "selected beta = 0"
+  conditions). Raw deltas are stored in `deepcmp__per_client.csv` so any other threshold can be
+  applied.
+- Three seeds do not support inferential significance testing; all statements are descriptive.
+
+Derived tables: `deepcmp__global_per_seed.csv`, `deepcmp__global_summary.csv`,
+`deepcmp__per_client.csv`, `deepcmp__per_client_by_condition.csv`, `deepcmp__per_class.csv`,
+`deepcmp__confusion_delta.csv`, `deepcmp__client_support_join.csv`.
+
+## 4. Six-condition global metric comparison
+
+Test metrics, mean over seeds `{0,1,2}`; `Δ = proposed − original` (source:
+`deepcmp__global_summary.csv`).
+
+| Dataset | alpha | beta* | rare-F1 orig | rare-F1 prop | **Δ rare-F1** | Δ rare-recall | Δ macro-F1 | Δ weighted-F1 | Δ accuracy |
+|---|---|---|---|---|---|---|---|---|---|
+| X-IIoTID | 0.25 | 0.00 | 0.6328 | 0.6321 | **−0.0007** | −0.0007 | −0.0007 | −0.0008 | −0.0014 |
+| X-IIoTID | 0.50 | 0.75 | 0.6425 | 0.6655 | **+0.0230** | −0.0144 | +0.0222 | +0.0089 | +0.0069 |
+| X-IIoTID | 0.75 | 0.50 | 0.6039 | 0.6052 | **+0.0014** | −0.0077 | +0.0058 | +0.0061 | +0.0075 |
+| 5G-NIDD | 0.25 | 0.00 | 0.8423 | 0.8412 | **−0.0010** | +0.0007 | −0.0028 | −0.0010 | −0.0007 |
+| 5G-NIDD | 0.50 | 1.00 | 0.8805 | 0.8720 | **−0.0086** | −0.0058 | −0.0038 | −0.0005 | −0.0007 |
+| 5G-NIDD | 0.75 | 0.50 | 0.8799 | 0.8967 | **+0.0168** | −0.0015 | +0.0117 | +0.0014 | +0.0013 |
+
+Per-seed rare-class macro-F1 deltas (seed 0 / 1 / 2), showing the seed spread:
+
+| Dataset | alpha | beta* | Δ seed 0 | Δ seed 1 | Δ seed 2 |
+|---|---|---|---|---|---|
+| X-IIoTID | 0.25 | 0.00 | +0.0016 | −0.0020 | −0.0016 |
+| X-IIoTID | 0.50 | 0.75 | +0.0266 | +0.0163 | +0.0261 |
+| X-IIoTID | 0.75 | 0.50 | −0.0097 | +0.0101 | +0.0036 |
+| 5G-NIDD | 0.25 | 0.00 | −0.0050 | −0.0010 | +0.0029 |
+| 5G-NIDD | 0.50 | 1.00 | +0.0090 | −0.0811 | +0.0464 |
+| 5G-NIDD | 0.75 | 0.50 | −0.0011 | +0.0450 | +0.0067 |
+
+Interpretation:
+
+- **Two conditions are numerically null**: both `alpha=0.25` conditions selected `beta=0`, so the
+  selected run is the equal-weight `support_weighted` run; their small deltas are floating-point
+  divergence, not a weighting effect.
+- Among the **four genuine weighting conditions (beta>0)**, rare-class macro-F1 improved in three
+  (X-IIoTID a0.50 +0.023, X-IIoTID a0.75 +0.001, 5G-NIDD a0.75 +0.017) and degraded in one
+  (5G-NIDD a0.50 −0.009).
+- 5G-NIDD a0.50 is dominated by **seed variance** (−0.081 at seed 1 versus +0.046 at seed 2); its
+  pooled mean is not a reliable condition-level statement.
+- **Rare-class recall drops in five of six conditions** (and in **all four** beta>0 conditions)
+  even when rare-class macro-F1 rises: the F1 gains are largely precision-driven (see §7).
+- Overall macro-F1 moves **in the same direction** as rare-class F1 in every condition; where rare-F1
+  improves, overall macro-F1 also improves (X-IIoTID a0.50 +0.022, 5G-NIDD a0.75 +0.012). There is
+  no condition with a rare-class gain accompanied by an overall-metric collapse.
+
+## 5. Rare-class performance trade-offs
+
+The recurring pattern is a **precision–recall trade-off**, not a uniform improvement:
+
+- X-IIoTID a0.50: rare-F1 +0.0230 while rare-recall −0.0144 → rare-class precision rose.
+- X-IIoTID a0.75: rare-F1 +0.0014 while rare-recall −0.0077.
+- 5G-NIDD a0.75: rare-F1 +0.0168 while rare-recall −0.0015.
+- 5G-NIDD a0.50: rare-F1 −0.0086 and rare-recall −0.0058 (both down; not a trade-off but a
+  regression, and seed-dominated).
+
+So the weighting can raise the rare-class **balance** metric by suppressing false positives on rare
+classes while giving up a little rare-class coverage.
+
+## 6. Paired per-client results (test, vs original_protean)
+
+Figure I: `artifacts/figures/final/figI_per_client_original_vs_proposed.png`.
+Per-observation table: `deepcmp__per_client.csv` (30 observations per condition).
+
+| Condition | beta* | improved | degraded | unchanged | mean Δ | min Δ | max Δ |
+|---|---|---|---|---|---|---|---|
+| X-IIoTID a0.25 | 0.00 | 1 | 1 | 28 | −0.0015 | −0.026 | +0.025 |
+| X-IIoTID a0.50 | 0.75 | 12 | 1 | 17 | +0.0302 | −0.257 | +0.324 |
+| X-IIoTID a0.75 | 0.50 | 8 | 2 | 20 | +0.0117 | −0.070 | +0.068 |
+| 5G-NIDD a0.25 | 0.00 | 1 | 0 | 29 | +0.0021 | −0.020 | +0.071 |
+| 5G-NIDD a0.50 | 1.00 | 15 | 6 | 9 | +0.0390 | −0.176 | +0.404 |
+| 5G-NIDD a0.75 | 0.50 | 8 | 1 | 21 | +0.0207 | −0.036 | +0.168 |
+
+Counts use the ±0.02 effectively-unchanged band on per-client test rare-class macro-F1.
+
+- At the two `beta=0` conditions the counts collapse to essentially "unchanged" (28/30 and 29/30),
+  confirming that the beta=0 run is numerically the equal-weight control.
+- At beta>0 conditions the response is **heterogeneous**: X-IIoTID a0.50 improves a majority
+  (12/30) with only 1 clear degradation but contains the single largest loss (−0.257); 5G-NIDD a0.50
+  improves 15/30 but degrades 6/30.
+- Client deltas are not small in the tails: individual clients move by ±0.17–0.40 at the strongest
+  conditions. A condition-level mean therefore hides material client-level redistribution.
+- Client ids are per-seed realizations; id `k` is not the same entity across seeds and is never
+  pooled.
+
+## 7. Per-class precision/recall/F1 changes (beta>0 conditions)
+
+Figure J: `artifacts/figures/final/figJ_per_class_original_vs_proposed.png`.
+Source: `deepcmp__per_class.csv` (per seed) aggregated as the mean delta over the three seeds; class
+support is the mean test support over seeds. X-IIoTID and 5G-NIDD labels are never mixed.
+
+X-IIoTID, alpha=0.50 (beta=0.75):
+
+| Class | Δ F1 | Δ recall | Δ precision | support~ |
+|---|---|---|---|---|
+| Tampering | **+0.144** | −0.009 | +0.212 | 1,024 |
+| Reconnaissance | +0.054 | +0.086 | −0.003 | 25,474 |
+| Lateral_movement | +0.037 | +0.059 | +0.004 | 6,319 |
+| Weaponization | +0.019 | −0.029 | +0.065 | 12,884 |
+| Normal | −0.006 | −0.012 | −0.000 | 83,651 |
+| C&C | **−0.056** | −0.105 | −0.034 | 573 |
+
+X-IIoTID, alpha=0.75 (beta=0.50): largest gains Reconnaissance (+0.033) and Tampering (+0.010);
+largest loss Exfiltration (−0.013).
+
+5G-NIDD, alpha=0.50 (beta=1.00): largest gain TCPConnectScan (+0.034, precision-driven); largest
+losses SYNFlood (−0.043, precision −0.073) and SlowrateDoS (−0.013). 5G-NIDD, alpha=0.75 (beta=0.50):
+largest gain ICMPFlood (+0.097, precision +0.102 on a very small support of ~231); losses SYNFlood
+(−0.009).
+
+Points to note:
+
+- Gains are **not confined to rare classes**: at X-IIoTID a0.50 the largest gain is on the
+  low-support class **Tampering**, but a large gain also appears on the higher-support class
+  **Reconnaissance** (recall-driven).
+- The largest per-class **degradations** are on low-support attack classes with volatile F1:
+  **C&C** at X-IIoTID a0.50 and **SYNFlood** at 5G-NIDD a0.50. Both are precision losses at very
+  small support (~573 and ~1,944).
+- **Benign detection is essentially unchanged** (Δ F1 +0.0009 to +0.0019; recall within ±0.002),
+  so the weighting does not trade benign detection for rare-class gains.
+- Increased rare-class recall does **not** systematically accompany increased false positives:
+  X-IIoTID Reconnaissance recall rose +0.086 with precision −0.003, i.e. almost free recall.
+  Conversely, Tampering F1 rose via precision, not recall.
+
+## 8. Confusion-matrix difference analysis
+
+Figure K shows the paired difference for **5G-NIDD, alpha=0.75, seed=0** (the same pair as Figure H):
+`artifacts/figures/final/figK_confusion_delta_5gnidd_a0.75_seed0.png`. Rows = true class, columns =
+predicted class; both matrices have identical class order and 243,177 test samples. Source:
+`deepcmp__confusion_delta.csv`.
+
+Largest raw-count changes for this pair (proposed − original):
+
+| Change | true → predicted | original | proposed | Δ count | Δ row-rate |
+|---|---|---|---|---|---|
+| Reduced error | HTTPFlood → SlowrateDoS | 759 | 212 | **−547** | −1.94 pp |
+| Reduced error | Benign → TCPConnectScan | 2,015 | 1,495 | **−520** | −0.54 pp |
+| Increased error | SlowrateDoS → HTTPFlood | 1,079 | 1,308 | **+229** | +1.57 pp |
+| Increased error | Benign → UDPFlood | 56,899 | 57,054 | **+155** | +0.16 pp |
+| Diagonal | HTTPFlood → HTTPFlood | 27,399 | 27,946 | +547 | +1.94 pp |
+| Diagonal | Benign → Benign | 36,263 | 36,551 | +288 | +0.30 pp |
+| Diagonal | SlowrateDoS → SlowrateDoS | 13,480 | 13,241 | −239 | −1.63 pp |
+
+- The dominant residual error, **Benign → UDPFlood (~57k samples)**, is essentially unchanged; the
+  proposed weighting neither fixes nor meaningfully worsens it. The +155 cell must not be described
+  as an improvement.
+- The main benefit here is a **net reduction of the HTTPFlood ↔ SlowrateDoS confusion**: the
+  HTTPFlood → SlowrateDoS cell lost 547 samples while the reverse gained 229, and HTTPFlood diagonal
+  gained 547 — consistent with higher HTTPFlood precision.
+- The main cost here is a **small loss of SlowrateDoS diagonal** (−239) with a +229 shift into
+  SlowrateDoS → HTTPFlood.
+- Across all four beta>0 conditions (summed raw counts), the largest **reduced** misclassifications
+  are X-IIoTID `Reconnaissance → Exploitation` (−6,773), `Reconnaissance → Weaponization` (−3,398),
+  `Lateral_movement → Tampering` (−1,699) and 5G-NIDD `Benign → TCPConnectScan` (−1,484); the largest
+  **increased** ones are X-IIoTID `Normal → C&C` (+2,546) and `Weaponization → Exploitation` (+1,286).
+  Raw counts are support-sensitive, so these are descriptive; the row-normalized view (Figure K,
+  right panel) is the support-fair comparison.
+
+## 9. Support imbalance versus client outcomes
+
+Join: per-client test delta (beta>0 conditions only) against the client's training support from
+`rare_unseen.csv` (`deepcmp__client_support_join.csv`, n = 120 client-seed observations).
+
+| Statistic | Value |
+|---|---|
+| Pearson(Δ rare-F1, rare-class support fraction) | −0.092 |
+| Pearson(Δ rare-F1, total client samples) | −0.009 |
+| Low rare-support quartile (≤72 rare-class samples) mean Δ | **+0.041** |
+| High rare-support quartile (≥290) mean Δ | **+0.008** |
+| Mean Δ for clients with 0 missing classes (n=101) | +0.022 |
+| Mean Δ for clients with exactly 1 missing class (n=16) | +0.045 |
+
+- There is **no evidence that low-support clients are disproportionately harmed**; the linear
+  association is near zero and the low-support quartile improved, on average, at least as much as the
+  high-support quartile.
+- Clients with one missing class improved more on average, but n is small.
+- **Limitation:** this is a descriptive association across 120 noisy observations, not a causal test;
+  the beta>0 conditions confound `beta`, dataset, and alpha, and no significance test is valid with
+  three seeds.
+
+## 10. Prototype displacement and interpretation
+
+From `final__prototype_displacement_by_beta.csv` (fixed round-0 snapshots, identical local prototypes
+and `N_ij`, no retraining), mean `‖Cbar_j(1) − Cbar_j(0)‖`:
+
+| Dataset | alpha=0.25 | alpha=0.50 | alpha=0.75 |
+|---|---|---|---|
+| X-IIoTID | 3.422 | 2.099 | 1.305 |
+| 5G-NIDD | 2.919 | 2.340 | 1.385 |
+
+- Displacement is monotone in beta and largest where realized support imbalance is largest.
+- The observed per-class/per-client detection changes are **consistent with** the mechanism chain
+  support imbalance → weight redistribution → prototype displacement → detection change, but the
+  experiment **does not isolate** the displacement→detection step (training is end-to-end).
+  Fixed-snapshot displacement must not be read as detection quality.
+
+## 11. Conditions where the proposed method helps
+
+- **X-IIoTID α=0.50 (beta=0.75)**: the clearest case. Rare-class macro-F1 +0.0230, overall macro-F1
+  +0.0222, accuracy +0.0069; 12/30 client observations improved (1 degraded). Gains driven by
+  Reconnaissance recall and Tampering precision; rare-class recall traded down slightly.
+- **5G-NIDD α=0.75 (beta=0.50)**: rare-class macro-F1 +0.0168, macro-F1 +0.0117; 8/30 improved,
+  1 degraded.
+- **X-IIoTID α=0.75 (beta=0.50)**: marginal rare-F1 gain (+0.0014) with broader macro-F1/accuracy
+  gains (+0.0058/+0.0075).
+
+## 12. Conditions where the proposed method hurts
+
+- **5G-NIDD α=0.50 (beta=1.00)**: rare-class macro-F1 −0.0086 and macro-F1 −0.0038, seed-dominated
+  (−0.081 at seed 1). 6/30 client observations degraded. The clearest case against stronger weighting.
+- **Both α=0.25 conditions**: the selected beta is 0 (equal weighting), so there is no weighting
+  benefit; the small deltas are numerical. At `beta=1` these conditions show the largest per-client
+  degradations, which is why validation did not select a nonzero beta there.
+- **Specific classes**: X-IIoTID **C&C** (−0.056 F1) and 5G-NIDD **SYNFlood** (−0.043 F1) degrade at
+  the strongest `beta`, both low-support classes, and the 5G-NIDD α=0.75 seed-0 confusion shows a
+  small SlowrateDoS recall loss.
+- **No condition** shows a large, consistent, cross-seed rare-class improvement; the largest positive
+  condition mean is +0.023 and its strongest seed-rate change is within the seed spread.
+
+## 13. Effect sizes and practical relevance
+
+- Genuine rare-class macro-F1 effects are **≤ ~0.023** in magnitude, i.e. **≤ 2.3 percentage
+  points**, and the largest is roughly the size of the between-seed standard deviation at that
+  condition. They are small relative to seed noise.
+- The clearest overall-metric movement is X-IIoTID α=0.50 (macro-F1 +0.022, accuracy +0.007); the
+  other positive conditions move by ≤ ~0.012 macro-F1.
+- Client-level redistribution is larger than the condition-level effect: individual clients move by
+  up to −0.26 / +0.32 (X-IIoTID α=0.50) even though the mean is +0.03.
+- Practically, the weighting is **not a reliable blanket improvement**; it is a small, condition- and
+  client-dependent adjustment whose sign is not predictable from alpha alone.
+
+## 14. Threats to validity
+
+- **Only three seeds**; no inferential test is valid. Condition means are dominated by seed variance
+  at 5G-NIDD α=0.50.
+- **Two `beta=0` conditions are numerical, not weighting effects**; they are retained for
+  completeness and labelled.
+- **`S_j` is a reproduction assumption** (no official PROTEAN code). All comparisons are conditional
+  on both modes sharing it.
+- **Rare-class composition differs by dataset**; absolute cross-dataset levels are not comparable.
+- **Per-class metrics on tiny supports** (C&C ~573, ICMPFlood ~231, SYNFlood ~1,944) are volatile.
+- **Confusion-matrix raw counts** are support-sensitive; use row-normalized rates for support-fair
+  comparison.
+- Client ids are per-seed realizations and are not pooled across seeds.
+- `original_protean` is never substituted by `beta=0`.
+
+## 15. Evidence-backed scientific conclusions
+
+1. Over the six conditions, the validation-selected beta produces **small, mixed-sign** test
+   differences from `original_protean`; it is not a universal improvement.
+2. Among the four genuine `beta>0` conditions, rare-class macro-F1 improved in **three** and degraded
+   in **one**; the largest gain is ~2.3 pp and is within seed variability.
+3. When rare-class macro-F1 rises, **overall macro-F1 rises with it**, but **rare-class recall
+   usually falls** — the gains are precision-driven.
+4. Benefits and harms are **client-specific and class-specific**; within a condition some clients
+   gain and others lose, with individual changes larger than the condition mean.
+5. The dominant classification error (Benign → UDPFlood) is **unchanged** by the weighting. The
+   measurable benefit at the analysed pair is a reduced HTTPFlood ↔ SlowrateDoS confusion.
+6. Low-support clients are **not** systematically disadvantaged in the observed data, but the
+   evidence is descriptive only.
+7. Where weighting does not help (both α=0.25 conditions), validation correctly selects `beta=0`.
+
+**Final question — what measurable benefit does beta-controlled prototype aggregation provide over
+original PROTEAN, under which conditions, and at what cost?**
+The evidence supports a **modest, condition-dependent benefit**: at X-IIoTID α=0.50 a rare-class
+macro-F1 gain of about **+0.023** with a matching overall macro-F1 gain, and smaller gains at X-IIoTID
+α=0.75 and 5G-NIDD α=0.75. The cost is a small **rare-class recall reduction** (precision–recall
+trade-off), some **client- and class-level regressions** (e.g. X-IIoTID C&C, 5G-NIDD SYNFlood), and a
+clear **regression at 5G-NIDD α=0.50**. At α=0.25 the best tested beta is 0 (no benefit). The
+evidence does **not** support a consistent advantage, and no claim of statistical superiority is
+made.
+
+---
+
+### Source map
+
+| Claim group | Source |
+|---|---|
+| Pairing, global/per-client/per-class/confusion deltas | `scripts/deep_comparison.py` → `artifacts/analysis/deepcmp__*.csv` |
+| Selected betas | `artifacts/analysis/final__selection_strength.csv` |
+| Global test means/stds | `artifacts/analysis/final__six_condition_master.csv` |
+| Selected-vs-original descriptive deltas | `artifacts/analysis/final__selected_vs_original.csv` |
+| Prototype displacement | `artifacts/analysis/final__prototype_displacement_by_beta.csv` |
+| Per-client / per-class / confusion raw | `results/<run>/per_client.csv`, `test_per_class.csv`, `test_confusion_matrix.json` |
+| Figures I, J, K | `scripts/make_comparison_figures.py` → `artifacts/figures/final/figI/figJ/figK*.{png,pdf}` |