research: add paired original vs proposed PROTEAN analysis

README.md

@@ -583,7 +583,79 @@ difference; the matrices come from the same manifest, split, seed, and class ord
 ![Paired confusion matrices](artifacts/figures/final/figH_confusion_matrix_5gnidd_a0.75_seed0.png)
 
 
-## 17. Main Scientific Findings
+## 17. Deep Comparison: Original PROTEAN vs Proposed (Result)
+
+A dedicated paired analysis compares `original_protean` against
+`support_weighted` at the validation-selected beta for every dataset-alpha condition, using the
+tests split and hard-coded pairing checks (same dataset, alpha, seed, and `manifest_content_hash`).
+Full report: **`docs/research/ORIGINAL_VS_PROPOSED_DEEP_COMPARISON.md`**. Derived tables:
+`artifacts/analysis/deepcmp__*.csv` (produced by `scripts/deep_comparison.py`). Figures:
+`scripts/make_comparison_figures.py`.
+
+**Global test overview** (Δ = proposed − original, mean over seeds `{0,1,2}`; rare-class macro-F1 shown
+in full, other metrics as deltas):
+
+| Dataset | alpha | beta* | rare-F1 original | rare-F1 proposed | Δ rare-F1 | Δ rare-recall | Δ macro-F1 | Δ accuracy |
+|---|---|---|---|---|---|---|---|---|
+| X-IIoTID | 0.25 | 0.00 | 0.6328 | 0.6321 | −0.0007 | −0.0007 | −0.0007 | −0.0014 |
+| X-IIoTID | 0.50 | 0.75 | 0.6425 | 0.6655 | **+0.0230** | −0.0144 | +0.0222 | +0.0069 |
+| X-IIoTID | 0.75 | 0.50 | 0.6039 | 0.6052 | +0.0014 | −0.0077 | +0.0058 | +0.0075 |
+| 5G-NIDD | 0.25 | 0.00 | 0.8423 | 0.8412 | −0.0010 | +0.0007 | −0.0028 | −0.0007 |
+| 5G-NIDD | 0.50 | 1.00 | 0.8805 | 0.8720 | −0.0086 | −0.0058 | −0.0038 | −0.0007 |
+| 5G-NIDD | 0.75 | 0.50 | 0.8799 | 0.8967 | **+0.0168** | −0.0015 | +0.0117 | +0.0013 |
+
+Both `alpha=0.25` conditions selected `beta=0` and are numerically null. Among the four genuine
+`beta>0` conditions, rare-class macro-F1 improved in three and degraded in one; overall macro-F1 moves
+with it, while rare-class recall usually falls (precision–recall trade-off). 5G-NIDD a0.50 is
+seed-dominated. No statistical significance is claimed.
+
+### Fig I — Paired per-client test rare-class macro-F1 (proposed vs ORIGINAL PROTEAN)
+
+Each point is one (client, seed) test observation, faceted by dataset and alpha; charcoal/teal/red/gray
+and markers encode outcome and seed. At `beta>0` conditions the response is client-specific (e.g.
+X-IIoTID a0.50: 12/30 improved, 1 degraded, 17 unchanged), with individual changes larger than the
+condition mean; at `beta=0` conditions almost all observations are unchanged.
+
+![Paired per-client comparison](artifacts/figures/final/figI_per_client_original_vs_proposed.png)
+
+### Fig J — Per-class test F1 change (beta>0 conditions)
+
+Signed Δ F1 per class, panels per dataset-alpha; class support is annotated. Largest gains are
+X-IIoTID `Tampering` (+0.14) and `Reconnaissance` (+0.05) and 5G-NIDD `TCPConnectScan` (+0.03) /
+`ICMPFlood` (+0.10, tiny support); largest losses are X-IIoTID `C&C` (−0.06) and 5G-NIDD `SYNFlood`
+(−0.04). Benign detection is essentially unchanged.
+
+![Per-class comparison](artifacts/figures/final/figJ_per_class_original_vs_proposed.png)
+
+### Fig K — Confusion-matrix difference (proposed − original)
+
+For the paired condition 5G-NIDD, `alpha=0.75`, `seed=0` (identical manifest, split, class order,
+243,177 test samples): left = raw count difference, right = row-normalized rate difference. The
+dominant error `Benign → UDPFlood` is essentially unchanged (56,899 → 57,054, +155). The main reduced
+errors are `HTTPFlood → SlowrateDoS` (−547) and `Benign → TCPConnectScan` (−520); the main increased
+error is `SlowrateDoS → HTTPFlood` (+229). Red = positive delta, blue = negative; no significance is
+inferred.
+
+![Confusion-matrix difference](artifacts/figures/final/figK_confusion_delta_5gnidd_a0.75_seed0.png)
+
+**Per-client findings.** The proposed method redistributes benefit: within a condition some clients
+improve and others degrade, and the low-rare-support quartile is not systematically disadvantaged
+(descriptive; Pearson Δ-vs-support-fraction ≈ −0.09, n=120). Client ids are per-seed realizations.
+
+**Per-class findings.** Rare-class gains are partly precision-driven (recall often falls) and are not
+confined to rare classes; the largest regressions are on low-support attack classes (X-IIoTID `C&C`,
+5G-NIDD `SYNFlood`).
+
+**Confusion-delta findings.** The weighting does not fix the dominant Benign→UDPFlood error; the
+measurable benefit at the analysed pair is a reduced HTTPFlood↔SlowrateDoS confusion, with a small
+SlowrateDoS recall cost.
+
+**Practical implications and limitations.** Effects are small (≤ ~2.3 pp rare-F1) and
+condition-dependent; the largest gain is comparable to seed noise. Practical benefit is modest, not a
+blanket improvement, and comes with rare-recall and some client- or class-level costs. Pairing,
+metric definitions, and threats to validity are detailed in the report.
+
+## 18. Main Scientific Findings
 
 1. **Changing `beta` changes the global prototype.** Contribution weights become proportional to
    support, and for a fixed snapshot mean prototype displacement grows monotonically with `beta` in
@@ -602,7 +674,7 @@ difference; the matrices come from the same manifest, split, seed, and class ord
    not a proxy for detection quality.
 8. **The dominant classification error (Benign → UDPFlood) is not addressed by `beta`.**
 
-## 18. Limitations and Novelty Boundary
+## 19. Limitations and Novelty Boundary
 
 **Limitations.**
 
@@ -623,7 +695,7 @@ contribution is the controlled `beta` parameterization inside PROTEAN, the valid
 condition-wise selection, the paired multi-seed design, and the behavioral findings. No claim is
 made that the method universally improves PROTEAN.
 
-## 19. Repository Structure
+## 20. Repository Structure
 
 ```
 src/protean_research/   data loading/preprocessing/partitioning, models, aggregation, federated
@@ -643,7 +715,7 @@ artifacts/figures/final/ thesis-ready figures
 results/                local run outputs (git-ignored; 108 non-smoke run dirs + 1 smoke)
 ```
 
-## 20. Reproduction and Tests
+## 21. Reproduction and Tests
 
 ```bash
 # unit + integration tests (expected: 75 passed)
@@ -682,7 +754,7 @@ rerun the grid.
   the shared `S_j`.
 - `beta` selection uses validation only; the test set is never used for selection.
 
-## 21. Complete Artifact Index
+## 22. Complete Artifact Index
 
 Per-run outputs (`results/<dataset>__alpha<alpha>__seed<seed>__<config>/`):
 
@@ -720,7 +792,7 @@ Analysis outputs (`artifacts/analysis/`), by function:
 | Prototype snapshots & weights | `*__prototype_snapshot.json`, `*__prototype_weights.csv`, `*__prototype_effect*.csv` |
 
 Partition manifests: `artifacts/partitions/<dataset>__alpha<alpha>__seed<seed>.json` (18 files).
-Figures: `artifacts/figures/final/` — 9 figures (figA–figH) as PNG (220 dpi) plus matching vector PDF;
+Figures: `artifacts/figures/final/` — 12 figures (figA–figK) as PNG (220 dpi) plus matching vector PDF;
 regenerated reproducibly by `scripts/make_final_figures.py`.
 
 Key documents:
@@ -729,6 +801,7 @@ Key documents:
 |---|---|
 | `docs/research/EXPERIMENT_PROTOCOL.md` | frozen reproduction protocol |
 | `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md` | authoritative six-condition synthesis |
+| `docs/research/ORIGINAL_VS_PROPOSED_DEEP_COMPARISON.md` | paired original-vs-proposed deep comparison |
 | `docs/research/CROSS_DATASET_BETA_ANALYSIS.md` | cross-dataset comparison |
 | `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md` | X-IIoTID cross-alpha analysis |
 | `docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md` | 5G-NIDD cross-alpha analysis |
@@ -741,7 +814,7 @@ Key documents:
 | `docs/handoff/SESSION_HANDOFF.md` | current state and next steps |
 | `planningawal/PROTEAN_Revisi_Lurus.md` | immutable research specification |
 
-## 22. References
+## 23. References
 
 - Chennoufi, S., Han, Y., Blanc, G., De Cristofaro, E., & Kiennert, C. (2025). PROTEAN: Federated
   Intrusion Detection in Non-IID Environments through Prototype-Based Knowledge Sharing. ESORICS