diff --git a/README.md b/README.md index 4321835..239e84e 100644 --- a/README.md +++ b/README.md @@ -473,8 +473,10 @@ Representative confusion-matrix files: ## 16. Final Figures and Visualizations All figures live in `artifacts/figures/final/` and are regenerated from the audited -`artifacts/analysis/final__*` tables by `scripts/make_final_figures.py` (no training; every number is -loaded from an artifact). Each figure is saved as a high-resolution PNG (220 dpi) and a vector PDF. +`artifacts/analysis/final__*` tables by `scripts/make_final_figures.py` and +`scripts/make_comparison_figures.py` (no training; every number is loaded from an artifact). Each +figure is saved as a high-resolution PNG (220 dpi) **and** a matching vector PDF, and its supporting +source table is linked where useful. **Naming and legend conventions used throughout.** @@ -494,9 +496,12 @@ loaded from an artifact). Each figure is saved as a high-resolution PNG (220 dpi | B | Beta selection surface | none (selection, not performance) | | C | Dataset heterogeneity | none (data distribution) | | D, E | Fixed-snapshot comparison vs `beta = 0` | `beta = 0` control | -| F | Direct comparison vs `original_protean` | `original_protean` | -| G | Per-client comparison vs `beta = 0` | `beta = 0` control | -| H | Paired `original_protean` vs selected beta | `original_protean` | +| F | Direct comparison vs `original_protean` (test) | `original_protean` | +| G | Per-client comparison vs `beta = 0` (validation) | `beta = 0` control | +| H | Paired `original_protean` vs selected beta confusion matrices | `original_protean` | +| I1, I2 | Per-client test comparison vs `original_protean` | `original_protean` | +| J1, J2 | Per-class test comparison vs `original_protean` | `original_protean` | +| K | Ranked confusion change vs `original_protean` (+ full matrix supplement) | `original_protean` | ### Fig A1 / A2 — Validation-based beta selection (support_weighted) @@ -548,18 +553,23 @@ the imbalance-displacement link is mechanical under the beta formula and is not ![Support CV vs displacement](artifacts/figures/final/figE_support_cv_vs_displacement.png) -### Fig F — Selected-beta PROTEAN vs ORIGINAL PROTEAN (test) +### Fig F — Selected-beta PROTEAN vs ORIGINAL PROTEAN (test), two panels -**Compared:** `original_protean` (charcoal) against `support_weighted` at the validation-selected -beta (teal), per condition. **Level/split:** test, mean over seeds `{0,1,2}`. **Metric:** test -rare-class macro-F1; error bars = ±1 sample std across seeds. **Observation:** signed deltas are -`+0.0230`, `+0.0014`, `+0.0168` (positive), `−0.0086` (negative), and `−0.0007`/`−0.0010` at the two -`beta = 0` conditions (floating-point ~0). **Limits:** no statistical significance test (three -seeds); the largest gain is within seed noise; the weighting does not consistently outperform -`original_protean`. +**Compared:** `original_protean` against `support_weighted` at the validation-selected beta, per +condition, test split. **How to read:** Panel A shows the absolute mean test rare-class macro-F1 +(±1 sample std across seeds) and is deliberately near-flat; Panel B shows the signed difference +`proposed − original` with the exact value printed at each bar and the selected beta annotated. +**Main result:** three positive deltas (X-IIoTID a0.50 +0.0230, X-IIoTID a0.75 +0.0014, 5G-NIDD a0.75 ++0.0168), one negative delta (5G-NIDD a0.50 −0.0086), and two `beta=0` conditions that are +effectively unchanged (−0.0007 / −0.0010). **Limitation:** no significance test (three seeds); a +positive mean delta is not a proven improvement. ![Selected vs original test rare macro-F1](artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.png) +Links: [PNG](artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.png) · +[PDF](artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.pdf) · source +[`final__six_condition_master.csv`](artifacts/analysis/final__six_condition_master.csv). + ### Fig G — Per-client effect of selected beta relative to the beta=0 control **Compared:** selected-beta `support_weighted` against the `beta = 0` control (not @@ -571,17 +581,25 @@ client ids are per-seed realizations and are not pooled. ![Client effect at selected beta](artifacts/figures/final/figG_client_effect_at_selected_beta.png) -### Fig H (supplemental) — Paired test confusion matrices +### Fig H — Paired test confusion matrices (original vs beta-tuned) **Compared:** `original_protean` (left) against selected-beta `support_weighted` (right) for a valid -paired condition (5G-NIDD, `alpha = 0.75`, `seed = 0`). **Level/split:** test; global counts over the -evaluation split. **Metric:** raw 9×9 counts (log colour scale), identical class order and colour -scale. **Observation:** the dominant off-diagonal error is `Benign → UDPFlood` (~56.9k vs ~57.1k) and -is essentially unchanged by beta. **Limits:** no statistical significance is inferred from the visual -difference; the matrices come from the same manifest, split, seed, and class order. +paired condition (5G-NIDD, `alpha = 0.75`, `seed = 0`), test split. **How to read:** rows = actual +class, columns = predicted class; the diagonal is correct and off-diagonal is misclassified. Raw +counts, identical class order and colour scale; both matrices contain the same 243,177 test samples. +**Main result:** the dominant residual error `Benign → UDPFlood` is essentially unchanged (56,899 → +57,054, +155). The summary below the figure lists the most important changes (`HTTPFlood → +SlowrateDoS` −547; `Benign → TCPConnectScan` −520; `SlowrateDoS → HTTPFlood` +229; diagonal +`HTTPFlood` +547, `Benign` +288, `SlowrateDoS` −239). **Limitation:** no significance test; the +`Benign → UDPFlood` increase is **not** an improvement. ![Paired confusion matrices](artifacts/figures/final/figH_confusion_matrix_5gnidd_a0.75_seed0.png) +Links: [PNG](artifacts/figures/final/figH_confusion_matrix_5gnidd_a0.75_seed0.png) · +[PDF](artifacts/figures/final/figH_confusion_matrix_5gnidd_a0.75_seed0.pdf) · source +`results/5gnidd__alpha0.75__seed0__original_protean/test_confusion_matrix.json` and +`results/5gnidd__alpha0.75__seed0__support_weighted__beta0.5/test_confusion_matrix.json`. + ## 17. Deep Comparison: Original PROTEAN vs Proposed (Result) @@ -609,34 +627,71 @@ Both `alpha=0.25` conditions selected `beta=0` and are numerically null. Among t with it, while rare-class recall usually falls (precision–recall trade-off). 5G-NIDD a0.50 is seed-dominated. No statistical significance is claimed. -### Fig I — Paired per-client test rare-class macro-F1 (proposed vs ORIGINAL PROTEAN) +### Fig I1 / I2 — Paired per-client test rare-class macro-F1 (proposed vs ORIGINAL PROTEAN) -Each point is one (client, seed) test observation, faceted by dataset and alpha; charcoal/teal/red/gray -and markers encode outcome and seed. At `beta>0` conditions the response is client-specific (e.g. -X-IIoTID a0.50: 12/30 improved, 1 degraded, 17 unchanged), with individual changes larger than the -condition mean; at `beta=0` conditions almost all observations are unchanged. +One figure per dataset, three alpha panels. **Compared:** `original_protean` against selected-beta +`support_weighted` on the test split, each client on its own evaluation partition. **How to read:** +x = client id, one marker per seed (circle = seed 0, square = seed 1, triangle = seed 2), +y = Δ rare-class macro-F1; the shaded band is the ±0.02 effectively-unchanged threshold; the strongest +positive and negative observation in each panel is labelled `c s`. **Main result:** +X-IIoTID α=0.25 essentially unchanged (1 / 1 / 28 improved/degraded/unchanged), α=0.50 12 / 1 / 17 +(mean +0.030, range −0.257 … +0.324), α=0.75 8 / 2 / 20; 5G-NIDD α=0.25 1 / 0 / 29, α=0.50 15 / 6 / 9 +(seed-dominated), α=0.75 8 / 1 / 21. **Limitation:** client ids are per-seed realizations (id `k` is +not the same entity across seeds); at α=0.25 the selected beta is 0 so the residual spread is +numerical, not a weighting effect. -![Paired per-client comparison](artifacts/figures/final/figI_per_client_original_vs_proposed.png) +![Per-client comparison, X-IIoTID](artifacts/figures/final/figI1_xiiotid_per_client_original_vs_proposed.png) -### Fig J — Per-class test F1 change (beta>0 conditions) +![Per-client comparison, 5G-NIDD](artifacts/figures/final/figI2_5gnidd_per_client_original_vs_proposed.png) -Signed Δ F1 per class, panels per dataset-alpha; class support is annotated. Largest gains are -X-IIoTID `Tampering` (+0.14) and `Reconnaissance` (+0.05) and 5G-NIDD `TCPConnectScan` (+0.03) / -`ICMPFlood` (+0.10, tiny support); largest losses are X-IIoTID `C&C` (−0.06) and 5G-NIDD `SYNFlood` -(−0.04). Benign detection is essentially unchanged. +Links: [I1 PNG](artifacts/figures/final/figI1_xiiotid_per_client_original_vs_proposed.png) · +[I1 PDF](artifacts/figures/final/figI1_xiiotid_per_client_original_vs_proposed.pdf) · +[I2 PNG](artifacts/figures/final/figI2_5gnidd_per_client_original_vs_proposed.png) · +[I2 PDF](artifacts/figures/final/figI2_5gnidd_per_client_original_vs_proposed.pdf) · source +[`deepcmp__per_client.csv`](artifacts/analysis/deepcmp__per_client.csv). -![Per-class comparison](artifacts/figures/final/figJ_per_class_original_vs_proposed.png) +### Fig J1 / J2 — Per-class test F1 change (beta>0 conditions) -### Fig K — Confusion-matrix difference (proposed − original) +One figure per dataset, two alpha panels. **Compared:** `original_protean` against selected-beta +`support_weighted`, test split. **How to read:** horizontal bars are the mean ΔF1 over seeds `{0,1,2}` +(teal = increased, red = decreased) with the exact signed ΔF1 printed at each bar; test support `n≈` +is on a second label line; both panels share one x-scale so bar lengths are comparable. **Main +result:** X-IIoTID largest gains `Tampering` +0.1438 and `Reconnaissance` +0.0542, largest loss `C&C` +−0.0556; 5G-NIDD largest gain `TCPConnectScan` +0.0342 / `ICMPFlood` +0.0970 (tiny support), largest +loss `SYNFlood` −0.0426. **Limitation:** the delta is **F1**, not precision or recall (e.g. Tampering ++0.1438 F1 corresponds to +0.2120 precision and −0.0091 recall); no significance test. + +![Per-class comparison, X-IIoTID](artifacts/figures/final/figJ1_xiiotid_per_class_original_vs_proposed.png) + +![Per-class comparison, 5G-NIDD](artifacts/figures/final/figJ2_5gnidd_per_class_original_vs_proposed.png) + +Links: [J1 PNG](artifacts/figures/final/figJ1_xiiotid_per_class_original_vs_proposed.png) · +[J1 PDF](artifacts/figures/final/figJ1_xiiotid_per_class_original_vs_proposed.pdf) · +[J2 PNG](artifacts/figures/final/figJ2_5gnidd_per_class_original_vs_proposed.png) · +[J2 PDF](artifacts/figures/final/figJ2_5gnidd_per_class_original_vs_proposed.pdf) · source +[`deepcmp__per_class.csv`](artifacts/analysis/deepcmp__per_class.csv). + +### Fig K — Ranked confusion change vs ORIGINAL PROTEAN (+ full-matrix supplement) For the paired condition 5G-NIDD, `alpha=0.75`, `seed=0` (identical manifest, split, class order, -243,177 test samples): left = raw count difference, right = row-normalized rate difference. The -dominant error `Benign → UDPFlood` is essentially unchanged (56,899 → 57,054, +155). The main reduced -errors are `HTTPFlood → SlowrateDoS` (−547) and `Benign → TCPConnectScan` (−520); the main increased -error is `SlowrateDoS → HTTPFlood` (+229). Red = positive delta, blue = negative; no significance is -inferred. +243,177 test samples). **How to read:** the ranked view lists the largest off-diagonal changes as raw +test counts; the upper panel shows *more* errors after weighting, the lower panel *fewer*. +**Main result:** the largest reduced errors are `HTTPFlood → SlowrateDoS` (−547; 759 → 212) and +`Benign → TCPConnectScan` (−520; 2,015 → 1,495); the largest increased errors are `SlowrateDoS → +HTTPFlood` (+229) and `Benign → UDPFlood` (+155). The dominant error `Benign → UDPFlood` is essentially +unchanged and is **not** an improvement. **Limitation:** no significance test; the figure is a single +paired condition, not a condition average. The full difference matrices (raw count and row-rate) are +kept as a supplemental figure for traceability. -![Confusion-matrix difference](artifacts/figures/final/figK_confusion_delta_5gnidd_a0.75_seed0.png) +![Ranked confusion change](artifacts/figures/final/figK_confusion_error_changes_5gnidd_a0.75_seed0.png) + +![Full confusion-matrix difference, supplemental](artifacts/figures/final/figK_confusion_delta_5gnidd_a0.75_seed0.png) + +Links: [ranked PNG](artifacts/figures/final/figK_confusion_error_changes_5gnidd_a0.75_seed0.png) · +[ranked PDF](artifacts/figures/final/figK_confusion_error_changes_5gnidd_a0.75_seed0.pdf) · +[full-matrix PNG](artifacts/figures/final/figK_confusion_delta_5gnidd_a0.75_seed0.png) · +[full-matrix PDF](artifacts/figures/final/figK_confusion_delta_5gnidd_a0.75_seed0.pdf) · source +[`deepcmp__confusion_delta.csv`](artifacts/analysis/deepcmp__confusion_delta.csv). **Per-client findings.** The proposed method redistributes benefit: within a condition some clients improve and others degrade, and the low-rare-support quartile is not systematically disadvantaged @@ -792,7 +847,8 @@ Analysis outputs (`artifacts/analysis/`), by function: | Prototype snapshots & weights | `*__prototype_snapshot.json`, `*__prototype_weights.csv`, `*__prototype_effect*.csv` | Partition manifests: `artifacts/partitions/__alpha__seed.json` (18 files). -Figures: `artifacts/figures/final/` — 12 figures (figA–figK) as PNG (220 dpi) plus matching vector PDF; +Figures: `artifacts/figures/final/` — 15 figures (figA–figK, including the split I1/I2 and J1/J2 and +the K full-matrix supplement) as PNG (220 dpi) plus matching vector PDF; regenerated reproducibly by `scripts/make_final_figures.py`. Key documents: diff --git a/artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.pdf b/artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.pdf index 8962651..c94028b 100644 Binary files a/artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.pdf and b/artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.pdf differ diff --git a/artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.png b/artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.png index c994564..9ff0e4d 100644 Binary files a/artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.png and b/artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.png differ diff --git a/artifacts/figures/final/figH_confusion_matrix_5gnidd_a0.75_seed0.pdf b/artifacts/figures/final/figH_confusion_matrix_5gnidd_a0.75_seed0.pdf index 0527396..04a6b3f 100644 Binary files a/artifacts/figures/final/figH_confusion_matrix_5gnidd_a0.75_seed0.pdf and b/artifacts/figures/final/figH_confusion_matrix_5gnidd_a0.75_seed0.pdf differ diff --git a/artifacts/figures/final/figH_confusion_matrix_5gnidd_a0.75_seed0.png b/artifacts/figures/final/figH_confusion_matrix_5gnidd_a0.75_seed0.png index a65f9ba..8def404 100644 Binary files a/artifacts/figures/final/figH_confusion_matrix_5gnidd_a0.75_seed0.png and b/artifacts/figures/final/figH_confusion_matrix_5gnidd_a0.75_seed0.png differ diff --git a/artifacts/figures/final/figI_per_client_original_vs_proposed.pdf b/artifacts/figures/final/figI1_xiiotid_per_client_original_vs_proposed.pdf similarity index 57% rename from artifacts/figures/final/figI_per_client_original_vs_proposed.pdf rename to artifacts/figures/final/figI1_xiiotid_per_client_original_vs_proposed.pdf index 912ba34..9f7af1c 100644 Binary files a/artifacts/figures/final/figI_per_client_original_vs_proposed.pdf and b/artifacts/figures/final/figI1_xiiotid_per_client_original_vs_proposed.pdf differ diff --git a/artifacts/figures/final/figI1_xiiotid_per_client_original_vs_proposed.png b/artifacts/figures/final/figI1_xiiotid_per_client_original_vs_proposed.png new file mode 100644 index 0000000..8602b93 Binary files /dev/null and b/artifacts/figures/final/figI1_xiiotid_per_client_original_vs_proposed.png differ diff --git a/artifacts/figures/final/figI2_5gnidd_per_client_original_vs_proposed.pdf b/artifacts/figures/final/figI2_5gnidd_per_client_original_vs_proposed.pdf new file mode 100644 index 0000000..053d602 Binary files /dev/null and b/artifacts/figures/final/figI2_5gnidd_per_client_original_vs_proposed.pdf differ diff --git a/artifacts/figures/final/figI2_5gnidd_per_client_original_vs_proposed.png b/artifacts/figures/final/figI2_5gnidd_per_client_original_vs_proposed.png new file mode 100644 index 0000000..31c4f53 Binary files /dev/null and b/artifacts/figures/final/figI2_5gnidd_per_client_original_vs_proposed.png differ diff --git a/artifacts/figures/final/figI_per_client_original_vs_proposed.png b/artifacts/figures/final/figI_per_client_original_vs_proposed.png deleted file mode 100644 index 2fbf903..0000000 Binary files a/artifacts/figures/final/figI_per_client_original_vs_proposed.png and /dev/null differ diff --git a/artifacts/figures/final/figJ_per_class_original_vs_proposed.pdf b/artifacts/figures/final/figJ1_xiiotid_per_class_original_vs_proposed.pdf similarity index 60% rename from artifacts/figures/final/figJ_per_class_original_vs_proposed.pdf rename to artifacts/figures/final/figJ1_xiiotid_per_class_original_vs_proposed.pdf index fad8f1e..0b42b2a 100644 Binary files a/artifacts/figures/final/figJ_per_class_original_vs_proposed.pdf and b/artifacts/figures/final/figJ1_xiiotid_per_class_original_vs_proposed.pdf differ diff --git a/artifacts/figures/final/figJ1_xiiotid_per_class_original_vs_proposed.png b/artifacts/figures/final/figJ1_xiiotid_per_class_original_vs_proposed.png new file mode 100644 index 0000000..90df82a Binary files /dev/null and b/artifacts/figures/final/figJ1_xiiotid_per_class_original_vs_proposed.png differ diff --git a/artifacts/figures/final/figJ2_5gnidd_per_class_original_vs_proposed.pdf b/artifacts/figures/final/figJ2_5gnidd_per_class_original_vs_proposed.pdf new file mode 100644 index 0000000..9a1ea04 Binary files /dev/null and b/artifacts/figures/final/figJ2_5gnidd_per_class_original_vs_proposed.pdf differ diff --git a/artifacts/figures/final/figJ2_5gnidd_per_class_original_vs_proposed.png b/artifacts/figures/final/figJ2_5gnidd_per_class_original_vs_proposed.png new file mode 100644 index 0000000..47106ab Binary files /dev/null and b/artifacts/figures/final/figJ2_5gnidd_per_class_original_vs_proposed.png differ diff --git a/artifacts/figures/final/figJ_per_class_original_vs_proposed.png b/artifacts/figures/final/figJ_per_class_original_vs_proposed.png deleted file mode 100644 index edb1f77..0000000 Binary files a/artifacts/figures/final/figJ_per_class_original_vs_proposed.png and /dev/null differ diff --git a/artifacts/figures/final/figK_confusion_delta_5gnidd_a0.75_seed0.pdf b/artifacts/figures/final/figK_confusion_delta_5gnidd_a0.75_seed0.pdf index 7c0ee78..09e2c27 100644 Binary files a/artifacts/figures/final/figK_confusion_delta_5gnidd_a0.75_seed0.pdf and b/artifacts/figures/final/figK_confusion_delta_5gnidd_a0.75_seed0.pdf differ diff --git a/artifacts/figures/final/figK_confusion_delta_5gnidd_a0.75_seed0.png b/artifacts/figures/final/figK_confusion_delta_5gnidd_a0.75_seed0.png index f90e630..0fa071e 100644 Binary files a/artifacts/figures/final/figK_confusion_delta_5gnidd_a0.75_seed0.png and b/artifacts/figures/final/figK_confusion_delta_5gnidd_a0.75_seed0.png differ diff --git a/artifacts/figures/final/figK_confusion_error_changes_5gnidd_a0.75_seed0.pdf b/artifacts/figures/final/figK_confusion_error_changes_5gnidd_a0.75_seed0.pdf new file mode 100644 index 0000000..45b966b Binary files /dev/null and b/artifacts/figures/final/figK_confusion_error_changes_5gnidd_a0.75_seed0.pdf differ diff --git a/artifacts/figures/final/figK_confusion_error_changes_5gnidd_a0.75_seed0.png b/artifacts/figures/final/figK_confusion_error_changes_5gnidd_a0.75_seed0.png new file mode 100644 index 0000000..52bbd27 Binary files /dev/null and b/artifacts/figures/final/figK_confusion_error_changes_5gnidd_a0.75_seed0.png differ diff --git a/docs/research/ORIGINAL_VS_PROPOSED_DEEP_COMPARISON.md b/docs/research/ORIGINAL_VS_PROPOSED_DEEP_COMPARISON.md index 8a4d45e..b64df7f 100644 --- a/docs/research/ORIGINAL_VS_PROPOSED_DEEP_COMPARISON.md +++ b/docs/research/ORIGINAL_VS_PROPOSED_DEEP_COMPARISON.md @@ -133,7 +133,9 @@ classes while giving up a little rare-class coverage. ## 6. Paired per-client results (test, vs original_protean) -Figure I: `artifacts/figures/final/figI_per_client_original_vs_proposed.png`. +Figures I1/I2 (one per dataset): +`artifacts/figures/final/figI1_xiiotid_per_client_original_vs_proposed.png`, +`artifacts/figures/final/figI2_5gnidd_per_client_original_vs_proposed.png`. Per-observation table: `deepcmp__per_client.csv` (30 observations per condition). | Condition | beta* | improved | degraded | unchanged | mean Δ | min Δ | max Δ | @@ -159,7 +161,9 @@ Counts use the ±0.02 effectively-unchanged band on per-client test rare-class m ## 7. Per-class precision/recall/F1 changes (beta>0 conditions) -Figure J: `artifacts/figures/final/figJ_per_class_original_vs_proposed.png`. +Figures J1/J2 (one per dataset): +`artifacts/figures/final/figJ1_xiiotid_per_class_original_vs_proposed.png`, +`artifacts/figures/final/figJ2_5gnidd_per_class_original_vs_proposed.png`. Source: `deepcmp__per_class.csv` (per seed) aggregated as the mean delta over the three seeds; class support is the mean test support over seeds. X-IIoTID and 5G-NIDD labels are never mixed. @@ -198,8 +202,11 @@ Points to note: ## 8. Confusion-matrix difference analysis -Figure K shows the paired difference for **5G-NIDD, alpha=0.75, seed=0** (the same pair as Figure H): -`artifacts/figures/final/figK_confusion_delta_5gnidd_a0.75_seed0.png`. Rows = true class, columns = +The ranked-change figure (primary) is +`artifacts/figures/final/figK_confusion_error_changes_5gnidd_a0.75_seed0.png`; the full difference +matrices (raw count and row-rate) are kept as the supplemental +`artifacts/figures/final/figK_confusion_delta_5gnidd_a0.75_seed0.png`. Both describe the paired +difference for **5G-NIDD, alpha=0.75, seed=0** (the same pair as Figure H). Rows = true class, columns = predicted class; both matrices have identical class order and 243,177 test samples. Source: `deepcmp__confusion_delta.csv`. @@ -227,8 +234,8 @@ Largest raw-count changes for this pair (proposed − original): are X-IIoTID `Reconnaissance → Exploitation` (−6,773), `Reconnaissance → Weaponization` (−3,398), `Lateral_movement → Tampering` (−1,699) and 5G-NIDD `Benign → TCPConnectScan` (−1,484); the largest **increased** ones are X-IIoTID `Normal → C&C` (+2,546) and `Weaponization → Exploitation` (+1,286). - Raw counts are support-sensitive, so these are descriptive; the row-normalized view (Figure K, - right panel) is the support-fair comparison. + Raw counts are support-sensitive, so these are descriptive; the row-normalized view in the + supplemental Figure K matrix (right panel) is the support-fair comparison. ## 9. Support imbalance versus client outcomes diff --git a/scripts/make_comparison_figures.py b/scripts/make_comparison_figures.py index ac46800..868ae59 100644 --- a/scripts/make_comparison_figures.py +++ b/scripts/make_comparison_figures.py @@ -1,13 +1,16 @@ #!/usr/bin/env python3 """Figures I, J, K: original_protean vs support_weighted(selected beta). -Figure I paired per-client rare-class F1 deltas (test), vs original_protean. -Figure J per-class test F1 change, beta>0 conditions. -Figure K test confusion-matrix difference (5G-NIDD, alpha=0.75, seed=0). +Readability overhaul. Produces: -Reads derived tables written by ``scripts/deep_comparison.py`` and the raw -per-run confusion matrices. Writes PNG (220 dpi) and PDF to -``artifacts/figures/final``. No training, no result mutation. + figI1 / figI2 paired per-client test rare-class F1 deltas, one figure per dataset + figJ1 / figJ2 per-class test F1 change, one figure per dataset (two alpha panels) + figK_confusion_error_changes_* ranked reduced/increased misclassifications (primary) + figK_confusion_delta_* full difference matrix (supplemental, preserved) + +Reads derived tables from ``scripts/deep_comparison.py`` and raw confusion +matrices. Writes PNG (220 dpi) and PDF to ``artifacts/figures/final``. +No training, no result mutation. Run: .venv/bin/python scripts/make_comparison_figures.py @@ -17,9 +20,9 @@ from __future__ import annotations import csv import json +import statistics from collections import defaultdict from pathlib import Path -import statistics import matplotlib @@ -35,11 +38,13 @@ OUT = ROOT / "artifacts" / "figures" / "final" CHARCOAL = "#3A3A3A" TEAL = "#0E7C7B" +GREEN = "#1B7F3B" RED = "#C0392B" GREY = "#9A9A9A" -DATASETS = ["xiiotid", "5gnidd"] DATASET_NAME = {"xiiotid": "X-IIoTID", "5gnidd": "5G-NIDD"} ALPHAS = [0.25, 0.50, 0.75] +SELECTED = {"xiiotid": {0.25: 0.0, 0.5: 0.75, 0.75: 0.5}, + "5gnidd": {0.25: 0.0, 0.5: 1.0, 0.75: 0.5}} TOL = 0.02 SEED_MARKERS = {0: "o", 1: "s", 2: "^"} @@ -48,10 +53,10 @@ def _style(): plt.rcParams.update({ "figure.facecolor": "white", "axes.facecolor": "white", "axes.edgecolor": "#5A5A5A", "axes.linewidth": 0.9, - "axes.titlesize": 11.5, "axes.titleweight": "bold", - "axes.labelsize": 10.5, "xtick.labelsize": 9, - "ytick.labelsize": 9, "legend.fontsize": 9, - "font.size": 10, "axes.spines.top": False, "axes.spines.right": False, + "axes.titlesize": 12, "axes.titleweight": "bold", + "axes.labelsize": 11, "xtick.labelsize": 10, "ytick.labelsize": 10, + "legend.fontsize": 9.5, "font.size": 10.5, + "axes.spines.top": False, "axes.spines.right": False, "grid.color": "#B0B0B0", "grid.alpha": 0.3, "grid.linestyle": ":", "savefig.dpi": 220, "savefig.bbox": "tight", }) @@ -71,102 +76,175 @@ def _read(path): # --------------------------------------------------------------------------- -def figure_i(): - rows = _read(ANALYSIS / "deepcmp__per_client.csv") - fig, axes = plt.subplots(2, 3, figsize=(14.0, 7.4), constrained_layout=True, - sharey=True) - ylim = 0.47 - for ri, ds in enumerate(DATASETS): - for ci, alpha in enumerate(ALPHAS): - ax = axes[ri][ci] - ax.set_axisbelow(True) - ax.grid(axis="y") - for seed in (0, 1, 2): - sub = [r for r in rows if r["dataset"] == ds - and abs(float(r["alpha"]) - alpha) < 1e-9 - and int(r["seed"]) == seed] - xs = [int(r["client_id"]) for r in sub] - ys = [float(r["delta_rare_f1"]) for r in sub] - cols = [TEAL if y > TOL else (RED if y < -TOL else GREY) for y in ys] - ax.scatter(xs, ys, s=48, marker=SEED_MARKERS[seed], c=cols, - edgecolor="#333333", linewidth=0.5, alpha=0.9, zorder=4) - ax.axhspan(-TOL, TOL, color="#DDDDDD", alpha=0.35, zorder=1) - ax.axhline(0.0, color="#555555", lw=1.0, zorder=2) - beta = {"xiiotid": {0.25: 0.0, 0.5: 0.75, 0.75: 0.5}, - "5gnidd": {0.25: 0.0, 0.5: 1.0, 0.75: 0.5}}[ds][alpha] - deltas = [float(r["delta_rare_f1"]) for r in rows - if r["dataset"] == ds and abs(float(r["alpha"]) - alpha) < 1e-9] - ax.set_ylim(-ylim, ylim) - ax.set_xlim(-0.7, 9.7) - ax.set_xticks(range(10)) - ax.set_title(f"{DATASET_NAME[ds]} \u03b1={alpha:.2f} " - f"(beta={beta:g})") - ax.text(0.98, 0.04, f"mean \u0394={statistics.mean(deltas):+.3f}", - transform=ax.transAxes, ha="right", va="bottom", - fontsize=8.5, color="#333333") - if ri == 1: - ax.set_xlabel("client id (per-seed partition)") - if ci == 0: - ax.set_ylabel("test rare-class macro-F1\n\u0394 (proposed \u2212 original)") - handles = [plt.Line2D([0], [0], marker="o", ls="", ms=7, mfc=TEAL, mec="#333", label="improved (> +0.02)"), - plt.Line2D([0], [0], marker="o", ls="", ms=7, mfc=RED, mec="#333", label="degraded (< -0.02)"), - plt.Line2D([0], [0], marker="o", ls="", ms=7, mfc=GREY, mec="#333", label="unchanged (|\u0394| \u2264 0.02)"), - plt.Line2D([0], [0], marker="o", ls="", ms=7, mfc="white", mec="#333", label="seed 0"), - plt.Line2D([0], [0], marker="s", ls="", ms=7, mfc="white", mec="#333", label="seed 1"), - plt.Line2D([0], [0], marker="^", ls="", ms=7, mfc="white", mec="#333", label="seed 2")] +# Figure I: paired per-client rare-class F1 vs original_protean (one per dataset) +# --------------------------------------------------------------------------- +def figure_i(dataset: str) -> None: + rows = [r for r in _read(ANALYSIS / "deepcmp__per_client.csv") + if r["dataset"] == dataset] + all_d = [float(r["delta_rare_f1"]) for r in rows] + ymax = max(abs(min(all_d)), abs(max(all_d))) * 1.12 + + fig, axes = plt.subplots(1, 3, figsize=(16.0, 5.6), sharey=True, + constrained_layout=True) + for ax, alpha in zip(axes, ALPHAS): + ax.set_axisbelow(True) + ax.grid(axis="y") + ax.axhspan(-TOL, TOL, color="#DDDDDD", alpha=0.4, zorder=1) + ax.axhline(0.0, color="#444444", lw=1.1, zorder=2) + sub = [r for r in rows if abs(float(r["alpha"]) - alpha) < 1e-9] + for seed in (0, 1, 2): + pts = [r for r in sub if int(r["seed"]) == seed] + xs = [int(r["client_id"]) + (seed - 1) * 0.18 for r in pts] + ys = [float(r["delta_rare_f1"]) for r in pts] + cols = [TEAL if y > TOL else (RED if y < -TOL else GREY) for y in ys] + ax.scatter(xs, ys, s=62, marker=SEED_MARKERS[seed], c=cols, + edgecolor="#333333", linewidth=0.6, zorder=4) + # annotate the strongest positive and negative observation + for r in (max(sub, key=lambda r: float(r["delta_rare_f1"])), + min(sub, key=lambda r: float(r["delta_rare_f1"]))): + y = float(r["delta_rare_f1"]) + ax.annotate(f"c{r['client_id']} s{r['seed']}", (int(r["client_id"]), y), + textcoords="offset points", xytext=(6, 6 if y >= 0 else -12), + fontsize=8.2, color="#333333") + imp = sum(1 for r in sub if float(r["delta_rare_f1"]) > TOL) + deg = sum(1 for r in sub if float(r["delta_rare_f1"]) < -TOL) + unc = len(sub) - imp - deg + ax.text(0.5, 0.985, f"improved {imp} | degraded {deg} | unchanged {unc}", + transform=ax.transAxes, ha="center", va="top", fontsize=9, + bbox=dict(boxstyle="round", fc="white", ec="#CCCCCC")) + ax.set_ylim(-ymax, ymax) + ax.set_xlim(-0.75, 9.75) + ax.set_xticks(range(10)) + ax.set_xlabel("client id (per-seed partition)") + beta = SELECTED[dataset][alpha] + ax.set_title(f"{DATASET_NAME[dataset]} \u03b1={alpha:.2f} (beta={beta:g})") + axes[0].set_ylabel("test rare-class macro-F1\n\u0394 (proposed \u2212 ORIGINAL PROTEAN)") + handles = [plt.Line2D([0], [0], marker="o", ls="", ms=8, mfc=TEAL, mec="#333", label=f"improved (> +{TOL})"), + plt.Line2D([0], [0], marker="o", ls="", ms=8, mfc=RED, mec="#333", label=f"degraded (< \u2212{TOL})"), + plt.Line2D([0], [0], marker="o", ls="", ms=8, mfc=GREY, mec="#333", label=f"unchanged (|\u0394| \u2264 {TOL})"), + plt.Line2D([0], [0], marker="o", ls="", ms=8, mfc="white", mec="#333", label="seed 0"), + plt.Line2D([0], [0], marker="s", ls="", ms=8, mfc="white", mec="#333", label="seed 1"), + plt.Line2D([0], [0], marker="^", ls="", ms=8, mfc="white", mec="#333", label="seed 2")] fig.legend(handles=handles, loc="upper center", bbox_to_anchor=(0.5, 1.05), ncols=6, frameon=True, edgecolor="#CCCCCC") - fig.suptitle("Paired per-client test rare-class macro-F1: proposed (selected beta) vs ORIGINAL PROTEAN", - fontsize=12.5, weight="bold", y=1.11) - fig.text(0.5, -0.03, - "Each point is one (client, seed) test observation; x = client id, marker = seed. Points compare the " - "support_weighted run at the validation-selected beta against original_protean on the same partition. " - "At \u03b1=0.25 the selected beta is 0, so the residual spread is numerical (equal-weight equivalence), not a weighting effect. " - "Do not treat client id across seeds as the same real-world client.", - ha="center", va="top", fontsize=8.4, color="#444444", linespacing=1.5) - _save(fig, "figI_per_client_original_vs_proposed") + fig.suptitle(f"Per-client test rare-class macro-F1: proposed vs ORIGINAL PROTEAN \u2014 {DATASET_NAME[dataset]}", + fontsize=13, weight="bold", y=1.11) + fig.text(0.5, -0.04, + "Each point is one (client, seed) test observation on the client's own evaluation partition; same partition for both runs. " + "The shaded band is the \u00b10.02 effectively-unchanged threshold. Client ids are per-seed realizations (id k is not the same entity across seeds). " + "At \u03b1=0.25 the selected beta is 0, so the residual spread is numerical, not a weighting effect.", + ha="center", va="top", fontsize=8.6, color="#444444", linespacing=1.5) + _save(fig, f"figI{'1' if dataset == 'xiiotid' else '2'}_{dataset}_per_client_original_vs_proposed") # --------------------------------------------------------------------------- -def figure_j(): - rows = _read(ANALYSIS / "deepcmp__per_class.csv") - fig, axes = plt.subplots(2, 2, figsize=(13.4, 8.2), constrained_layout=True) - for ri, ds in enumerate(DATASETS): - for ci, alpha in enumerate([0.5, 0.75]): - ax = axes[ri][ci] - ax.set_axisbelow(True) - ax.grid(axis="x") - stats = defaultdict(lambda: {"f1": [], "sup": []}) - for r in rows: - if r["dataset"] == ds and abs(float(r["alpha"]) - alpha) < 1e-9: - stats[r["class_name"]]["f1"].append(float(r["delta_f1"])) - stats[r["class_name"]]["sup"].append(int(r["support"])) - names = sorted(stats, key=lambda n: statistics.mean(stats[n]["f1"])) - ys = np.arange(len(names)) - vals = [statistics.mean(stats[n]["f1"]) for n in names] - sup = [int(statistics.mean(stats[n]["sup"])) for n in names] - cols = [TEAL if v >= 0 else RED for v in vals] - ax.barh(ys, vals, color=cols, edgecolor="#333333", linewidth=0.6, zorder=3) - ax.axvline(0.0, color="#555555", lw=1.0, zorder=2) - ax.set_yticks(ys) - ax.set_yticklabels([f"{n} (n\u2248{s:,})" for n, s in zip(names, sup)], - fontsize=8.2) - beta = {"xiiotid": {0.5: 0.75, 0.75: 0.5}, - "5gnidd": {0.5: 1.0, 0.75: 0.5}}[ds][alpha] - ax.set_title(f"{DATASET_NAME[ds]} \u03b1={alpha:.2f} beta={beta:g}") - ax.set_xlabel("\u0394 F1 (proposed \u2212 original), test") - fig.suptitle("Per-class test F1 change for beta>0 conditions (mean over seeds {0,1,2})", +# Figure J: per-class test F1 change (one per dataset; two alpha panels) +# --------------------------------------------------------------------------- +def figure_j(dataset: str) -> None: + rows = [r for r in _read(ANALYSIS / "deepcmp__per_class.csv") + if r["dataset"] == dataset] + alphas = [0.5, 0.75] + stats = {} + for alpha in alphas: + agg = defaultdict(lambda: {"f1": [], "sup": []}) + for r in rows: + if abs(float(r["alpha"]) - alpha) < 1e-9: + agg[r["class_name"]]["f1"].append(float(r["delta_f1"])) + agg[r["class_name"]]["sup"].append(int(r["support"])) + stats[alpha] = {n: (statistics.mean(v["f1"]), int(statistics.mean(v["sup"]))) + for n, v in agg.items()} + amax = max(abs(v[0]) for a in alphas for v in stats[a].values()) + xlim = amax * 1.35 + + fig, axes = plt.subplots(1, 2, figsize=(15.5, 6.4), constrained_layout=True) + for ax, alpha in zip(axes, alphas): + ax.set_axisbelow(True) + ax.grid(axis="x") + items = sorted(stats[alpha].items(), key=lambda kv: kv[1][0]) + names = [n for n, _ in items] + vals = [v[0] for _, v in items] + sups = [v[1] for _, v in items] + ys = np.arange(len(names)) + cols = [TEAL if v >= 0 else RED for v in vals] + bars = ax.barh(ys, vals, color=cols, edgecolor="#333333", linewidth=0.6, zorder=3) + ax.axvline(0.0, color="#444444", lw=1.1, zorder=2) + for y, v in zip(ys, vals): + off = amax * 0.03 + ax.annotate(f"{v:+.4f}", (v + (off if v >= 0 else -off), y), + ha="left" if v >= 0 else "right", va="center", + fontsize=9.2, weight="bold", + color="#1B7F3B" if v >= 0 else "#8E1F14") + ax.set_yticks(ys) + ax.set_yticklabels([f"{n}\ntest n\u2248{s:,}" for n, s in zip(names, sups)], + fontsize=10) + ax.set_xlim(-xlim, xlim) + ax.set_xlabel("\u0394 F1 (proposed \u2212 ORIGINAL PROTEAN), test") + beta = SELECTED[dataset][alpha] + ax.set_title(f"{DATASET_NAME[dataset]} \u03b1={alpha:.2f} (beta={beta:g})") + fig.suptitle(f"Per-class test F1 change vs ORIGINAL PROTEAN \u2014 {DATASET_NAME[dataset]}", + fontsize=13, weight="bold", y=1.04) + fig.text(0.5, -0.02, + "Bars are mean \u0394F1 over seeds {0,1,2}; both panels use the same x-scale so bar lengths are comparable. " + "Teal = F1 increased, red = F1 decreased and the exact signed \u0394F1 is printed at each bar. Delta is F1, NOT precision or recall. " + "Only beta>0 conditions are shown (\u03b1=0.25 selected beta=0). Test support is the mean over seeds. No significance test (3 seeds).", + ha="center", va="top", fontsize=8.6, color="#444444", linespacing=1.5) + _save(fig, f"figJ{'1' if dataset == 'xiiotid' else '2'}_{dataset}_per_class_original_vs_proposed") + + +# --------------------------------------------------------------------------- +# Figure K (primary): ranked error changes for 5G-NIDD alpha=0.75 seed=0 +# --------------------------------------------------------------------------- +def figure_k_ranked(dataset="5gnidd", alpha=0.75, seed=0): + rows = [r for r in _read(ANALYSIS / "deepcmp__confusion_delta.csv") + if r["dataset"] == dataset and abs(float(r["alpha"]) - alpha) < 1e-9 + and int(r["seed"]) == seed] + off = [r for r in rows if r["true_class"] != r["pred_class"]] + reduced = sorted(off, key=lambda r: int(r["delta"]))[:5] + increased = sorted(off, key=lambda r: -int(r["delta"]))[:5] + + fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(11.5, 7.0), + constrained_layout=True, sharex=True) + for ax, grp, colour, sign_label in ( + (ax1, increased, RED, "more errors after weighting"), + (ax2, reduced, TEAL, "fewer errors after weighting")): + ax.set_axisbelow(True) + ax.grid(axis="x") + order = sorted(grp, key=lambda r: -abs(int(r["delta"]))) + labels = [f"{r['true_class']} \u2192 {r['pred_class']}" for r in order] + deltas = [int(r["delta"]) for r in order] + ys = np.arange(len(order)) + ax.barh(ys, deltas, color=colour, edgecolor="#333333", linewidth=0.6, zorder=3) + ax.axvline(0.0, color="#444444", lw=1.1, zorder=2) + for y, r in zip(ys, order): + d = int(r["delta"]) + ax.annotate(f"{d:+d} ({r['orig_count']} \u2192 {r['prop_count']})", + (d, y), textcoords="offset points", + xytext=(6 if d >= 0 else -6, 0), va="center", + ha="left" if d >= 0 else "right", fontsize=9, + weight="bold", color="#8E1F14" if d > 0 else "#0E5C5A") + ax.set_yticks(ys) + ax.set_yticklabels(labels, fontsize=10) + ax.set_title(("\u25b2 More errors after weighting" if colour == RED + else "\u25bc Fewer errors after weighting"), fontsize=10.5) + all_show = [int(r["delta"]) for r in increased] + [int(r["delta"]) for r in reduced] + ax1.set_xlim(min(all_show) * 1.5, max(all_show) * 1.7) + ax2.set_xlabel("change in test sample count (proposed \u2212 original; " + "negative = fewer, positive = more)") + dname = DATASET_NAME[dataset] + fig.suptitle(f"Ranked change in misclassifications vs ORIGINAL PROTEAN \u2014 " + f"{dname}, \u03b1={alpha:g}, seed={seed}, test", fontsize=12.5, weight="bold", y=1.04) - fig.text(0.5, -0.015, - "Only beta>0 conditions are shown; \u03b1=0.25 selected beta=0 so it carries no weighting effect. " - "Class support n\u2248 is the mean test support over seeds. Teal = F1 increased, red = F1 decreased. " - "X-IIoTID and 5G-NIDD class labels are never mixed. No significance testing (3 seeds).", - ha="center", va="top", fontsize=8.4, color="#444444", linespacing=1.5) - _save(fig, "figJ_per_class_original_vs_proposed") + fig.text(0.5, -0.02, + "Each bar is one off-diagonal (true \u2192 predicted) confusion cell; raw test counts over the same 243,177 samples. " + "Below zero = fewer such errors under the proposed selected-beta run; above zero = more. " + "The dominant error Benign \u2192 UDPFlood is ~unchanged (56,899 \u2192 57,054, +155 more) and remains the largest error. " + "Full difference matrices are in the supplemental figure. No statistical significance is inferred.", + ha="center", va="top", fontsize=8.6, color="#444444", linespacing=1.5) + _save(fig, f"figK_confusion_error_changes_{dataset}_a{alpha:g}_seed{seed}") -# --------------------------------------------------------------------------- -def figure_k(dataset="5gnidd", alpha=0.75, seed=0, beta=0.5): +def figure_k_full(dataset="5gnidd", alpha=0.75, seed=0, beta=0.5): + """Supplemental full difference matrices (raw counts and row-rate).""" odir = RESULTS / f"{dataset}__alpha{alpha:g}__seed{seed}__original_protean" sdir = RESULTS / f"{dataset}__alpha{alpha:g}__seed{seed}__support_weighted__beta{beta:g}" names = json.loads((odir / "metadata.json").read_text())["class_names"] @@ -187,42 +265,40 @@ def figure_k(dataset="5gnidd", alpha=0.75, seed=0, beta=0.5): for ax in axes: ax.set_xticks(range(len(names))) ax.set_yticks(range(len(names))) - ax.set_xticklabels(names, rotation=45, ha="right", fontsize=8) - ax.set_yticklabels(names, fontsize=8) + ax.set_xticklabels(names, rotation=45, ha="right", fontsize=9) + ax.set_yticklabels(names, fontsize=9) ax.set_xlabel("predicted class") axes[0].set_ylabel("true class") - axes[0].set_title("A. Difference in raw counts\n(proposed \u2212 original)") - axes[1].set_title("B. Difference in row-normalized rate\n(percentage points, proposed \u2212 original)") - # annotate the strongest cells on each panel + axes[0].set_title("A. Raw count difference\n(proposed \u2212 original)") + axes[1].set_title("B. Row-normalized rate difference\n(percentage points)") for ax, M, is_rate in ((axes[0], diff, False), (axes[1], diff_rate * 100, True)): - flat = [(abs(M[i, j]), i, j) for i in range(9) for j in range(9) if i != j] + flat = [(abs(M[i, j]), i, j) for i in range(len(names)) for j in range(len(names)) if i != j] for _, i, j in sorted(flat, reverse=True)[:6]: v = M[i, j] ax.text(j, i, f"{v:+.0f}" if not is_rate else f"{v:+.1f}", - ha="center", va="center", fontsize=6.4, fontweight="bold", - color="white" if abs(v) > (vmax_c if not is_rate else vmax_r) * 0.5 - else "#222222") - cb0 = fig.colorbar(im0, ax=axes[0], fraction=0.046, pad=0.02) - cb0.set_label("count delta") - cb1 = fig.colorbar(im1, ax=axes[1], fraction=0.046, pad=0.02) - cb1.set_label("row-rate delta (pp)") - dname = DATASET_NAME[dataset] - fig.suptitle(f"Confusion-matrix change: proposed (beta={beta:g}) minus original PROTEAN - " - f"{dname}, alpha={alpha:g}, seed={seed}, test", + ha="center", va="center", fontsize=6.6, fontweight="bold", + color="white" if abs(v) > (vmax_c if not is_rate else vmax_r) * 0.5 else "#222222") + fig.colorbar(im0, ax=axes[0], fraction=0.046, pad=0.02).set_label("count delta") + fig.colorbar(im1, ax=axes[1], fraction=0.046, pad=0.02).set_label("row-rate delta (pp)") + fig.suptitle(f"Supplemental: full confusion-matrix change (proposed \u2212 original) \u2014 " + f"{DATASET_NAME[dataset]}, \u03b1={alpha:g}, seed={seed}, test", fontsize=12, weight="bold", y=1.05) fig.text(0.5, -0.02, - "Rows = true class, columns = predicted class; identical class order and sample counts (243,177) for both runs. " - "Diagonal: positive = more correct predictions; off-diagonal: positive = more errors of that (true\u2192predicted) type. " - "Red = positive delta (proposed has MORE samples in the cell); blue = negative delta (fewer). Values are descriptive only; no significance is inferred.", - ha="center", va="top", fontsize=8.4, color="#444444", linespacing=1.5) + "Rows = true class, columns = predicted class; identical class order and 243,177 samples. " + "Diagonal: positive = more correct predictions; off-diagonal: positive = more errors of that type. " + "Red = positive delta, blue = negative. See the ranked figure for the practical reading.", + ha="center", va="top", fontsize=8.6, color="#444444", linespacing=1.5) _save(fig, f"figK_confusion_delta_{dataset}_a{alpha:g}_seed{seed}") def main(): _style() - figure_i() - figure_j() - figure_k() + figure_i("xiiotid") + figure_i("5gnidd") + figure_j("xiiotid") + figure_j("5gnidd") + figure_k_ranked() + figure_k_full() if __name__ == "__main__": diff --git a/scripts/make_final_figures.py b/scripts/make_final_figures.py index 9008406..1aaca7c 100644 --- a/scripts/make_final_figures.py +++ b/scripts/make_final_figures.py @@ -327,43 +327,69 @@ def figure_f() -> None: sel_e = np.array([float(master[c]["selected_rare_macro_f1_std"]) for c in conditions]) betas = [float(master[c]["selected_beta"]) for c in conditions] delta = sel - orig - - fig, ax = plt.subplots(figsize=(11.4, 6.0), constrained_layout=True) - ax.set_axisbelow(True) - ax.grid(axis="y") x = np.arange(len(conditions)) - w = 0.36 - ax.bar(x - w / 2, orig, w, yerr=orig_e, capsize=3.5, color=CHARCOAL, - edgecolor="#1A1A1A", linewidth=0.7, label="original_protean", zorder=3) - ax.bar(x + w / 2, sel, w, yerr=sel_e, capsize=3.5, color=TEAL, - edgecolor="#0A4F4E", linewidth=0.7, hatch="///", - label="support_weighted (validation-selected beta)", zorder=3) + w = 0.38 - for xi, db, sb in zip(x, delta, betas): - if sb == 0.0: - colour = GREY - elif db > 0: - colour = GREEN - else: - colour = RED - sign = "+" if db >= 0 else "\u2212" + fig, (axA, axB) = plt.subplots(2, 1, figsize=(11.2, 8.6), + constrained_layout=True, + gridspec_kw={"height_ratios": [1.15, 1.0]}) + + # --- Panel A: absolute levels --------------------------------------- + axA.set_axisbelow(True) + axA.grid(axis="y") + axA.bar(x - w / 2, orig, w, yerr=orig_e, capsize=3.5, color=CHARCOAL, + edgecolor="#1A1A1A", linewidth=0.7, label="original_protean", zorder=3) + axA.bar(x + w / 2, sel, w, yerr=sel_e, capsize=3.5, color=TEAL, + edgecolor="#0A4F4E", linewidth=0.7, hatch="///", + label="proposed: support_weighted (selected beta)", zorder=3) + for xi, sb in zip(x, betas): top = max(orig[xi] + orig_e[xi], sel[xi] + sel_e[xi]) - ax.annotate(f"{sign}{abs(db):.4f}", (xi, top + 0.012), ha="center", - fontsize=9.5, weight="bold", color=colour) - ax.annotate(f"beta={sb:g}", (xi, top + 0.030), ha="center", fontsize=8.5, - color="#444444") - ax.set_xticks(x) - ax.set_xticklabels(labels) - ax.set_ylabel("test rare-class macro-F1\n(mean over seeds {0,1,2})") - ax.set_ylim(0.5, max(max(orig + orig_e), max(sel + sel_e)) + 0.10) - ax.set_title("Test rare-class macro-F1: original PROTEAN vs validation-selected beta") - ax.legend(loc="upper left", frameon=True, edgecolor="#CCCCCC") + axA.annotate(f"beta={sb:g}", (xi, top + 0.006), ha="center", + fontsize=9, color="#444444") + axA.set_xticks(x) + axA.set_xticklabels([]) + axA.set_ylabel("test rare-class macro-F1\n(mean over seeds {0,1,2})") + axA.set_ylim(0.5, max(max(orig + orig_e), max(sel + sel_e)) + 0.075) + axA.set_title("A. Absolute test rare-class macro-F1") + axA.legend(loc="upper left", frameon=True, edgecolor="#CCCCCC") + + # --- Panel B: signed difference ------------------------------------- + axB.set_axisbelow(True) + axB.grid(axis="y") + cols = [GREY if sb == 0 else (GREEN if d > 0 else RED) + for d, sb in zip(delta, betas)] + axB.bar(x, delta, 0.55, color=cols, edgecolor="#333333", linewidth=0.6, zorder=3) + axB.axhline(0.0, color="#444444", lw=1.1, zorder=2) + for xi, d, sb in zip(x, delta, betas): + sign = "\u2212" if d < 0 else "+" + colr = "#1B7F3B" if (sb > 0 and d > 0) else ("#8E1F14" if (sb > 0 and d < 0) else "#555555") + axB.annotate(f"{sign}{abs(d):.4f}", (xi, d), textcoords="offset points", + xytext=(0, 6 if d >= 0 else -14), ha="center", fontsize=10.5, + weight="bold", color=colr) + axB.set_xticks(x) + axB.set_xticklabels(labels) + axB.set_ylabel("\u0394 rare-class macro-F1\n(proposed \u2212 original)") + m = max(abs(delta)) * 1.65 + axB.set_ylim(-m, m) + axB.set_title("B. Signed difference (proposed \u2212 original)") + handles = [plt.Rectangle((0, 0), 1, 1, facecolor=GREEN, edgecolor="#333"), + plt.Rectangle((0, 0), 1, 1, facecolor=RED, edgecolor="#333"), + plt.Rectangle((0, 0), 1, 1, facecolor=GREY, edgecolor="#333")] + axB.legend(handles, ["positive (3 conditions)", "negative (1 condition)", + "selected beta = 0 / ~0 (2 conditions)"], + title="signed delta", loc="upper right", frameon=True, + edgecolor="#CCCCCC", ncols=1) + + fig.suptitle("Original PROTEAN vs proposed selected-beta: test rare-class macro-F1", + fontsize=13, weight="bold", y=1.02) _footnote(fig, - "Bar height = mean test rare-class macro-F1 over seeds {0,1,2}; error bars = \u00b11 sample std across seeds.\n" - "Delta (top) = selected-beta minus original_protean; green = positive, red = negative, grey = selected beta 0\n" - "(floating-point ~0). No statistical significance test is applied (3 seeds). Gains are condition-dependent;\n" + "Panel A: bar height = mean test rare-class macro-F1 over seeds {0,1,2}; error bars = \u00b11 sample std across seeds " + "(the levels are close by design, so panel B is the informative view).\n" + "Panel B: delta = proposed \u2212 original; the exact value is printed at each bar. Green = positive (X-IIoTID a0.50, " + "X-IIoTID a0.75, 5G-NIDD a0.75); red = negative (5G-NIDD a0.50); grey = selected beta 0 (both a0.25, floating-point ~0).\n" + "No statistical significance test is applied (3 seeds). A positive mean delta does not establish significance; " "the proposed weighting does NOT consistently outperform original PROTEAN.", - y=-0.13) + y=-0.10) _save(fig, "figF_selected_vs_original_test_rare_macro_f1") @@ -458,15 +484,34 @@ def figure_h(dataset="5gnidd", alpha=0.75, seed=0, selected_beta=0.5) -> bool: class_names = meta["class_names"] mats = {k: np.array(json.loads(p.read_text())) for k, p in paths.items()} + o, s = mats["original_protean"], mats["selected_beta"] + diff = s - o + idx = {n: i for i, n in enumerate(class_names)} + + # Computed summary of the most important changes (never hard-coded). + off = [(int(diff[i, j]), i, j) for i in range(len(class_names)) + for j in range(len(class_names)) if i != j] + reduced = sorted(off, key=lambda t: t[0])[:2] + increased = sorted(off, key=lambda t: -t[0])[:2] + diag = sorted(range(len(class_names)), key=lambda i: -abs(int(diff[i, i])))[:3] + lines = ["Most important ranked changes (proposed \u2212 original):"] + for d, i, j in reduced: + lines.append(f" \u2022 {class_names[i]} \u2192 {class_names[j]}: " + f"{int(o[i, j])} \u2192 {int(s[i, j])} ({d} fewer)") + for d, i, j in increased: + note = " \u2190 dominant residual error; NOT an improvement" \ + if (class_names[i], class_names[j]) == ("Benign", "UDPFlood") else "" + lines.append(f" \u2022 {class_names[i]} \u2192 {class_names[j]}: " + f"{int(o[i, j])} \u2192 {int(s[i, j])} (+{d} more){note}") + lines.append(" \u2022 diagonal (correct): " + ", ".join( + f"{class_names[i]} {int(diff[i, i]):+d}" for i in diag)) from matplotlib.colors import LogNorm vmax = max(m.max() for m in mats.values()) vmin = 1 - fig, axes = plt.subplots(1, 2, figsize=(13.2, 6.2), constrained_layout=True) - titles = { - "original_protean": "ORIGINAL PROTEAN", - "selected_beta": f"SELECTED-BETA PROTEAN (beta={selected_beta:g})", - } + fig, axes = plt.subplots(1, 2, figsize=(13.6, 6.0), constrained_layout=True) + titles = {"original_protean": "LEFT: ORIGINAL PROTEAN", + "selected_beta": f"RIGHT: BETA-TUNED PROTEAN (beta={selected_beta:g})"} im = None for ax, key in zip(axes, ["original_protean", "selected_beta"]): m = mats[key] @@ -474,8 +519,8 @@ def figure_h(dataset="5gnidd", alpha=0.75, seed=0, selected_beta=0.5) -> bool: aspect="equal") ax.set_xticks(range(len(class_names))) ax.set_yticks(range(len(class_names))) - ax.set_xticklabels(class_names, rotation=45, ha="right", fontsize=8.2) - ax.set_yticklabels(class_names, fontsize=8.2) + ax.set_xticklabels(class_names, rotation=45, ha="right", fontsize=9) + ax.set_yticklabels(class_names, fontsize=9) ax.set_xlabel("predicted class") ax.set_title(titles[key], fontsize=11) thr = m.max() * 0.06 @@ -485,7 +530,7 @@ def figure_h(dataset="5gnidd", alpha=0.75, seed=0, selected_beta=0.5) -> bool: if v == 0: continue txt = f"{v/1000:.1f}k" if v >= 10000 else str(v) - ax.text(j, i, txt, ha="center", va="center", fontsize=6.4, + ax.text(j, i, txt, ha="center", va="center", fontsize=6.8, color="white" if m[i, j] > thr else "#333333") axes[0].set_ylabel("true class") cbar = fig.colorbar(im, ax=axes, fraction=0.025, pad=0.02) @@ -494,11 +539,13 @@ def figure_h(dataset="5gnidd", alpha=0.75, seed=0, selected_beta=0.5) -> bool: fig.suptitle( f"Paired test confusion matrices - {dname}, alpha={alpha:g}, seed={seed} " f"(same manifest, split, seed, class order)", - fontsize=11.5, weight="bold", y=1.05) + fontsize=12, weight="bold", y=1.05) _footnote(fig, - "Left = original_protean; right = support_weighted at the validation-selected beta. Raw counts; identical class order and colour scale.\n" - "Dominant off-diagonal error: Benign predicted as UDPFlood, essentially unchanged by beta. No statistical significance is inferred.", - y=-0.20) + "Rows = actual class, columns = predicted class; diagonal = correct, off-diagonal = misclassified. Raw counts, " + "identical class order and colour scale; both matrices contain the same 243,177 test samples.\n" + + "\n".join(lines) + + "\nNo statistical significance is inferred from the visual difference.", + y=-0.13) _save(fig, f"figH_confusion_matrix_{dataset}_a{alpha:g}_seed{seed}") return True