diff --git a/README.md b/README.md index 0c33191..8bd3334 100644 --- a/README.md +++ b/README.md @@ -472,47 +472,117 @@ Representative confusion-matrix files: ## 16. Final Figures and Visualizations -All figures are under `artifacts/figures/final/` and load from the `final__*` analysis tables. They -were generated with matplotlib; no training was involved. +All figures live in `artifacts/figures/final/` and are regenerated from the audited +`artifacts/analysis/final__*` tables by `scripts/make_final_figures.py` (no training; every number is +loaded from an artifact). Each figure is saved as a high-resolution PNG (220 dpi) and a vector PDF. -**Fig A — pooled validation rare-class macro-F1 vs beta** (per dataset). Axes: `beta` (x) vs pooled -validation per-client-mean rare-class macro-F1 (y); one line per alpha. Evidence: the validation -curve is non-monotone and the selected `beta` differs by condition. +**Naming and legend conventions used throughout.** + +- **Charcoal (`original_protean`)**: the frozen reproduced baseline. +- **Teal (`support_weighted`, validation-selected beta)**: the proposed method at the beta selected by + pooled validation for that condition. +- **Gray (`beta = 0` control)**: the equal-weight `support_weighted` run; a control, *not* + automatically the same recorded configuration as `original_protean`. +- Alpha conditions are distinguished by colour **and** marker **and** line style, so colour is never + the only cue. + +**Comparison types (so figures are not misread).** + +| Figure | Comparison type | Baseline actually plotted | +|---|---|---| +| A (both) | Beta candidate comparison within `support_weighted` | none (support_weighted candidates only) | +| B | Beta selection surface | none (selection, not performance) | +| C | Dataset heterogeneity | none (data distribution) | +| D, E | Fixed-snapshot comparison vs `beta = 0` | `beta = 0` control | +| F | Direct comparison vs `original_protean` | `original_protean` | +| G | Per-client comparison vs `beta = 0` | `beta = 0` control | +| H | Paired `original_protean` vs selected beta | `original_protean` | + +### Fig A1 / A2 — Validation-based beta selection (support_weighted) + +**Compared:** the five `support_weighted` beta candidates, separately for each alpha. **Level/split:** +pooled validation, per-client mean across seeds `{0,1,2}`. **Metric:** rare-class macro-F1 +(per-client mean). **Observation:** the validation response is non-monotone and the selected beta +differs by condition. **Limits:** this is a selection figure; no `original_protean` run is plotted, +and the error bars are ±1 sample std across three project-defined seeds, not a confidence interval. ![Pooled validation beta curve, 5G-NIDD](artifacts/figures/final/figA_5gnidd_pooled_validation_beta_curve.png) ![Pooled validation beta curve, X-IIoTID](artifacts/figures/final/figA_xiiotid_pooled_validation_beta_curve.png) -**Fig B — selected beta by condition.** Bar chart of the six validation-selected betas; shows that -selections take `{0.00, 0.50, 1.00}` and are not universal. +### Fig B — Validation-selected beta by dataset and alpha + +**Compared:** none (selection only). **Level:** pooled validation. **Metric:** selected beta under +the selection rule. **Observation:** selections take `{0.00, 0.50, 1.00}` and are not universal. +**Limits:** selection surface, not a performance comparison; a higher beta is not automatically +better. No `original_protean` comparison is implied. ![Selected beta by condition](artifacts/figures/final/figB_selected_beta_by_condition.png) -**Fig C — realized support CV.** Condition-level mean class-support CV; shows realized imbalance -decreases as alpha rises. +### Fig C — Realized class-support imbalance + +**Compared:** none (data distribution). **Level:** training partitions. **Metric:** mean class-support +CV of `N_ij` across contributing clients (std/mean), averaged over classes and seeds. **Observation:** +realized imbalance decreases as the configured alpha rises. **Limits:** this describes the data, +not the relative performance of original vs modified PROTEAN; Dirichlet alpha is the configuration +knob and is not the same as the realized imbalance. ![Realized support CV](artifacts/figures/final/figC_realized_support_cv.png) -**Fig D — prototype displacement at beta=0 vs beta=1.** Fixed-snapshot mean displacement per class; -shows displacement grows with `beta` and larger where support imbalance is larger. +### Fig D — Support-weighted aggregation: beta=1 vs beta=0 control + +**Compared:** `beta = 1` against the `beta = 0` control (not `original_protean`). **Level:** +fixed round-0 snapshots, same local prototypes and `N_ij`, no retraining. **Metric:** mean +`‖Cbar_j(1) − Cbar_j(0)‖` across classes. **Observation:** displacement increases with beta and is +largest where realized imbalance is largest. **Limits:** displacement is a representation metric, +not detection quality. ![Prototype displacement beta0 vs beta1](artifacts/figures/final/figD_prototype_displacement_beta0_vs_beta1.png) -**Fig E — support CV vs displacement.** Scatter of realized support CV (x) vs beta=1 displacement -(y) across conditions; a descriptive co-variation, not a causal relation. +### Fig E — Realized support CV vs prototype displacement + +**Compared:** `beta = 1` displacement against the `beta = 0` control, across conditions. **Level:** +fixed snapshots. **Metric:** realized support CV (x) vs `beta = 1` displacement (y). **Observation:** +larger realized imbalance accompanies larger displacement. **Limits:** descriptive association only; +the imbalance-displacement link is mechanical under the beta formula and is not causal evidence. ![Support CV vs displacement](artifacts/figures/final/figE_support_cv_vs_displacement.png) -**Fig F — selected vs original test rare-class macro-F1.** Paired comparison per condition; shows -small, mixed-sign differences. +### Fig F — Selected-beta PROTEAN vs ORIGINAL PROTEAN (test) + +**Compared:** `original_protean` (charcoal) against `support_weighted` at the validation-selected +beta (teal), per condition. **Level/split:** test, mean over seeds `{0,1,2}`. **Metric:** test +rare-class macro-F1; error bars = ±1 sample std across seeds. **Observation:** signed deltas are +`+0.0230`, `+0.0014`, `+0.0168` (positive), `−0.0086` (negative), and `−0.0007`/`−0.0010` at the two +`beta = 0` conditions (floating-point ~0). **Limits:** no statistical significance test (three +seeds); the largest gain is within seed noise; the weighting does not consistently outperform +`original_protean`. ![Selected vs original test rare macro-F1](artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.png) -**Fig G — per-client effect at the selected beta.** Per-client validation delta relative to -`beta = 0`; shows within-condition heterogeneity across clients. +### Fig G — Per-client effect of selected beta relative to the beta=0 control + +**Compared:** selected-beta `support_weighted` against the `beta = 0` control (not +`original_protean`). **Level/split:** validation, per client, per seed. **Metric:** per-client +rare-class macro-F1 delta; a client is "improved"/"degraded" if its delta is above/below 0, +"unchanged" otherwise. **Observation:** within a condition some clients improve and others degrade; +at both `alpha = 0.25` conditions the selected beta is 0, so all clients are unchanged. **Limits:** +client ids are per-seed realizations and are not pooled. ![Client effect at selected beta](artifacts/figures/final/figG_client_effect_at_selected_beta.png) +### Fig H (supplemental) — Paired test confusion matrices + +**Compared:** `original_protean` (left) against selected-beta `support_weighted` (right) for a valid +paired condition (5G-NIDD, `alpha = 0.75`, `seed = 0`). **Level/split:** test; global counts over the +evaluation split. **Metric:** raw 9×9 counts (log colour scale), identical class order and colour +scale. **Observation:** the dominant off-diagonal error is `Benign → UDPFlood` (~56.9k vs ~57.1k) and +is essentially unchanged by beta. **Limits:** no statistical significance is inferred from the visual +difference; the matrices come from the same manifest, split, seed, and class order. + +![Paired confusion matrices](artifacts/figures/final/figH_confusion_matrix_5gnidd_a0.75_seed0.png) + + ## 17. Main Scientific Findings 1. **Changing `beta` changes the global prototype.** Contribution weights become proportional to @@ -650,7 +720,8 @@ Analysis outputs (`artifacts/analysis/`), by function: | Prototype snapshots & weights | `*__prototype_snapshot.json`, `*__prototype_weights.csv`, `*__prototype_effect*.csv` | Partition manifests: `artifacts/partitions/__alpha__seed.json` (18 files). -Figures: `artifacts/figures/final/figA–figG*.png` (8 files). +Figures: `artifacts/figures/final/` — 9 figures (figA–figH) as PNG (220 dpi) plus matching vector PDF; +regenerated reproducibly by `scripts/make_final_figures.py`. Key documents: diff --git a/artifacts/figures/final/figA_5gnidd_pooled_validation_beta_curve.pdf b/artifacts/figures/final/figA_5gnidd_pooled_validation_beta_curve.pdf new file mode 100644 index 0000000..a9cd36d Binary files /dev/null and b/artifacts/figures/final/figA_5gnidd_pooled_validation_beta_curve.pdf differ diff --git a/artifacts/figures/final/figA_5gnidd_pooled_validation_beta_curve.png b/artifacts/figures/final/figA_5gnidd_pooled_validation_beta_curve.png index c4e4b9a..883280f 100644 Binary files a/artifacts/figures/final/figA_5gnidd_pooled_validation_beta_curve.png and b/artifacts/figures/final/figA_5gnidd_pooled_validation_beta_curve.png differ diff --git a/artifacts/figures/final/figA_xiiotid_pooled_validation_beta_curve.pdf b/artifacts/figures/final/figA_xiiotid_pooled_validation_beta_curve.pdf new file mode 100644 index 0000000..e61df77 Binary files /dev/null and b/artifacts/figures/final/figA_xiiotid_pooled_validation_beta_curve.pdf differ diff --git a/artifacts/figures/final/figA_xiiotid_pooled_validation_beta_curve.png b/artifacts/figures/final/figA_xiiotid_pooled_validation_beta_curve.png index 482f179..636c11c 100644 Binary files a/artifacts/figures/final/figA_xiiotid_pooled_validation_beta_curve.png and b/artifacts/figures/final/figA_xiiotid_pooled_validation_beta_curve.png differ diff --git a/artifacts/figures/final/figB_selected_beta_by_condition.pdf b/artifacts/figures/final/figB_selected_beta_by_condition.pdf new file mode 100644 index 0000000..04ae715 Binary files /dev/null and b/artifacts/figures/final/figB_selected_beta_by_condition.pdf differ diff --git a/artifacts/figures/final/figB_selected_beta_by_condition.png b/artifacts/figures/final/figB_selected_beta_by_condition.png index 7da31c5..b04d86d 100644 Binary files a/artifacts/figures/final/figB_selected_beta_by_condition.png and b/artifacts/figures/final/figB_selected_beta_by_condition.png differ diff --git a/artifacts/figures/final/figC_realized_support_cv.pdf b/artifacts/figures/final/figC_realized_support_cv.pdf new file mode 100644 index 0000000..54ce01d Binary files /dev/null and b/artifacts/figures/final/figC_realized_support_cv.pdf differ diff --git a/artifacts/figures/final/figC_realized_support_cv.png b/artifacts/figures/final/figC_realized_support_cv.png index 544bbab..e178b15 100644 Binary files a/artifacts/figures/final/figC_realized_support_cv.png and b/artifacts/figures/final/figC_realized_support_cv.png differ diff --git a/artifacts/figures/final/figD_prototype_displacement_beta0_vs_beta1.pdf b/artifacts/figures/final/figD_prototype_displacement_beta0_vs_beta1.pdf new file mode 100644 index 0000000..e948c74 Binary files /dev/null and b/artifacts/figures/final/figD_prototype_displacement_beta0_vs_beta1.pdf differ diff --git a/artifacts/figures/final/figD_prototype_displacement_beta0_vs_beta1.png b/artifacts/figures/final/figD_prototype_displacement_beta0_vs_beta1.png index 9f8cfc8..5fd53da 100644 Binary files a/artifacts/figures/final/figD_prototype_displacement_beta0_vs_beta1.png and b/artifacts/figures/final/figD_prototype_displacement_beta0_vs_beta1.png differ diff --git a/artifacts/figures/final/figE_support_cv_vs_displacement.pdf b/artifacts/figures/final/figE_support_cv_vs_displacement.pdf new file mode 100644 index 0000000..fa953e8 Binary files /dev/null and b/artifacts/figures/final/figE_support_cv_vs_displacement.pdf differ diff --git a/artifacts/figures/final/figE_support_cv_vs_displacement.png b/artifacts/figures/final/figE_support_cv_vs_displacement.png index eb14dab..a1006ba 100644 Binary files a/artifacts/figures/final/figE_support_cv_vs_displacement.png and b/artifacts/figures/final/figE_support_cv_vs_displacement.png differ diff --git a/artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.pdf b/artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.pdf new file mode 100644 index 0000000..8962651 Binary files /dev/null and b/artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.pdf differ diff --git a/artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.png b/artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.png index 405a787..c994564 100644 Binary files a/artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.png and b/artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.png differ diff --git a/artifacts/figures/final/figG_client_effect_at_selected_beta.pdf b/artifacts/figures/final/figG_client_effect_at_selected_beta.pdf new file mode 100644 index 0000000..7ed31a3 Binary files /dev/null and b/artifacts/figures/final/figG_client_effect_at_selected_beta.pdf differ diff --git a/artifacts/figures/final/figG_client_effect_at_selected_beta.png b/artifacts/figures/final/figG_client_effect_at_selected_beta.png index 324282b..a78a2ad 100644 Binary files a/artifacts/figures/final/figG_client_effect_at_selected_beta.png and b/artifacts/figures/final/figG_client_effect_at_selected_beta.png differ diff --git a/artifacts/figures/final/figH_confusion_matrix_5gnidd_a0.75_seed0.pdf b/artifacts/figures/final/figH_confusion_matrix_5gnidd_a0.75_seed0.pdf new file mode 100644 index 0000000..0527396 Binary files /dev/null and b/artifacts/figures/final/figH_confusion_matrix_5gnidd_a0.75_seed0.pdf differ diff --git a/artifacts/figures/final/figH_confusion_matrix_5gnidd_a0.75_seed0.png b/artifacts/figures/final/figH_confusion_matrix_5gnidd_a0.75_seed0.png new file mode 100644 index 0000000..a65f9ba Binary files /dev/null and b/artifacts/figures/final/figH_confusion_matrix_5gnidd_a0.75_seed0.png differ diff --git a/scripts/make_final_figures.py b/scripts/make_final_figures.py new file mode 100644 index 0000000..9008406 --- /dev/null +++ b/scripts/make_final_figures.py @@ -0,0 +1,520 @@ +#!/usr/bin/env python3 +"""Regenerate the final PROTEAN research figures. + +Reads only audited machine-readable artifacts +(``artifacts/analysis/final__*.csv`` and per-run +``results/*/test_confusion_matrix.json``) and writes publication-quality +PNG (>=200 dpi) and PDF figures to ``artifacts/figures/final``. + +This script does not run training, modify results, or alter any +experimental value. Every plotted number is loaded from an authoritative +artifact; see the ``SOURCE`` comment in each function. + +Run: + .venv/bin/python scripts/make_final_figures.py +""" + +from __future__ import annotations + +import csv +import json +from pathlib import Path + +import matplotlib + +matplotlib.use("Agg") +import matplotlib.pyplot as plt +import numpy as np +from matplotlib.lines import Line2D + +ROOT = Path(__file__).resolve().parents[1] +ANALYSIS = ROOT / "artifacts" / "analysis" +RESULTS = ROOT / "results" +OUT = ROOT / "artifacts" / "figures" / "final" + +# --------------------------------------------------------------------------- +# Consistent visual language +# --------------------------------------------------------------------------- +CHARCOAL = "#3A3A3A" # original_protean +TEAL = "#0E7C7B" # support_weighted, validation-selected beta +CONTROL = "#9A9A9A" # beta=0 control reference +ALPHA_STYLE = { + 0.25: dict(color="#0072B2", marker="o", ls="-", label="alpha = 0.25"), + 0.50: dict(color="#D55E00", marker="s", ls="--", label="alpha = 0.50"), + 0.75: dict(color="#009E73", marker="^", ls="-.", label="alpha = 0.75"), +} +DATASET_STYLE = { + "xiiotid": dict(color="#0072B2", marker="o", label="X-IIoTID"), + "5gnidd": dict(color="#D55E00", marker="s", label="5G-NIDD"), +} +DATASET_NAME = {"xiiotid": "X-IIoTID", "5gnidd": "5G-NIDD"} +DATASETS = ["xiiotid", "5gnidd"] +ALPHAS = [0.25, 0.50, 0.75] +BETAS = [0.0, 0.25, 0.5, 0.75, 1.0] +SELECTED = {("xiiotid", 0.25): 0.0, ("xiiotid", 0.5): 0.75, ("xiiotid", 0.75): 0.5, + ("5gnidd", 0.25): 0.0, ("5gnidd", 0.5): 1.0, ("5gnidd", 0.75): 0.5} +GREEN, RED, GREY = "#1B7F3B", "#B22222", "#6B6B6B" + + +def _apply_style() -> None: + plt.rcParams.update({ + "figure.facecolor": "white", + "axes.facecolor": "white", + "axes.edgecolor": "#5A5A5A", + "axes.linewidth": 0.9, + "axes.titlesize": 13, + "axes.titleweight": "bold", + "axes.labelsize": 11, + "xtick.labelsize": 10, + "ytick.labelsize": 10, + "legend.fontsize": 9.5, + "font.size": 10.5, + "axes.spines.top": False, + "axes.spines.right": False, + "grid.color": "#B0B0B0", + "grid.alpha": 0.35, + "grid.linestyle": ":", + "savefig.dpi": 220, + "savefig.bbox": "tight", + }) + + +def _save(fig, name: str) -> None: + OUT.mkdir(parents=True, exist_ok=True) + for ext in ("png", "pdf"): + fig.savefig(OUT / f"{name}.{ext}") + plt.close(fig) + print("wrote", OUT / f"{name}.png") + + +def _read_csv(path: Path) -> list[dict]: + with path.open() as fh: + return list(csv.DictReader(fh)) + + +def _footnote(fig, text: str, y: float) -> None: + """Place a wrapped caption below the axes; bbox_inches='tight' expands.""" + fig.text(0.5, y, text, ha="center", va="top", fontsize=8.7, + color="#444444", linespacing=1.5) + + +# --------------------------------------------------------------------------- +# Figure A: pooled validation beta curves (support_weighted only) +# --------------------------------------------------------------------------- +def figure_a(dataset: str) -> None: + # SOURCE: artifacts/analysis/final__pooled_validation_curves.csv + rows = _read_csv(ANALYSIS / "final__pooled_validation_curves.csv") + fig, ax = plt.subplots(figsize=(8.4, 5.4), constrained_layout=True) + ax.set_axisbelow(True) + ax.grid(axis="y") + ymin, ymax = 1e9, -1e9 + + for alpha in ALPHAS: + vals = {float(r["beta"]): r for r in rows + if r["dataset"] == dataset and float(r["alpha"]) == alpha} + x = np.array(BETAS) + y = np.array([float(vals[b]["pooled_validation_mean"]) for b in BETAS]) + e = np.array([float(vals[b]["sample_std"]) for b in BETAS]) + ymin = min(ymin, float((y - e).min())) + ymax = max(ymax, float((y + e).max())) + st = ALPHA_STYLE[alpha] + ax.errorbar(x, y, yerr=e, capsize=3, elinewidth=1.0, lw=1.8, + color=st["color"], marker=st["marker"], ls=st["ls"], + ms=6.5, mfc="white", mec=st["color"], mew=1.4, + label=st["label"], zorder=3) + sb = SELECTED[(dataset, alpha)] + ax.scatter([sb], [float(vals[sb]["pooled_validation_mean"])], s=210, + marker="*", facecolor=st["color"], edgecolor="white", + linewidth=0.8, zorder=5) + if sb == 0.0: + ax.annotate(f"selected beta={sb:g}", + (sb, float(vals[sb]["pooled_validation_mean"])), + textcoords="offset points", xytext=(26, 9), ha="left", + fontsize=8.5, color=st["color"], weight="bold") + elif sb == 1.0: + ax.annotate(f"selected beta={sb:g}", + (sb, float(vals[sb]["pooled_validation_mean"])), + textcoords="offset points", xytext=(-26, 9), ha="right", + fontsize=8.5, color=st["color"], weight="bold") + else: + ax.annotate(f"selected beta={sb:g}", + (sb, float(vals[sb]["pooled_validation_mean"])), + textcoords="offset points", xytext=(0, 13), ha="center", + fontsize=8.5, color=st["color"], weight="bold") + + dname = DATASET_NAME[dataset] + ax.set_xlabel("beta candidate (support_weighted aggregation)") + ax.set_ylabel("pooled validation rare-class macro-F1\n(per-client mean)") + ax.set_xticks(BETAS) + ax.set_xticklabels([f"{b:.2f}" for b in BETAS]) + ax.set_ylim(ymin - 0.012, ymax + 0.10) + ax.set_title(f"{dname}: validation-based beta selection") + ax.legend(title="non-IID condition", loc="upper center", ncols=3, + frameon=True, edgecolor="#CCCCCC", framealpha=0.95) + _footnote(fig, + "All curves are support_weighted configurations; no original_protean run is plotted here.\n" + "Metric: pooled validation per-client-mean rare-class macro-F1, averaged over seeds {0,1,2};\n" + "error bars = \u00b11 sample std across seeds. Star = argmax (selected beta) per condition; selection uses validation only.", + y=-0.13) + _save(fig, f"figA_{dataset}_pooled_validation_beta_curve") + + +# --------------------------------------------------------------------------- +# Figure B: validation-selected beta by dataset and alpha +# --------------------------------------------------------------------------- +def figure_b() -> None: + # SOURCE: artifacts/analysis/final__selection_strength.csv + rows = _read_csv(ANALYSIS / "final__selection_strength.csv") + sel = {(r["dataset"], float(r["alpha"])): float(r["selected_beta"]) for r in rows} + + fig, ax = plt.subplots(figsize=(8.8, 5.2), constrained_layout=True) + ax.set_axisbelow(True) + ax.grid(axis="y") + width = 0.24 + xbase = np.arange(len(DATASETS)) + for k, alpha in enumerate(ALPHAS): + st = ALPHA_STYLE[alpha] + xs = xbase + (k - 1) * width + ys = [sel[(d, alpha)] for d in DATASETS] + bars = ax.bar(xs, ys, width, color=st["color"], edgecolor="#333333", + linewidth=0.7, label=st["label"], zorder=3) + for b, v in zip(bars, ys): + ax.annotate(f"{v:.2f}", (b.get_x() + b.get_width() / 2, v), + textcoords="offset points", xytext=(0, 3), + ha="center", fontsize=9, weight="bold") + ax.set_xticks(xbase) + ax.set_xticklabels([DATASET_NAME[d] for d in DATASETS]) + ax.set_ylabel("validation-selected beta") + ax.set_ylim(0, 1.20) + ax.set_title("Validation-selected beta by dataset and non-IID condition") + ax.legend(title="Dirichlet alpha", loc="upper center", ncols=3, + frameon=True, edgecolor="#CCCCCC") + _footnote(fig, + "Selection = best among the tested candidates {0, 0.25, 0.50, 0.75, 1.00} under pooled validation\n" + "per-client-mean rare-class macro-F1 (support_weighted). This is a selection surface, not a performance\n" + "comparison; a higher beta is not automatically better.", + y=-0.13) + _save(fig, "figB_selected_beta_by_condition") + + +# --------------------------------------------------------------------------- +# Figure C: realized class-support imbalance (mean support CV) +# --------------------------------------------------------------------------- +def figure_c() -> None: + # SOURCE: artifacts/analysis/final__realized_heterogeneity.csv + rows = _read_csv(ANALYSIS / "final__realized_heterogeneity.csv") + cv = {(r["dataset"], float(r["alpha"])): float(r["support_cv_mean"]) for r in rows} + + fig, ax = plt.subplots(figsize=(8.8, 5.2), constrained_layout=True) + ax.set_axisbelow(True) + ax.grid(axis="y") + width = 0.24 + xbase = np.arange(len(DATASETS)) + for k, alpha in enumerate(ALPHAS): + st = ALPHA_STYLE[alpha] + xs = xbase + (k - 1) * width + ys = [cv[(d, alpha)] for d in DATASETS] + bars = ax.bar(xs, ys, width, color=st["color"], edgecolor="#333333", + linewidth=0.7, label=st["label"], zorder=3) + for b, v in zip(bars, ys): + ax.annotate(f"{v:.3f}", (b.get_x() + b.get_width() / 2, v), + textcoords="offset points", xytext=(0, 3), + ha="center", fontsize=9) + ax.set_xticks(xbase) + ax.set_xticklabels([DATASET_NAME[d] for d in DATASETS]) + ax.set_ylabel("mean class-support CV of N_ij") + ax.set_ylim(0, max(cv.values()) * 1.20) + ax.set_title("Realized class-support imbalance after Dirichlet partitioning") + ax.legend(title="configured Dirichlet alpha", loc="upper right", + frameon=True, edgecolor="#CCCCCC") + _footnote(fig, + "Coefficient of variation (std/mean) of N_ij across contributing clients, averaged over classes and seeds.\n" + "Larger CV = stronger realized support imbalance. alpha is the configuration knob; this is the realized data\n" + "property, not a performance comparison.", + y=-0.13) + _save(fig, "figC_realized_support_cv") + + +# --------------------------------------------------------------------------- +# Figure D: prototype displacement beta=1 vs beta=0 control +# --------------------------------------------------------------------------- +def figure_d() -> None: + # SOURCE: artifacts/analysis/final__prototype_displacement_by_beta.csv + rows = _read_csv(ANALYSIS / "final__prototype_displacement_by_beta.csv") + disp = {(r["dataset"], float(r["alpha"]), float(r["beta"])): float(r["mean_displacement"]) + for r in rows} + + conditions = [(d, a) for d in DATASETS for a in ALPHAS] + labels = [f"{DATASET_NAME[d]} \u03b1={a:.2f}" for d, a in conditions] + vals = [disp[(d, a, 1.0)] for d, a in conditions] + colors = [DATASET_STYLE[d]["color"] for d, _ in conditions] + + fig, ax = plt.subplots(figsize=(8.8, 5.4), constrained_layout=True) + ax.set_axisbelow(True) + ax.grid(axis="x") + ypos = np.arange(len(conditions)) + bars = ax.barh(ypos, vals, color=colors, edgecolor="#333333", linewidth=0.7, zorder=3) + for b, v in zip(bars, vals): + ax.annotate(f"{v:.3f}", (v, b.get_y() + b.get_height() / 2), + textcoords="offset points", xytext=(4, 0), va="center", + fontsize=9, weight="bold") + ax.set_yticks(ypos) + ax.set_yticklabels(labels) + ax.invert_yaxis() + ax.set_xlabel(r"mean $\||\bar{C}_j(\beta=1) - \bar{C}_j(\beta=0)\||$ across classes") + ax.set_xlim(0, max(vals) * 1.18) + ax.set_title("Support-weighted aggregation: beta=1 vs beta=0 control") + handles = [Line2D([0], [0], color=DATASET_STYLE[d]["color"], lw=8, + label=DATASET_NAME[d]) for d in DATASETS] + ax.legend(handles=handles, title="dataset", loc="lower right", + frameon=True, edgecolor="#CCCCCC") + _footnote(fig, + "Direct aggregation effect on fixed round-0 snapshots using identical local prototypes and N_ij; no retraining.\n" + "beta=0 is the reference CONTROL (equal weighting over S_j) and is NOT automatically identical to original_protean.\n" + "Displacement is a representation metric, not detection quality.", + y=-0.11) + _save(fig, "figD_prototype_displacement_beta0_vs_beta1") + + +# --------------------------------------------------------------------------- +# Figure E: realized support CV vs prototype displacement +# --------------------------------------------------------------------------- +def figure_e() -> None: + # SOURCE: final__realized_heterogeneity.csv + final__prototype_displacement_by_beta.csv + het = _read_csv(ANALYSIS / "final__realized_heterogeneity.csv") + disp = _read_csv(ANALYSIS / "final__prototype_displacement_by_beta.csv") + cv = {(r["dataset"], float(r["alpha"])): float(r["support_cv_mean"]) for r in het} + d1 = {(r["dataset"], float(r["alpha"])): float(r["mean_displacement"]) + for r in disp if float(r["beta"]) == 1.0} + + fig, ax = plt.subplots(figsize=(8.2, 5.6), constrained_layout=True) + ax.set_axisbelow(True) + ax.grid(True) + for d in DATASETS: + st = DATASET_STYLE[d] + xs = [cv[(d, a)] for a in ALPHAS] + ys = [d1[(d, a)] for a in ALPHAS] + ax.scatter(xs, ys, s=140, marker=st["marker"], color=st["color"], + edgecolor="white", linewidth=1.2, label=st["label"], zorder=4) + off = (7, -15) if d == "xiiotid" else (9, 6) + for a, x, y in zip(ALPHAS, xs, ys): + ax.annotate(f"\u03b1={a:.2f}", (x, y), textcoords="offset points", + xytext=off, fontsize=8.8, color=st["color"]) + ax.set_xlabel("realized mean class-support CV of N_ij") + ax.set_ylabel(r"mean prototype displacement $\||\bar{C}_j(1)-\bar{C}_j(0)\||$") + ax.set_title("Realized support imbalance vs prototype displacement (beta=1 vs beta=0)") + ax.legend(title="dataset", loc="upper left", frameon=True, edgecolor="#CCCCCC") + _footnote(fig, + "Each point is one (dataset, alpha) condition. Descriptive association only;\n" + "the imbalance-displacement link is mechanical under the beta formula and this plot is not causal evidence.", + y=-0.13) + _save(fig, "figE_support_cv_vs_displacement") + + +# --------------------------------------------------------------------------- +# Figure F: selected-beta support_weighted vs ORIGINAL PROTEAN (test) +# --------------------------------------------------------------------------- +def figure_f() -> None: + # SOURCE: artifacts/analysis/final__six_condition_master.csv + rows = _read_csv(ANALYSIS / "final__six_condition_master.csv") + master = {(r["dataset"], float(r["alpha"])): r for r in rows} + conditions = [(d, a) for d in DATASETS for a in ALPHAS] + labels = [f"{DATASET_NAME[d]}\n\u03b1={a:.2f}" for d, a in conditions] + + orig = np.array([float(master[c]["original_rare_macro_f1_mean"]) for c in conditions]) + orig_e = np.array([float(master[c]["original_rare_macro_f1_std"]) for c in conditions]) + sel = np.array([float(master[c]["selected_rare_macro_f1_mean"]) for c in conditions]) + sel_e = np.array([float(master[c]["selected_rare_macro_f1_std"]) for c in conditions]) + betas = [float(master[c]["selected_beta"]) for c in conditions] + delta = sel - orig + + fig, ax = plt.subplots(figsize=(11.4, 6.0), constrained_layout=True) + ax.set_axisbelow(True) + ax.grid(axis="y") + x = np.arange(len(conditions)) + w = 0.36 + ax.bar(x - w / 2, orig, w, yerr=orig_e, capsize=3.5, color=CHARCOAL, + edgecolor="#1A1A1A", linewidth=0.7, label="original_protean", zorder=3) + ax.bar(x + w / 2, sel, w, yerr=sel_e, capsize=3.5, color=TEAL, + edgecolor="#0A4F4E", linewidth=0.7, hatch="///", + label="support_weighted (validation-selected beta)", zorder=3) + + for xi, db, sb in zip(x, delta, betas): + if sb == 0.0: + colour = GREY + elif db > 0: + colour = GREEN + else: + colour = RED + sign = "+" if db >= 0 else "\u2212" + top = max(orig[xi] + orig_e[xi], sel[xi] + sel_e[xi]) + ax.annotate(f"{sign}{abs(db):.4f}", (xi, top + 0.012), ha="center", + fontsize=9.5, weight="bold", color=colour) + ax.annotate(f"beta={sb:g}", (xi, top + 0.030), ha="center", fontsize=8.5, + color="#444444") + ax.set_xticks(x) + ax.set_xticklabels(labels) + ax.set_ylabel("test rare-class macro-F1\n(mean over seeds {0,1,2})") + ax.set_ylim(0.5, max(max(orig + orig_e), max(sel + sel_e)) + 0.10) + ax.set_title("Test rare-class macro-F1: original PROTEAN vs validation-selected beta") + ax.legend(loc="upper left", frameon=True, edgecolor="#CCCCCC") + _footnote(fig, + "Bar height = mean test rare-class macro-F1 over seeds {0,1,2}; error bars = \u00b11 sample std across seeds.\n" + "Delta (top) = selected-beta minus original_protean; green = positive, red = negative, grey = selected beta 0\n" + "(floating-point ~0). No statistical significance test is applied (3 seeds). Gains are condition-dependent;\n" + "the proposed weighting does NOT consistently outperform original PROTEAN.", + y=-0.13) + _save(fig, "figF_selected_vs_original_test_rare_macro_f1") + + +# --------------------------------------------------------------------------- +# Figure G: per-client effect of selected beta relative to beta=0 control +# --------------------------------------------------------------------------- +def figure_g() -> None: + # SOURCE: artifacts/analysis/final__client_effect_summary.csv + rows = _read_csv(ANALYSIS / "final__client_effect_summary.csv") + rec = {(r["dataset"], float(r["alpha"]), int(r["seed"]), float(r["beta"])): r + for r in rows} + imp_col, deg_col, unc_col = "#2E8B57", "#C0392B", "#BFBFBF" + + fig, axes = plt.subplots(1, 2, figsize=(12.6, 5.4), constrained_layout=True, + sharey=True) + for ax, d in zip(axes, DATASETS): + ax.set_axisbelow(True) + ax.grid(axis="y") + xticks, xticklabels = [], [] + pos = 0.0 + for a in ALPHAS: + for s in (0, 1, 2): + r = rec[(d, a, s, SELECTED[(d, a)])] + imp, deg, unc = (int(r["n_improve_vs_beta0"]), + int(r["n_degrade_vs_beta0"]), + int(r["n_unchanged"])) + ax.bar([pos], [imp], width=0.82, color=imp_col, edgecolor="#1F5C39", + linewidth=0.6, hatch="///", zorder=3) + ax.bar([pos], [deg], bottom=[imp], width=0.82, color=deg_col, + edgecolor="#7A1F14", linewidth=0.6, hatch="\\\\\\", zorder=3) + ax.bar([pos], [unc], bottom=[imp + deg], width=0.82, color=unc_col, + edgecolor="#7A7A7A", linewidth=0.6, hatch="..", zorder=3) + xticks.append(pos) + xticklabels.append(f"seed {s}") + pos += 1 + pos += 0.8 + ax.set_xticks(xticks) + ax.set_xticklabels(xticklabels, fontsize=8.2) + for k, a in enumerate(ALPHAS): + center = k * 3 + 1 + k * 0.8 + ax.text(center, 10.8, f"\u03b1={a:.2f}", ha="center", fontsize=10, + weight="bold") + if k < len(ALPHAS) - 1: + ax.axvline(k * 3 + 2.5 + k * 0.8 + 0.4, color="#DDDDDD", lw=0.8) + ax.set_ylim(0, 11.9) + ax.set_yticks([0, 2, 4, 6, 8, 10]) + ax.set_title(DATASET_NAME[d]) + ax.set_xlabel("seed within each alpha condition") + + axes[0].set_ylabel("clients (out of 10)") + handles = [plt.Rectangle((0, 0), 1, 1, facecolor=imp_col, edgecolor="#1F5C39", hatch="///"), + plt.Rectangle((0, 0), 1, 1, facecolor=deg_col, edgecolor="#7A1F14", hatch="\\\\\\"), + plt.Rectangle((0, 0), 1, 1, facecolor=unc_col, edgecolor="#7A7A7A", hatch="..")] + fig.legend(handles, ["improved", "degraded", "unchanged"], title="vs beta=0 control", + loc="upper center", bbox_to_anchor=(0.5, 1.04), ncols=3, + frameon=True, edgecolor="#CCCCCC") + fig.suptitle("Per-client effect of the validation-selected beta relative to the beta=0 control (validation split)", + fontsize=12.5, weight="bold", y=1.12) + _footnote(fig, + "Comparison is the selected-beta support_weighted run vs the support_weighted beta=0 run (CONTROL), using per-client validation\n" + "rare-class macro-F1; it is NOT a comparison against original_protean. At both alpha=0.25 conditions the selected beta is 0,\n" + "so all clients are unchanged. Client ids are per-seed realizations and are not pooled.", + y=-0.11) + _save(fig, "figG_client_effect_at_selected_beta") + + +# --------------------------------------------------------------------------- +# Supplemental Figure H: paired test confusion matrices +# --------------------------------------------------------------------------- +def figure_h(dataset="5gnidd", alpha=0.75, seed=0, selected_beta=0.5) -> bool: + astr = f"{alpha:g}" + runs = { + "original_protean": f"{dataset}__alpha{astr}__seed{seed}__original_protean", + "selected_beta": f"{dataset}__alpha{astr}__seed{seed}__support_weighted__beta{selected_beta:g}", + } + paths = {k: RESULTS / v / "test_confusion_matrix.json" for k, v in runs.items()} + meta_path = RESULTS / runs["original_protean"] / "metadata.json" + if not all(p.exists() for p in paths.values()) or not meta_path.exists(): + print("figH skipped: missing inputs") + return False + + meta = json.loads(meta_path.read_text()) + meta_sel = json.loads((RESULTS / runs["selected_beta"] / "metadata.json").read_text()) + same_partition = (meta["dataset"] == meta_sel["dataset"] == dataset + and abs(float(meta["alpha"]) - float(meta_sel["alpha"])) < 1e-9 + and int(meta["seed"]) == int(meta_sel["seed"]) == seed + and meta["manifest_content_hash"] == meta_sel["manifest_content_hash"] + and abs(float(meta_sel["beta"]) - selected_beta) < 1e-9) + if not same_partition or selected_beta != SELECTED[(dataset, alpha)]: + print("figH skipped: paired-condition verification failed") + return False + + class_names = meta["class_names"] + mats = {k: np.array(json.loads(p.read_text())) for k, p in paths.items()} + + from matplotlib.colors import LogNorm + vmax = max(m.max() for m in mats.values()) + vmin = 1 + fig, axes = plt.subplots(1, 2, figsize=(13.2, 6.2), constrained_layout=True) + titles = { + "original_protean": "ORIGINAL PROTEAN", + "selected_beta": f"SELECTED-BETA PROTEAN (beta={selected_beta:g})", + } + im = None + for ax, key in zip(axes, ["original_protean", "selected_beta"]): + m = mats[key] + im = ax.imshow(m, cmap="YlGnBu", norm=LogNorm(vmin=vmin, vmax=vmax), + aspect="equal") + ax.set_xticks(range(len(class_names))) + ax.set_yticks(range(len(class_names))) + ax.set_xticklabels(class_names, rotation=45, ha="right", fontsize=8.2) + ax.set_yticklabels(class_names, fontsize=8.2) + ax.set_xlabel("predicted class") + ax.set_title(titles[key], fontsize=11) + thr = m.max() * 0.06 + for i in range(m.shape[0]): + for j in range(m.shape[1]): + v = int(m[i, j]) + if v == 0: + continue + txt = f"{v/1000:.1f}k" if v >= 10000 else str(v) + ax.text(j, i, txt, ha="center", va="center", fontsize=6.4, + color="white" if m[i, j] > thr else "#333333") + axes[0].set_ylabel("true class") + cbar = fig.colorbar(im, ax=axes, fraction=0.025, pad=0.02) + cbar.set_label("test samples (raw counts, log colour scale)") + dname = DATASET_NAME.get(dataset, dataset) + fig.suptitle( + f"Paired test confusion matrices - {dname}, alpha={alpha:g}, seed={seed} " + f"(same manifest, split, seed, class order)", + fontsize=11.5, weight="bold", y=1.05) + _footnote(fig, + "Left = original_protean; right = support_weighted at the validation-selected beta. Raw counts; identical class order and colour scale.\n" + "Dominant off-diagonal error: Benign predicted as UDPFlood, essentially unchanged by beta. No statistical significance is inferred.", + y=-0.20) + _save(fig, f"figH_confusion_matrix_{dataset}_a{alpha:g}_seed{seed}") + return True + + +def main() -> None: + _apply_style() + for d in DATASETS: + figure_a(d) + figure_b() + figure_c() + figure_d() + figure_e() + figure_f() + figure_g() + figure_h() + + +if __name__ == "__main__": + main()