docs: redesign final research figures and update README

This commit is contained in:
willtanoe committed 2026-10-09 00:47:01 +07:00
1 parent 8a9bdcf63c
commit 87f9b14ada
20 files changed
+609 -18

No files matched your search

+89 -18
View File
@@ -472,47 +472,117 @@ Representative confusion-matrix files:
## 16. Final Figures and Visualizations
All figures are under `artifacts/figures/final/` and load from the `final__*` analysis tables. They
were generated with matplotlib; no training was involved.
All figures live in `artifacts/figures/final/` and are regenerated from the audited
`artifacts/analysis/final__*` tables by `scripts/make_final_figures.py` (no training; every number is
loaded from an artifact). Each figure is saved as a high-resolution PNG (220 dpi) and a vector PDF.
**Fig A — pooled validation rare-class macro-F1 vs beta** (per dataset). Axes: `beta` (x) vs pooled
validation per-client-mean rare-class macro-F1 (y); one line per alpha. Evidence: the validation
curve is non-monotone and the selected `beta` differs by condition.
**Naming and legend conventions used throughout.**
- **Charcoal (`original_protean`)**: the frozen reproduced baseline.
- **Teal (`support_weighted`, validation-selected beta)**: the proposed method at the beta selected by
pooled validation for that condition.
- **Gray (`beta = 0` control)**: the equal-weight `support_weighted` run; a control, *not*
automatically the same recorded configuration as `original_protean`.
- Alpha conditions are distinguished by colour **and** marker **and** line style, so colour is never
the only cue.
**Comparison types (so figures are not misread).**
| Figure | Comparison type | Baseline actually plotted |
|---|---|---|
| A (both) | Beta candidate comparison within `support_weighted` | none (support_weighted candidates only) |
| B | Beta selection surface | none (selection, not performance) |
| C | Dataset heterogeneity | none (data distribution) |
| D, E | Fixed-snapshot comparison vs `beta = 0` | `beta = 0` control |
| F | Direct comparison vs `original_protean` | `original_protean` |
| G | Per-client comparison vs `beta = 0` | `beta = 0` control |
| H | Paired `original_protean` vs selected beta | `original_protean` |
### Fig A1 / A2 — Validation-based beta selection (support_weighted)
**Compared:** the five `support_weighted` beta candidates, separately for each alpha. **Level/split:**
pooled validation, per-client mean across seeds `{0,1,2}`. **Metric:** rare-class macro-F1
(per-client mean). **Observation:** the validation response is non-monotone and the selected beta
differs by condition. **Limits:** this is a selection figure; no `original_protean` run is plotted,
and the error bars are ±1 sample std across three project-defined seeds, not a confidence interval.
![Pooled validation beta curve, 5G-NIDD](artifacts/figures/final/figA_5gnidd_pooled_validation_beta_curve.png)
![Pooled validation beta curve, X-IIoTID](artifacts/figures/final/figA_xiiotid_pooled_validation_beta_curve.png)
**Fig B — selected beta by condition.** Bar chart of the six validation-selected betas; shows that
selections take `{0.00, 0.50, 1.00}` and are not universal.
### Fig B — Validation-selected beta by dataset and alpha
**Compared:** none (selection only). **Level:** pooled validation. **Metric:** selected beta under
the selection rule. **Observation:** selections take `{0.00, 0.50, 1.00}` and are not universal.
**Limits:** selection surface, not a performance comparison; a higher beta is not automatically
better. No `original_protean` comparison is implied.
![Selected beta by condition](artifacts/figures/final/figB_selected_beta_by_condition.png)
**Fig C — realized support CV.** Condition-level mean class-support CV; shows realized imbalance
decreases as alpha rises.
### Fig C — Realized class-support imbalance
**Compared:** none (data distribution). **Level:** training partitions. **Metric:** mean class-support
CV of `N_ij` across contributing clients (std/mean), averaged over classes and seeds. **Observation:**
realized imbalance decreases as the configured alpha rises. **Limits:** this describes the data,
not the relative performance of original vs modified PROTEAN; Dirichlet alpha is the configuration
knob and is not the same as the realized imbalance.
![Realized support CV](artifacts/figures/final/figC_realized_support_cv.png)
**Fig D — prototype displacement at beta=0 vs beta=1.** Fixed-snapshot mean displacement per class;
shows displacement grows with `beta` and larger where support imbalance is larger.
### Fig D — Support-weighted aggregation: beta=1 vs beta=0 control
**Compared:** `beta = 1` against the `beta = 0` control (not `original_protean`). **Level:**
fixed round-0 snapshots, same local prototypes and `N_ij`, no retraining. **Metric:** mean
`‖Cbar_j(1) − Cbar_j(0)‖` across classes. **Observation:** displacement increases with beta and is
largest where realized imbalance is largest. **Limits:** displacement is a representation metric,
not detection quality.
![Prototype displacement beta0 vs beta1](artifacts/figures/final/figD_prototype_displacement_beta0_vs_beta1.png)
**Fig E — support CV vs displacement.** Scatter of realized support CV (x) vs beta=1 displacement
(y) across conditions; a descriptive co-variation, not a causal relation.
### Fig E — Realized support CV vs prototype displacement
**Compared:** `beta = 1` displacement against the `beta = 0` control, across conditions. **Level:**
fixed snapshots. **Metric:** realized support CV (x) vs `beta = 1` displacement (y). **Observation:**
larger realized imbalance accompanies larger displacement. **Limits:** descriptive association only;
the imbalance-displacement link is mechanical under the beta formula and is not causal evidence.
![Support CV vs displacement](artifacts/figures/final/figE_support_cv_vs_displacement.png)
**Fig F — selected vs original test rare-class macro-F1.** Paired comparison per condition; shows
small, mixed-sign differences.
### Fig F — Selected-beta PROTEAN vs ORIGINAL PROTEAN (test)
**Compared:** `original_protean` (charcoal) against `support_weighted` at the validation-selected
beta (teal), per condition. **Level/split:** test, mean over seeds `{0,1,2}`. **Metric:** test
rare-class macro-F1; error bars = ±1 sample std across seeds. **Observation:** signed deltas are
`+0.0230`, `+0.0014`, `+0.0168` (positive), `−0.0086` (negative), and `−0.0007`/`−0.0010` at the two
`beta = 0` conditions (floating-point ~0). **Limits:** no statistical significance test (three
seeds); the largest gain is within seed noise; the weighting does not consistently outperform
`original_protean`.
![Selected vs original test rare macro-F1](artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.png)
**Fig G — per-client effect at the selected beta.** Per-client validation delta relative to
`beta = 0`; shows within-condition heterogeneity across clients.
### Fig G — Per-client effect of selected beta relative to the beta=0 control
**Compared:** selected-beta `support_weighted` against the `beta = 0` control (not
`original_protean`). **Level/split:** validation, per client, per seed. **Metric:** per-client
rare-class macro-F1 delta; a client is "improved"/"degraded" if its delta is above/below 0,
"unchanged" otherwise. **Observation:** within a condition some clients improve and others degrade;
at both `alpha = 0.25` conditions the selected beta is 0, so all clients are unchanged. **Limits:**
client ids are per-seed realizations and are not pooled.
![Client effect at selected beta](artifacts/figures/final/figG_client_effect_at_selected_beta.png)
### Fig H (supplemental) — Paired test confusion matrices
**Compared:** `original_protean` (left) against selected-beta `support_weighted` (right) for a valid
paired condition (5G-NIDD, `alpha = 0.75`, `seed = 0`). **Level/split:** test; global counts over the
evaluation split. **Metric:** raw 9×9 counts (log colour scale), identical class order and colour
scale. **Observation:** the dominant off-diagonal error is `Benign → UDPFlood` (~56.9k vs ~57.1k) and
is essentially unchanged by beta. **Limits:** no statistical significance is inferred from the visual
difference; the matrices come from the same manifest, split, seed, and class order.
![Paired confusion matrices](artifacts/figures/final/figH_confusion_matrix_5gnidd_a0.75_seed0.png)
## 17. Main Scientific Findings
1. **Changing `beta` changes the global prototype.** Contribution weights become proportional to
@@ -650,7 +720,8 @@ Analysis outputs (`artifacts/analysis/`), by function:
| Prototype snapshots & weights | `*__prototype_snapshot.json`, `*__prototype_weights.csv`, `*__prototype_effect*.csv` |
Partition manifests: `artifacts/partitions/<dataset>__alpha<alpha>__seed<seed>.json` (18 files).
Figures: `artifacts/figures/final/figA–figG*.png` (8 files).
Figures: `artifacts/figures/final/` — 9 figures (figA–figH) as PNG (220 dpi) plus matching vector PDF;
regenerated reproducibly by `scripts/make_final_figures.py`.
Key documents:
Binary file not shown.

Before

Width:  |  Height:  |  Size: 51 KiB

After

Width:  |  Height:  |  Size: 204 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 45 KiB

After

Width:  |  Height:  |  Size: 193 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 48 KiB

After

Width:  |  Height:  |  Size: 110 KiB

Binary file not shown.
Binary file not shown.

Before

Width:  |  Height:  |  Size: 48 KiB

After

Width:  |  Height:  |  Size: 128 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 68 KiB

After

Width:  |  Height:  |  Size: 162 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 48 KiB

After

Width:  |  Height:  |  Size: 137 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 55 KiB

After

Width:  |  Height:  |  Size: 244 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 45 KiB

After

Width:  |  Height:  |  Size: 291 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 285 KiB

+520
View File
@@ -0,0 +1,520 @@
#!/usr/bin/env python3
"""Regenerate the final PROTEAN research figures.
Reads only audited machine-readable artifacts
(``artifacts/analysis/final__*.csv`` and per-run
``results/*/test_confusion_matrix.json``) and writes publication-quality
PNG (>=200 dpi) and PDF figures to ``artifacts/figures/final``.
This script does not run training, modify results, or alter any
experimental value. Every plotted number is loaded from an authoritative
artifact; see the ``SOURCE`` comment in each function.
Run:
.venv/bin/python scripts/make_final_figures.py
"""
from __future__ import annotations
import csv
import json
from pathlib import Path
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
from matplotlib.lines import Line2D
ROOT = Path(__file__).resolve().parents[1]
ANALYSIS = ROOT / "artifacts" / "analysis"
RESULTS = ROOT / "results"
OUT = ROOT / "artifacts" / "figures" / "final"
# ---------------------------------------------------------------------------
# Consistent visual language
# ---------------------------------------------------------------------------
CHARCOAL = "#3A3A3A" # original_protean
TEAL = "#0E7C7B" # support_weighted, validation-selected beta
CONTROL = "#9A9A9A" # beta=0 control reference
ALPHA_STYLE = {
0.25: dict(color="#0072B2", marker="o", ls="-", label="alpha = 0.25"),
0.50: dict(color="#D55E00", marker="s", ls="--", label="alpha = 0.50"),
0.75: dict(color="#009E73", marker="^", ls="-.", label="alpha = 0.75"),
}
DATASET_STYLE = {
"xiiotid": dict(color="#0072B2", marker="o", label="X-IIoTID"),
"5gnidd": dict(color="#D55E00", marker="s", label="5G-NIDD"),
}
DATASET_NAME = {"xiiotid": "X-IIoTID", "5gnidd": "5G-NIDD"}
DATASETS = ["xiiotid", "5gnidd"]
ALPHAS = [0.25, 0.50, 0.75]
BETAS = [0.0, 0.25, 0.5, 0.75, 1.0]
SELECTED = {("xiiotid", 0.25): 0.0, ("xiiotid", 0.5): 0.75, ("xiiotid", 0.75): 0.5,
("5gnidd", 0.25): 0.0, ("5gnidd", 0.5): 1.0, ("5gnidd", 0.75): 0.5}
GREEN, RED, GREY = "#1B7F3B", "#B22222", "#6B6B6B"
def _apply_style() -> None:
plt.rcParams.update({
"figure.facecolor": "white",
"axes.facecolor": "white",
"axes.edgecolor": "#5A5A5A",
"axes.linewidth": 0.9,
"axes.titlesize": 13,
"axes.titleweight": "bold",
"axes.labelsize": 11,
"xtick.labelsize": 10,
"ytick.labelsize": 10,
"legend.fontsize": 9.5,
"font.size": 10.5,
"axes.spines.top": False,
"axes.spines.right": False,
"grid.color": "#B0B0B0",
"grid.alpha": 0.35,
"grid.linestyle": ":",
"savefig.dpi": 220,
"savefig.bbox": "tight",
})
def _save(fig, name: str) -> None:
OUT.mkdir(parents=True, exist_ok=True)
for ext in ("png", "pdf"):
fig.savefig(OUT / f"{name}.{ext}")
plt.close(fig)
print("wrote", OUT / f"{name}.png")
def _read_csv(path: Path) -> list[dict]:
with path.open() as fh:
return list(csv.DictReader(fh))
def _footnote(fig, text: str, y: float) -> None:
"""Place a wrapped caption below the axes; bbox_inches='tight' expands."""
fig.text(0.5, y, text, ha="center", va="top", fontsize=8.7,
color="#444444", linespacing=1.5)
# ---------------------------------------------------------------------------
# Figure A: pooled validation beta curves (support_weighted only)
# ---------------------------------------------------------------------------
def figure_a(dataset: str) -> None:
# SOURCE: artifacts/analysis/final__pooled_validation_curves.csv
rows = _read_csv(ANALYSIS / "final__pooled_validation_curves.csv")
fig, ax = plt.subplots(figsize=(8.4, 5.4), constrained_layout=True)
ax.set_axisbelow(True)
ax.grid(axis="y")
ymin, ymax = 1e9, -1e9
for alpha in ALPHAS:
vals = {float(r["beta"]): r for r in rows
if r["dataset"] == dataset and float(r["alpha"]) == alpha}
x = np.array(BETAS)
y = np.array([float(vals[b]["pooled_validation_mean"]) for b in BETAS])
e = np.array([float(vals[b]["sample_std"]) for b in BETAS])
ymin = min(ymin, float((y - e).min()))
ymax = max(ymax, float((y + e).max()))
st = ALPHA_STYLE[alpha]
ax.errorbar(x, y, yerr=e, capsize=3, elinewidth=1.0, lw=1.8,
color=st["color"], marker=st["marker"], ls=st["ls"],
ms=6.5, mfc="white", mec=st["color"], mew=1.4,
label=st["label"], zorder=3)
sb = SELECTED[(dataset, alpha)]
ax.scatter([sb], [float(vals[sb]["pooled_validation_mean"])], s=210,
marker="*", facecolor=st["color"], edgecolor="white",
linewidth=0.8, zorder=5)
if sb == 0.0:
ax.annotate(f"selected beta={sb:g}",
(sb, float(vals[sb]["pooled_validation_mean"])),
textcoords="offset points", xytext=(26, 9), ha="left",
fontsize=8.5, color=st["color"], weight="bold")
elif sb == 1.0:
ax.annotate(f"selected beta={sb:g}",
(sb, float(vals[sb]["pooled_validation_mean"])),
textcoords="offset points", xytext=(-26, 9), ha="right",
fontsize=8.5, color=st["color"], weight="bold")
else:
ax.annotate(f"selected beta={sb:g}",
(sb, float(vals[sb]["pooled_validation_mean"])),
textcoords="offset points", xytext=(0, 13), ha="center",
fontsize=8.5, color=st["color"], weight="bold")
dname = DATASET_NAME[dataset]
ax.set_xlabel("beta candidate (support_weighted aggregation)")
ax.set_ylabel("pooled validation rare-class macro-F1\n(per-client mean)")
ax.set_xticks(BETAS)
ax.set_xticklabels([f"{b:.2f}" for b in BETAS])
ax.set_ylim(ymin - 0.012, ymax + 0.10)
ax.set_title(f"{dname}: validation-based beta selection")
ax.legend(title="non-IID condition", loc="upper center", ncols=3,
frameon=True, edgecolor="#CCCCCC", framealpha=0.95)
_footnote(fig,
"All curves are support_weighted configurations; no original_protean run is plotted here.\n"
"Metric: pooled validation per-client-mean rare-class macro-F1, averaged over seeds {0,1,2};\n"
"error bars = \u00b11 sample std across seeds. Star = argmax (selected beta) per condition; selection uses validation only.",
y=-0.13)
_save(fig, f"figA_{dataset}_pooled_validation_beta_curve")
# ---------------------------------------------------------------------------
# Figure B: validation-selected beta by dataset and alpha
# ---------------------------------------------------------------------------
def figure_b() -> None:
# SOURCE: artifacts/analysis/final__selection_strength.csv
rows = _read_csv(ANALYSIS / "final__selection_strength.csv")
sel = {(r["dataset"], float(r["alpha"])): float(r["selected_beta"]) for r in rows}
fig, ax = plt.subplots(figsize=(8.8, 5.2), constrained_layout=True)
ax.set_axisbelow(True)
ax.grid(axis="y")
width = 0.24
xbase = np.arange(len(DATASETS))
for k, alpha in enumerate(ALPHAS):
st = ALPHA_STYLE[alpha]
xs = xbase + (k - 1) * width
ys = [sel[(d, alpha)] for d in DATASETS]
bars = ax.bar(xs, ys, width, color=st["color"], edgecolor="#333333",
linewidth=0.7, label=st["label"], zorder=3)
for b, v in zip(bars, ys):
ax.annotate(f"{v:.2f}", (b.get_x() + b.get_width() / 2, v),
textcoords="offset points", xytext=(0, 3),
ha="center", fontsize=9, weight="bold")
ax.set_xticks(xbase)
ax.set_xticklabels([DATASET_NAME[d] for d in DATASETS])
ax.set_ylabel("validation-selected beta")
ax.set_ylim(0, 1.20)
ax.set_title("Validation-selected beta by dataset and non-IID condition")
ax.legend(title="Dirichlet alpha", loc="upper center", ncols=3,
frameon=True, edgecolor="#CCCCCC")
_footnote(fig,
"Selection = best among the tested candidates {0, 0.25, 0.50, 0.75, 1.00} under pooled validation\n"
"per-client-mean rare-class macro-F1 (support_weighted). This is a selection surface, not a performance\n"
"comparison; a higher beta is not automatically better.",
y=-0.13)
_save(fig, "figB_selected_beta_by_condition")
# ---------------------------------------------------------------------------
# Figure C: realized class-support imbalance (mean support CV)
# ---------------------------------------------------------------------------
def figure_c() -> None:
# SOURCE: artifacts/analysis/final__realized_heterogeneity.csv
rows = _read_csv(ANALYSIS / "final__realized_heterogeneity.csv")
cv = {(r["dataset"], float(r["alpha"])): float(r["support_cv_mean"]) for r in rows}
fig, ax = plt.subplots(figsize=(8.8, 5.2), constrained_layout=True)
ax.set_axisbelow(True)
ax.grid(axis="y")
width = 0.24
xbase = np.arange(len(DATASETS))
for k, alpha in enumerate(ALPHAS):
st = ALPHA_STYLE[alpha]
xs = xbase + (k - 1) * width
ys = [cv[(d, alpha)] for d in DATASETS]
bars = ax.bar(xs, ys, width, color=st["color"], edgecolor="#333333",
linewidth=0.7, label=st["label"], zorder=3)
for b, v in zip(bars, ys):
ax.annotate(f"{v:.3f}", (b.get_x() + b.get_width() / 2, v),
textcoords="offset points", xytext=(0, 3),
ha="center", fontsize=9)
ax.set_xticks(xbase)
ax.set_xticklabels([DATASET_NAME[d] for d in DATASETS])
ax.set_ylabel("mean class-support CV of N_ij")
ax.set_ylim(0, max(cv.values()) * 1.20)
ax.set_title("Realized class-support imbalance after Dirichlet partitioning")
ax.legend(title="configured Dirichlet alpha", loc="upper right",
frameon=True, edgecolor="#CCCCCC")
_footnote(fig,
"Coefficient of variation (std/mean) of N_ij across contributing clients, averaged over classes and seeds.\n"
"Larger CV = stronger realized support imbalance. alpha is the configuration knob; this is the realized data\n"
"property, not a performance comparison.",
y=-0.13)
_save(fig, "figC_realized_support_cv")
# ---------------------------------------------------------------------------
# Figure D: prototype displacement beta=1 vs beta=0 control
# ---------------------------------------------------------------------------
def figure_d() -> None:
# SOURCE: artifacts/analysis/final__prototype_displacement_by_beta.csv
rows = _read_csv(ANALYSIS / "final__prototype_displacement_by_beta.csv")
disp = {(r["dataset"], float(r["alpha"]), float(r["beta"])): float(r["mean_displacement"])
for r in rows}
conditions = [(d, a) for d in DATASETS for a in ALPHAS]
labels = [f"{DATASET_NAME[d]} \u03b1={a:.2f}" for d, a in conditions]
vals = [disp[(d, a, 1.0)] for d, a in conditions]
colors = [DATASET_STYLE[d]["color"] for d, _ in conditions]
fig, ax = plt.subplots(figsize=(8.8, 5.4), constrained_layout=True)
ax.set_axisbelow(True)
ax.grid(axis="x")
ypos = np.arange(len(conditions))
bars = ax.barh(ypos, vals, color=colors, edgecolor="#333333", linewidth=0.7, zorder=3)
for b, v in zip(bars, vals):
ax.annotate(f"{v:.3f}", (v, b.get_y() + b.get_height() / 2),
textcoords="offset points", xytext=(4, 0), va="center",
fontsize=9, weight="bold")
ax.set_yticks(ypos)
ax.set_yticklabels(labels)
ax.invert_yaxis()
ax.set_xlabel(r"mean $\||\bar{C}_j(\beta=1) - \bar{C}_j(\beta=0)\||$ across classes")
ax.set_xlim(0, max(vals) * 1.18)
ax.set_title("Support-weighted aggregation: beta=1 vs beta=0 control")
handles = [Line2D([0], [0], color=DATASET_STYLE[d]["color"], lw=8,
label=DATASET_NAME[d]) for d in DATASETS]
ax.legend(handles=handles, title="dataset", loc="lower right",
frameon=True, edgecolor="#CCCCCC")
_footnote(fig,
"Direct aggregation effect on fixed round-0 snapshots using identical local prototypes and N_ij; no retraining.\n"
"beta=0 is the reference CONTROL (equal weighting over S_j) and is NOT automatically identical to original_protean.\n"
"Displacement is a representation metric, not detection quality.",
y=-0.11)
_save(fig, "figD_prototype_displacement_beta0_vs_beta1")
# ---------------------------------------------------------------------------
# Figure E: realized support CV vs prototype displacement
# ---------------------------------------------------------------------------
def figure_e() -> None:
# SOURCE: final__realized_heterogeneity.csv + final__prototype_displacement_by_beta.csv
het = _read_csv(ANALYSIS / "final__realized_heterogeneity.csv")
disp = _read_csv(ANALYSIS / "final__prototype_displacement_by_beta.csv")
cv = {(r["dataset"], float(r["alpha"])): float(r["support_cv_mean"]) for r in het}
d1 = {(r["dataset"], float(r["alpha"])): float(r["mean_displacement"])
for r in disp if float(r["beta"]) == 1.0}
fig, ax = plt.subplots(figsize=(8.2, 5.6), constrained_layout=True)
ax.set_axisbelow(True)
ax.grid(True)
for d in DATASETS:
st = DATASET_STYLE[d]
xs = [cv[(d, a)] for a in ALPHAS]
ys = [d1[(d, a)] for a in ALPHAS]
ax.scatter(xs, ys, s=140, marker=st["marker"], color=st["color"],
edgecolor="white", linewidth=1.2, label=st["label"], zorder=4)
off = (7, -15) if d == "xiiotid" else (9, 6)
for a, x, y in zip(ALPHAS, xs, ys):
ax.annotate(f"\u03b1={a:.2f}", (x, y), textcoords="offset points",
xytext=off, fontsize=8.8, color=st["color"])
ax.set_xlabel("realized mean class-support CV of N_ij")
ax.set_ylabel(r"mean prototype displacement $\||\bar{C}_j(1)-\bar{C}_j(0)\||$")
ax.set_title("Realized support imbalance vs prototype displacement (beta=1 vs beta=0)")
ax.legend(title="dataset", loc="upper left", frameon=True, edgecolor="#CCCCCC")
_footnote(fig,
"Each point is one (dataset, alpha) condition. Descriptive association only;\n"
"the imbalance-displacement link is mechanical under the beta formula and this plot is not causal evidence.",
y=-0.13)
_save(fig, "figE_support_cv_vs_displacement")
# ---------------------------------------------------------------------------
# Figure F: selected-beta support_weighted vs ORIGINAL PROTEAN (test)
# ---------------------------------------------------------------------------
def figure_f() -> None:
# SOURCE: artifacts/analysis/final__six_condition_master.csv
rows = _read_csv(ANALYSIS / "final__six_condition_master.csv")
master = {(r["dataset"], float(r["alpha"])): r for r in rows}
conditions = [(d, a) for d in DATASETS for a in ALPHAS]
labels = [f"{DATASET_NAME[d]}\n\u03b1={a:.2f}" for d, a in conditions]
orig = np.array([float(master[c]["original_rare_macro_f1_mean"]) for c in conditions])
orig_e = np.array([float(master[c]["original_rare_macro_f1_std"]) for c in conditions])
sel = np.array([float(master[c]["selected_rare_macro_f1_mean"]) for c in conditions])
sel_e = np.array([float(master[c]["selected_rare_macro_f1_std"]) for c in conditions])
betas = [float(master[c]["selected_beta"]) for c in conditions]
delta = sel - orig
fig, ax = plt.subplots(figsize=(11.4, 6.0), constrained_layout=True)
ax.set_axisbelow(True)
ax.grid(axis="y")
x = np.arange(len(conditions))
w = 0.36
ax.bar(x - w / 2, orig, w, yerr=orig_e, capsize=3.5, color=CHARCOAL,
edgecolor="#1A1A1A", linewidth=0.7, label="original_protean", zorder=3)
ax.bar(x + w / 2, sel, w, yerr=sel_e, capsize=3.5, color=TEAL,
edgecolor="#0A4F4E", linewidth=0.7, hatch="///",
label="support_weighted (validation-selected beta)", zorder=3)
for xi, db, sb in zip(x, delta, betas):
if sb == 0.0:
colour = GREY
elif db > 0:
colour = GREEN
else:
colour = RED
sign = "+" if db >= 0 else "\u2212"
top = max(orig[xi] + orig_e[xi], sel[xi] + sel_e[xi])
ax.annotate(f"{sign}{abs(db):.4f}", (xi, top + 0.012), ha="center",
fontsize=9.5, weight="bold", color=colour)
ax.annotate(f"beta={sb:g}", (xi, top + 0.030), ha="center", fontsize=8.5,
color="#444444")
ax.set_xticks(x)
ax.set_xticklabels(labels)
ax.set_ylabel("test rare-class macro-F1\n(mean over seeds {0,1,2})")
ax.set_ylim(0.5, max(max(orig + orig_e), max(sel + sel_e)) + 0.10)
ax.set_title("Test rare-class macro-F1: original PROTEAN vs validation-selected beta")
ax.legend(loc="upper left", frameon=True, edgecolor="#CCCCCC")
_footnote(fig,
"Bar height = mean test rare-class macro-F1 over seeds {0,1,2}; error bars = \u00b11 sample std across seeds.\n"
"Delta (top) = selected-beta minus original_protean; green = positive, red = negative, grey = selected beta 0\n"
"(floating-point ~0). No statistical significance test is applied (3 seeds). Gains are condition-dependent;\n"
"the proposed weighting does NOT consistently outperform original PROTEAN.",
y=-0.13)
_save(fig, "figF_selected_vs_original_test_rare_macro_f1")
# ---------------------------------------------------------------------------
# Figure G: per-client effect of selected beta relative to beta=0 control
# ---------------------------------------------------------------------------
def figure_g() -> None:
# SOURCE: artifacts/analysis/final__client_effect_summary.csv
rows = _read_csv(ANALYSIS / "final__client_effect_summary.csv")
rec = {(r["dataset"], float(r["alpha"]), int(r["seed"]), float(r["beta"])): r
for r in rows}
imp_col, deg_col, unc_col = "#2E8B57", "#C0392B", "#BFBFBF"
fig, axes = plt.subplots(1, 2, figsize=(12.6, 5.4), constrained_layout=True,
sharey=True)
for ax, d in zip(axes, DATASETS):
ax.set_axisbelow(True)
ax.grid(axis="y")
xticks, xticklabels = [], []
pos = 0.0
for a in ALPHAS:
for s in (0, 1, 2):
r = rec[(d, a, s, SELECTED[(d, a)])]
imp, deg, unc = (int(r["n_improve_vs_beta0"]),
int(r["n_degrade_vs_beta0"]),
int(r["n_unchanged"]))
ax.bar([pos], [imp], width=0.82, color=imp_col, edgecolor="#1F5C39",
linewidth=0.6, hatch="///", zorder=3)
ax.bar([pos], [deg], bottom=[imp], width=0.82, color=deg_col,
edgecolor="#7A1F14", linewidth=0.6, hatch="\\\\\\", zorder=3)
ax.bar([pos], [unc], bottom=[imp + deg], width=0.82, color=unc_col,
edgecolor="#7A7A7A", linewidth=0.6, hatch="..", zorder=3)
xticks.append(pos)
xticklabels.append(f"seed {s}")
pos += 1
pos += 0.8
ax.set_xticks(xticks)
ax.set_xticklabels(xticklabels, fontsize=8.2)
for k, a in enumerate(ALPHAS):
center = k * 3 + 1 + k * 0.8
ax.text(center, 10.8, f"\u03b1={a:.2f}", ha="center", fontsize=10,
weight="bold")
if k < len(ALPHAS) - 1:
ax.axvline(k * 3 + 2.5 + k * 0.8 + 0.4, color="#DDDDDD", lw=0.8)
ax.set_ylim(0, 11.9)
ax.set_yticks([0, 2, 4, 6, 8, 10])
ax.set_title(DATASET_NAME[d])
ax.set_xlabel("seed within each alpha condition")
axes[0].set_ylabel("clients (out of 10)")
handles = [plt.Rectangle((0, 0), 1, 1, facecolor=imp_col, edgecolor="#1F5C39", hatch="///"),
plt.Rectangle((0, 0), 1, 1, facecolor=deg_col, edgecolor="#7A1F14", hatch="\\\\\\"),
plt.Rectangle((0, 0), 1, 1, facecolor=unc_col, edgecolor="#7A7A7A", hatch="..")]
fig.legend(handles, ["improved", "degraded", "unchanged"], title="vs beta=0 control",
loc="upper center", bbox_to_anchor=(0.5, 1.04), ncols=3,
frameon=True, edgecolor="#CCCCCC")
fig.suptitle("Per-client effect of the validation-selected beta relative to the beta=0 control (validation split)",
fontsize=12.5, weight="bold", y=1.12)
_footnote(fig,
"Comparison is the selected-beta support_weighted run vs the support_weighted beta=0 run (CONTROL), using per-client validation\n"
"rare-class macro-F1; it is NOT a comparison against original_protean. At both alpha=0.25 conditions the selected beta is 0,\n"
"so all clients are unchanged. Client ids are per-seed realizations and are not pooled.",
y=-0.11)
_save(fig, "figG_client_effect_at_selected_beta")
# ---------------------------------------------------------------------------
# Supplemental Figure H: paired test confusion matrices
# ---------------------------------------------------------------------------
def figure_h(dataset="5gnidd", alpha=0.75, seed=0, selected_beta=0.5) -> bool:
astr = f"{alpha:g}"
runs = {
"original_protean": f"{dataset}__alpha{astr}__seed{seed}__original_protean",
"selected_beta": f"{dataset}__alpha{astr}__seed{seed}__support_weighted__beta{selected_beta:g}",
}
paths = {k: RESULTS / v / "test_confusion_matrix.json" for k, v in runs.items()}
meta_path = RESULTS / runs["original_protean"] / "metadata.json"
if not all(p.exists() for p in paths.values()) or not meta_path.exists():
print("figH skipped: missing inputs")
return False
meta = json.loads(meta_path.read_text())
meta_sel = json.loads((RESULTS / runs["selected_beta"] / "metadata.json").read_text())
same_partition = (meta["dataset"] == meta_sel["dataset"] == dataset
and abs(float(meta["alpha"]) - float(meta_sel["alpha"])) < 1e-9
and int(meta["seed"]) == int(meta_sel["seed"]) == seed
and meta["manifest_content_hash"] == meta_sel["manifest_content_hash"]
and abs(float(meta_sel["beta"]) - selected_beta) < 1e-9)
if not same_partition or selected_beta != SELECTED[(dataset, alpha)]:
print("figH skipped: paired-condition verification failed")
return False
class_names = meta["class_names"]
mats = {k: np.array(json.loads(p.read_text())) for k, p in paths.items()}
from matplotlib.colors import LogNorm
vmax = max(m.max() for m in mats.values())
vmin = 1
fig, axes = plt.subplots(1, 2, figsize=(13.2, 6.2), constrained_layout=True)
titles = {
"original_protean": "ORIGINAL PROTEAN",
"selected_beta": f"SELECTED-BETA PROTEAN (beta={selected_beta:g})",
}
im = None
for ax, key in zip(axes, ["original_protean", "selected_beta"]):
m = mats[key]
im = ax.imshow(m, cmap="YlGnBu", norm=LogNorm(vmin=vmin, vmax=vmax),
aspect="equal")
ax.set_xticks(range(len(class_names)))
ax.set_yticks(range(len(class_names)))
ax.set_xticklabels(class_names, rotation=45, ha="right", fontsize=8.2)
ax.set_yticklabels(class_names, fontsize=8.2)
ax.set_xlabel("predicted class")
ax.set_title(titles[key], fontsize=11)
thr = m.max() * 0.06
for i in range(m.shape[0]):
for j in range(m.shape[1]):
v = int(m[i, j])
if v == 0:
continue
txt = f"{v/1000:.1f}k" if v >= 10000 else str(v)
ax.text(j, i, txt, ha="center", va="center", fontsize=6.4,
color="white" if m[i, j] > thr else "#333333")
axes[0].set_ylabel("true class")
cbar = fig.colorbar(im, ax=axes, fraction=0.025, pad=0.02)
cbar.set_label("test samples (raw counts, log colour scale)")
dname = DATASET_NAME.get(dataset, dataset)
fig.suptitle(
f"Paired test confusion matrices - {dname}, alpha={alpha:g}, seed={seed} "
f"(same manifest, split, seed, class order)",
fontsize=11.5, weight="bold", y=1.05)
_footnote(fig,
"Left = original_protean; right = support_weighted at the validation-selected beta. Raw counts; identical class order and colour scale.\n"
"Dominant off-diagonal error: Benign predicted as UDPFlood, essentially unchanged by beta. No statistical significance is inferred.",
y=-0.20)
_save(fig, f"figH_confusion_matrix_{dataset}_a{alpha:g}_seed{seed}")
return True
def main() -> None:
_apply_style()
for d in DATASETS:
figure_a(d)
figure_b()
figure_c()
figure_d()
figure_e()
figure_f()
figure_g()
figure_h()
if __name__ == "__main__":
main()