docs: redesign final research figures and update README
No files matched your search
@@ -472,47 +472,117 @@ Representative confusion-matrix files:
|
||||
|
||||
## 16. Final Figures and Visualizations
|
||||
|
||||
All figures are under `artifacts/figures/final/` and load from the `final__*` analysis tables. They
|
||||
were generated with matplotlib; no training was involved.
|
||||
All figures live in `artifacts/figures/final/` and are regenerated from the audited
|
||||
`artifacts/analysis/final__*` tables by `scripts/make_final_figures.py` (no training; every number is
|
||||
loaded from an artifact). Each figure is saved as a high-resolution PNG (220 dpi) and a vector PDF.
|
||||
|
||||
**Fig A — pooled validation rare-class macro-F1 vs beta** (per dataset). Axes: `beta` (x) vs pooled
|
||||
validation per-client-mean rare-class macro-F1 (y); one line per alpha. Evidence: the validation
|
||||
curve is non-monotone and the selected `beta` differs by condition.
|
||||
**Naming and legend conventions used throughout.**
|
||||
|
||||
- **Charcoal (`original_protean`)**: the frozen reproduced baseline.
|
||||
- **Teal (`support_weighted`, validation-selected beta)**: the proposed method at the beta selected by
|
||||
pooled validation for that condition.
|
||||
- **Gray (`beta = 0` control)**: the equal-weight `support_weighted` run; a control, *not*
|
||||
automatically the same recorded configuration as `original_protean`.
|
||||
- Alpha conditions are distinguished by colour **and** marker **and** line style, so colour is never
|
||||
the only cue.
|
||||
|
||||
**Comparison types (so figures are not misread).**
|
||||
|
||||
| Figure | Comparison type | Baseline actually plotted |
|
||||
|---|---|---|
|
||||
| A (both) | Beta candidate comparison within `support_weighted` | none (support_weighted candidates only) |
|
||||
| B | Beta selection surface | none (selection, not performance) |
|
||||
| C | Dataset heterogeneity | none (data distribution) |
|
||||
| D, E | Fixed-snapshot comparison vs `beta = 0` | `beta = 0` control |
|
||||
| F | Direct comparison vs `original_protean` | `original_protean` |
|
||||
| G | Per-client comparison vs `beta = 0` | `beta = 0` control |
|
||||
| H | Paired `original_protean` vs selected beta | `original_protean` |
|
||||
|
||||
### Fig A1 / A2 — Validation-based beta selection (support_weighted)
|
||||
|
||||
**Compared:** the five `support_weighted` beta candidates, separately for each alpha. **Level/split:**
|
||||
pooled validation, per-client mean across seeds `{0,1,2}`. **Metric:** rare-class macro-F1
|
||||
(per-client mean). **Observation:** the validation response is non-monotone and the selected beta
|
||||
differs by condition. **Limits:** this is a selection figure; no `original_protean` run is plotted,
|
||||
and the error bars are ±1 sample std across three project-defined seeds, not a confidence interval.
|
||||
|
||||

|
||||
|
||||

|
||||
|
||||
**Fig B — selected beta by condition.** Bar chart of the six validation-selected betas; shows that
|
||||
selections take `{0.00, 0.50, 1.00}` and are not universal.
|
||||
### Fig B — Validation-selected beta by dataset and alpha
|
||||
|
||||
**Compared:** none (selection only). **Level:** pooled validation. **Metric:** selected beta under
|
||||
the selection rule. **Observation:** selections take `{0.00, 0.50, 1.00}` and are not universal.
|
||||
**Limits:** selection surface, not a performance comparison; a higher beta is not automatically
|
||||
better. No `original_protean` comparison is implied.
|
||||
|
||||

|
||||
|
||||
**Fig C — realized support CV.** Condition-level mean class-support CV; shows realized imbalance
|
||||
decreases as alpha rises.
|
||||
### Fig C — Realized class-support imbalance
|
||||
|
||||
**Compared:** none (data distribution). **Level:** training partitions. **Metric:** mean class-support
|
||||
CV of `N_ij` across contributing clients (std/mean), averaged over classes and seeds. **Observation:**
|
||||
realized imbalance decreases as the configured alpha rises. **Limits:** this describes the data,
|
||||
not the relative performance of original vs modified PROTEAN; Dirichlet alpha is the configuration
|
||||
knob and is not the same as the realized imbalance.
|
||||
|
||||

|
||||
|
||||
**Fig D — prototype displacement at beta=0 vs beta=1.** Fixed-snapshot mean displacement per class;
|
||||
shows displacement grows with `beta` and larger where support imbalance is larger.
|
||||
### Fig D — Support-weighted aggregation: beta=1 vs beta=0 control
|
||||
|
||||
**Compared:** `beta = 1` against the `beta = 0` control (not `original_protean`). **Level:**
|
||||
fixed round-0 snapshots, same local prototypes and `N_ij`, no retraining. **Metric:** mean
|
||||
`‖Cbar_j(1) − Cbar_j(0)‖` across classes. **Observation:** displacement increases with beta and is
|
||||
largest where realized imbalance is largest. **Limits:** displacement is a representation metric,
|
||||
not detection quality.
|
||||
|
||||

|
||||
|
||||
**Fig E — support CV vs displacement.** Scatter of realized support CV (x) vs beta=1 displacement
|
||||
(y) across conditions; a descriptive co-variation, not a causal relation.
|
||||
### Fig E — Realized support CV vs prototype displacement
|
||||
|
||||
**Compared:** `beta = 1` displacement against the `beta = 0` control, across conditions. **Level:**
|
||||
fixed snapshots. **Metric:** realized support CV (x) vs `beta = 1` displacement (y). **Observation:**
|
||||
larger realized imbalance accompanies larger displacement. **Limits:** descriptive association only;
|
||||
the imbalance-displacement link is mechanical under the beta formula and is not causal evidence.
|
||||
|
||||

|
||||
|
||||
**Fig F — selected vs original test rare-class macro-F1.** Paired comparison per condition; shows
|
||||
small, mixed-sign differences.
|
||||
### Fig F — Selected-beta PROTEAN vs ORIGINAL PROTEAN (test)
|
||||
|
||||
**Compared:** `original_protean` (charcoal) against `support_weighted` at the validation-selected
|
||||
beta (teal), per condition. **Level/split:** test, mean over seeds `{0,1,2}`. **Metric:** test
|
||||
rare-class macro-F1; error bars = ±1 sample std across seeds. **Observation:** signed deltas are
|
||||
`+0.0230`, `+0.0014`, `+0.0168` (positive), `−0.0086` (negative), and `−0.0007`/`−0.0010` at the two
|
||||
`beta = 0` conditions (floating-point ~0). **Limits:** no statistical significance test (three
|
||||
seeds); the largest gain is within seed noise; the weighting does not consistently outperform
|
||||
`original_protean`.
|
||||
|
||||

|
||||
|
||||
**Fig G — per-client effect at the selected beta.** Per-client validation delta relative to
|
||||
`beta = 0`; shows within-condition heterogeneity across clients.
|
||||
### Fig G — Per-client effect of selected beta relative to the beta=0 control
|
||||
|
||||
**Compared:** selected-beta `support_weighted` against the `beta = 0` control (not
|
||||
`original_protean`). **Level/split:** validation, per client, per seed. **Metric:** per-client
|
||||
rare-class macro-F1 delta; a client is "improved"/"degraded" if its delta is above/below 0,
|
||||
"unchanged" otherwise. **Observation:** within a condition some clients improve and others degrade;
|
||||
at both `alpha = 0.25` conditions the selected beta is 0, so all clients are unchanged. **Limits:**
|
||||
client ids are per-seed realizations and are not pooled.
|
||||
|
||||

|
||||
|
||||
### Fig H (supplemental) — Paired test confusion matrices
|
||||
|
||||
**Compared:** `original_protean` (left) against selected-beta `support_weighted` (right) for a valid
|
||||
paired condition (5G-NIDD, `alpha = 0.75`, `seed = 0`). **Level/split:** test; global counts over the
|
||||
evaluation split. **Metric:** raw 9×9 counts (log colour scale), identical class order and colour
|
||||
scale. **Observation:** the dominant off-diagonal error is `Benign → UDPFlood` (~56.9k vs ~57.1k) and
|
||||
is essentially unchanged by beta. **Limits:** no statistical significance is inferred from the visual
|
||||
difference; the matrices come from the same manifest, split, seed, and class order.
|
||||
|
||||

|
||||
|
||||
|
||||
## 17. Main Scientific Findings
|
||||
|
||||
1. **Changing `beta` changes the global prototype.** Contribution weights become proportional to
|
||||
@@ -650,7 +720,8 @@ Analysis outputs (`artifacts/analysis/`), by function:
|
||||
| Prototype snapshots & weights | `*__prototype_snapshot.json`, `*__prototype_weights.csv`, `*__prototype_effect*.csv` |
|
||||
|
||||
Partition manifests: `artifacts/partitions/<dataset>__alpha<alpha>__seed<seed>.json` (18 files).
|
||||
Figures: `artifacts/figures/final/figA–figG*.png` (8 files).
|
||||
Figures: `artifacts/figures/final/` — 9 figures (figA–figH) as PNG (220 dpi) plus matching vector PDF;
|
||||
regenerated reproducibly by `scripts/make_final_figures.py`.
|
||||
|
||||
Key documents:
|
||||
|
||||
|
||||
|
Before Width: | Height: | Size: 51 KiB After Width: | Height: | Size: 204 KiB |
|
Before Width: | Height: | Size: 45 KiB After Width: | Height: | Size: 193 KiB |
|
Before Width: | Height: | Size: 48 KiB After Width: | Height: | Size: 110 KiB |
|
Before Width: | Height: | Size: 48 KiB After Width: | Height: | Size: 128 KiB |
|
Before Width: | Height: | Size: 68 KiB After Width: | Height: | Size: 162 KiB |
|
Before Width: | Height: | Size: 48 KiB After Width: | Height: | Size: 137 KiB |
|
Before Width: | Height: | Size: 55 KiB After Width: | Height: | Size: 244 KiB |
|
Before Width: | Height: | Size: 45 KiB After Width: | Height: | Size: 291 KiB |
|
After Width: | Height: | Size: 285 KiB |
@@ -0,0 +1,520 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Regenerate the final PROTEAN research figures.
|
||||
|
||||
Reads only audited machine-readable artifacts
|
||||
(``artifacts/analysis/final__*.csv`` and per-run
|
||||
``results/*/test_confusion_matrix.json``) and writes publication-quality
|
||||
PNG (>=200 dpi) and PDF figures to ``artifacts/figures/final``.
|
||||
|
||||
This script does not run training, modify results, or alter any
|
||||
experimental value. Every plotted number is loaded from an authoritative
|
||||
artifact; see the ``SOURCE`` comment in each function.
|
||||
|
||||
Run:
|
||||
.venv/bin/python scripts/make_final_figures.py
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import matplotlib
|
||||
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
import numpy as np
|
||||
from matplotlib.lines import Line2D
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
ANALYSIS = ROOT / "artifacts" / "analysis"
|
||||
RESULTS = ROOT / "results"
|
||||
OUT = ROOT / "artifacts" / "figures" / "final"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Consistent visual language
|
||||
# ---------------------------------------------------------------------------
|
||||
CHARCOAL = "#3A3A3A" # original_protean
|
||||
TEAL = "#0E7C7B" # support_weighted, validation-selected beta
|
||||
CONTROL = "#9A9A9A" # beta=0 control reference
|
||||
ALPHA_STYLE = {
|
||||
0.25: dict(color="#0072B2", marker="o", ls="-", label="alpha = 0.25"),
|
||||
0.50: dict(color="#D55E00", marker="s", ls="--", label="alpha = 0.50"),
|
||||
0.75: dict(color="#009E73", marker="^", ls="-.", label="alpha = 0.75"),
|
||||
}
|
||||
DATASET_STYLE = {
|
||||
"xiiotid": dict(color="#0072B2", marker="o", label="X-IIoTID"),
|
||||
"5gnidd": dict(color="#D55E00", marker="s", label="5G-NIDD"),
|
||||
}
|
||||
DATASET_NAME = {"xiiotid": "X-IIoTID", "5gnidd": "5G-NIDD"}
|
||||
DATASETS = ["xiiotid", "5gnidd"]
|
||||
ALPHAS = [0.25, 0.50, 0.75]
|
||||
BETAS = [0.0, 0.25, 0.5, 0.75, 1.0]
|
||||
SELECTED = {("xiiotid", 0.25): 0.0, ("xiiotid", 0.5): 0.75, ("xiiotid", 0.75): 0.5,
|
||||
("5gnidd", 0.25): 0.0, ("5gnidd", 0.5): 1.0, ("5gnidd", 0.75): 0.5}
|
||||
GREEN, RED, GREY = "#1B7F3B", "#B22222", "#6B6B6B"
|
||||
|
||||
|
||||
def _apply_style() -> None:
|
||||
plt.rcParams.update({
|
||||
"figure.facecolor": "white",
|
||||
"axes.facecolor": "white",
|
||||
"axes.edgecolor": "#5A5A5A",
|
||||
"axes.linewidth": 0.9,
|
||||
"axes.titlesize": 13,
|
||||
"axes.titleweight": "bold",
|
||||
"axes.labelsize": 11,
|
||||
"xtick.labelsize": 10,
|
||||
"ytick.labelsize": 10,
|
||||
"legend.fontsize": 9.5,
|
||||
"font.size": 10.5,
|
||||
"axes.spines.top": False,
|
||||
"axes.spines.right": False,
|
||||
"grid.color": "#B0B0B0",
|
||||
"grid.alpha": 0.35,
|
||||
"grid.linestyle": ":",
|
||||
"savefig.dpi": 220,
|
||||
"savefig.bbox": "tight",
|
||||
})
|
||||
|
||||
|
||||
def _save(fig, name: str) -> None:
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
for ext in ("png", "pdf"):
|
||||
fig.savefig(OUT / f"{name}.{ext}")
|
||||
plt.close(fig)
|
||||
print("wrote", OUT / f"{name}.png")
|
||||
|
||||
|
||||
def _read_csv(path: Path) -> list[dict]:
|
||||
with path.open() as fh:
|
||||
return list(csv.DictReader(fh))
|
||||
|
||||
|
||||
def _footnote(fig, text: str, y: float) -> None:
|
||||
"""Place a wrapped caption below the axes; bbox_inches='tight' expands."""
|
||||
fig.text(0.5, y, text, ha="center", va="top", fontsize=8.7,
|
||||
color="#444444", linespacing=1.5)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Figure A: pooled validation beta curves (support_weighted only)
|
||||
# ---------------------------------------------------------------------------
|
||||
def figure_a(dataset: str) -> None:
|
||||
# SOURCE: artifacts/analysis/final__pooled_validation_curves.csv
|
||||
rows = _read_csv(ANALYSIS / "final__pooled_validation_curves.csv")
|
||||
fig, ax = plt.subplots(figsize=(8.4, 5.4), constrained_layout=True)
|
||||
ax.set_axisbelow(True)
|
||||
ax.grid(axis="y")
|
||||
ymin, ymax = 1e9, -1e9
|
||||
|
||||
for alpha in ALPHAS:
|
||||
vals = {float(r["beta"]): r for r in rows
|
||||
if r["dataset"] == dataset and float(r["alpha"]) == alpha}
|
||||
x = np.array(BETAS)
|
||||
y = np.array([float(vals[b]["pooled_validation_mean"]) for b in BETAS])
|
||||
e = np.array([float(vals[b]["sample_std"]) for b in BETAS])
|
||||
ymin = min(ymin, float((y - e).min()))
|
||||
ymax = max(ymax, float((y + e).max()))
|
||||
st = ALPHA_STYLE[alpha]
|
||||
ax.errorbar(x, y, yerr=e, capsize=3, elinewidth=1.0, lw=1.8,
|
||||
color=st["color"], marker=st["marker"], ls=st["ls"],
|
||||
ms=6.5, mfc="white", mec=st["color"], mew=1.4,
|
||||
label=st["label"], zorder=3)
|
||||
sb = SELECTED[(dataset, alpha)]
|
||||
ax.scatter([sb], [float(vals[sb]["pooled_validation_mean"])], s=210,
|
||||
marker="*", facecolor=st["color"], edgecolor="white",
|
||||
linewidth=0.8, zorder=5)
|
||||
if sb == 0.0:
|
||||
ax.annotate(f"selected beta={sb:g}",
|
||||
(sb, float(vals[sb]["pooled_validation_mean"])),
|
||||
textcoords="offset points", xytext=(26, 9), ha="left",
|
||||
fontsize=8.5, color=st["color"], weight="bold")
|
||||
elif sb == 1.0:
|
||||
ax.annotate(f"selected beta={sb:g}",
|
||||
(sb, float(vals[sb]["pooled_validation_mean"])),
|
||||
textcoords="offset points", xytext=(-26, 9), ha="right",
|
||||
fontsize=8.5, color=st["color"], weight="bold")
|
||||
else:
|
||||
ax.annotate(f"selected beta={sb:g}",
|
||||
(sb, float(vals[sb]["pooled_validation_mean"])),
|
||||
textcoords="offset points", xytext=(0, 13), ha="center",
|
||||
fontsize=8.5, color=st["color"], weight="bold")
|
||||
|
||||
dname = DATASET_NAME[dataset]
|
||||
ax.set_xlabel("beta candidate (support_weighted aggregation)")
|
||||
ax.set_ylabel("pooled validation rare-class macro-F1\n(per-client mean)")
|
||||
ax.set_xticks(BETAS)
|
||||
ax.set_xticklabels([f"{b:.2f}" for b in BETAS])
|
||||
ax.set_ylim(ymin - 0.012, ymax + 0.10)
|
||||
ax.set_title(f"{dname}: validation-based beta selection")
|
||||
ax.legend(title="non-IID condition", loc="upper center", ncols=3,
|
||||
frameon=True, edgecolor="#CCCCCC", framealpha=0.95)
|
||||
_footnote(fig,
|
||||
"All curves are support_weighted configurations; no original_protean run is plotted here.\n"
|
||||
"Metric: pooled validation per-client-mean rare-class macro-F1, averaged over seeds {0,1,2};\n"
|
||||
"error bars = \u00b11 sample std across seeds. Star = argmax (selected beta) per condition; selection uses validation only.",
|
||||
y=-0.13)
|
||||
_save(fig, f"figA_{dataset}_pooled_validation_beta_curve")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Figure B: validation-selected beta by dataset and alpha
|
||||
# ---------------------------------------------------------------------------
|
||||
def figure_b() -> None:
|
||||
# SOURCE: artifacts/analysis/final__selection_strength.csv
|
||||
rows = _read_csv(ANALYSIS / "final__selection_strength.csv")
|
||||
sel = {(r["dataset"], float(r["alpha"])): float(r["selected_beta"]) for r in rows}
|
||||
|
||||
fig, ax = plt.subplots(figsize=(8.8, 5.2), constrained_layout=True)
|
||||
ax.set_axisbelow(True)
|
||||
ax.grid(axis="y")
|
||||
width = 0.24
|
||||
xbase = np.arange(len(DATASETS))
|
||||
for k, alpha in enumerate(ALPHAS):
|
||||
st = ALPHA_STYLE[alpha]
|
||||
xs = xbase + (k - 1) * width
|
||||
ys = [sel[(d, alpha)] for d in DATASETS]
|
||||
bars = ax.bar(xs, ys, width, color=st["color"], edgecolor="#333333",
|
||||
linewidth=0.7, label=st["label"], zorder=3)
|
||||
for b, v in zip(bars, ys):
|
||||
ax.annotate(f"{v:.2f}", (b.get_x() + b.get_width() / 2, v),
|
||||
textcoords="offset points", xytext=(0, 3),
|
||||
ha="center", fontsize=9, weight="bold")
|
||||
ax.set_xticks(xbase)
|
||||
ax.set_xticklabels([DATASET_NAME[d] for d in DATASETS])
|
||||
ax.set_ylabel("validation-selected beta")
|
||||
ax.set_ylim(0, 1.20)
|
||||
ax.set_title("Validation-selected beta by dataset and non-IID condition")
|
||||
ax.legend(title="Dirichlet alpha", loc="upper center", ncols=3,
|
||||
frameon=True, edgecolor="#CCCCCC")
|
||||
_footnote(fig,
|
||||
"Selection = best among the tested candidates {0, 0.25, 0.50, 0.75, 1.00} under pooled validation\n"
|
||||
"per-client-mean rare-class macro-F1 (support_weighted). This is a selection surface, not a performance\n"
|
||||
"comparison; a higher beta is not automatically better.",
|
||||
y=-0.13)
|
||||
_save(fig, "figB_selected_beta_by_condition")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Figure C: realized class-support imbalance (mean support CV)
|
||||
# ---------------------------------------------------------------------------
|
||||
def figure_c() -> None:
|
||||
# SOURCE: artifacts/analysis/final__realized_heterogeneity.csv
|
||||
rows = _read_csv(ANALYSIS / "final__realized_heterogeneity.csv")
|
||||
cv = {(r["dataset"], float(r["alpha"])): float(r["support_cv_mean"]) for r in rows}
|
||||
|
||||
fig, ax = plt.subplots(figsize=(8.8, 5.2), constrained_layout=True)
|
||||
ax.set_axisbelow(True)
|
||||
ax.grid(axis="y")
|
||||
width = 0.24
|
||||
xbase = np.arange(len(DATASETS))
|
||||
for k, alpha in enumerate(ALPHAS):
|
||||
st = ALPHA_STYLE[alpha]
|
||||
xs = xbase + (k - 1) * width
|
||||
ys = [cv[(d, alpha)] for d in DATASETS]
|
||||
bars = ax.bar(xs, ys, width, color=st["color"], edgecolor="#333333",
|
||||
linewidth=0.7, label=st["label"], zorder=3)
|
||||
for b, v in zip(bars, ys):
|
||||
ax.annotate(f"{v:.3f}", (b.get_x() + b.get_width() / 2, v),
|
||||
textcoords="offset points", xytext=(0, 3),
|
||||
ha="center", fontsize=9)
|
||||
ax.set_xticks(xbase)
|
||||
ax.set_xticklabels([DATASET_NAME[d] for d in DATASETS])
|
||||
ax.set_ylabel("mean class-support CV of N_ij")
|
||||
ax.set_ylim(0, max(cv.values()) * 1.20)
|
||||
ax.set_title("Realized class-support imbalance after Dirichlet partitioning")
|
||||
ax.legend(title="configured Dirichlet alpha", loc="upper right",
|
||||
frameon=True, edgecolor="#CCCCCC")
|
||||
_footnote(fig,
|
||||
"Coefficient of variation (std/mean) of N_ij across contributing clients, averaged over classes and seeds.\n"
|
||||
"Larger CV = stronger realized support imbalance. alpha is the configuration knob; this is the realized data\n"
|
||||
"property, not a performance comparison.",
|
||||
y=-0.13)
|
||||
_save(fig, "figC_realized_support_cv")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Figure D: prototype displacement beta=1 vs beta=0 control
|
||||
# ---------------------------------------------------------------------------
|
||||
def figure_d() -> None:
|
||||
# SOURCE: artifacts/analysis/final__prototype_displacement_by_beta.csv
|
||||
rows = _read_csv(ANALYSIS / "final__prototype_displacement_by_beta.csv")
|
||||
disp = {(r["dataset"], float(r["alpha"]), float(r["beta"])): float(r["mean_displacement"])
|
||||
for r in rows}
|
||||
|
||||
conditions = [(d, a) for d in DATASETS for a in ALPHAS]
|
||||
labels = [f"{DATASET_NAME[d]} \u03b1={a:.2f}" for d, a in conditions]
|
||||
vals = [disp[(d, a, 1.0)] for d, a in conditions]
|
||||
colors = [DATASET_STYLE[d]["color"] for d, _ in conditions]
|
||||
|
||||
fig, ax = plt.subplots(figsize=(8.8, 5.4), constrained_layout=True)
|
||||
ax.set_axisbelow(True)
|
||||
ax.grid(axis="x")
|
||||
ypos = np.arange(len(conditions))
|
||||
bars = ax.barh(ypos, vals, color=colors, edgecolor="#333333", linewidth=0.7, zorder=3)
|
||||
for b, v in zip(bars, vals):
|
||||
ax.annotate(f"{v:.3f}", (v, b.get_y() + b.get_height() / 2),
|
||||
textcoords="offset points", xytext=(4, 0), va="center",
|
||||
fontsize=9, weight="bold")
|
||||
ax.set_yticks(ypos)
|
||||
ax.set_yticklabels(labels)
|
||||
ax.invert_yaxis()
|
||||
ax.set_xlabel(r"mean $\||\bar{C}_j(\beta=1) - \bar{C}_j(\beta=0)\||$ across classes")
|
||||
ax.set_xlim(0, max(vals) * 1.18)
|
||||
ax.set_title("Support-weighted aggregation: beta=1 vs beta=0 control")
|
||||
handles = [Line2D([0], [0], color=DATASET_STYLE[d]["color"], lw=8,
|
||||
label=DATASET_NAME[d]) for d in DATASETS]
|
||||
ax.legend(handles=handles, title="dataset", loc="lower right",
|
||||
frameon=True, edgecolor="#CCCCCC")
|
||||
_footnote(fig,
|
||||
"Direct aggregation effect on fixed round-0 snapshots using identical local prototypes and N_ij; no retraining.\n"
|
||||
"beta=0 is the reference CONTROL (equal weighting over S_j) and is NOT automatically identical to original_protean.\n"
|
||||
"Displacement is a representation metric, not detection quality.",
|
||||
y=-0.11)
|
||||
_save(fig, "figD_prototype_displacement_beta0_vs_beta1")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Figure E: realized support CV vs prototype displacement
|
||||
# ---------------------------------------------------------------------------
|
||||
def figure_e() -> None:
|
||||
# SOURCE: final__realized_heterogeneity.csv + final__prototype_displacement_by_beta.csv
|
||||
het = _read_csv(ANALYSIS / "final__realized_heterogeneity.csv")
|
||||
disp = _read_csv(ANALYSIS / "final__prototype_displacement_by_beta.csv")
|
||||
cv = {(r["dataset"], float(r["alpha"])): float(r["support_cv_mean"]) for r in het}
|
||||
d1 = {(r["dataset"], float(r["alpha"])): float(r["mean_displacement"])
|
||||
for r in disp if float(r["beta"]) == 1.0}
|
||||
|
||||
fig, ax = plt.subplots(figsize=(8.2, 5.6), constrained_layout=True)
|
||||
ax.set_axisbelow(True)
|
||||
ax.grid(True)
|
||||
for d in DATASETS:
|
||||
st = DATASET_STYLE[d]
|
||||
xs = [cv[(d, a)] for a in ALPHAS]
|
||||
ys = [d1[(d, a)] for a in ALPHAS]
|
||||
ax.scatter(xs, ys, s=140, marker=st["marker"], color=st["color"],
|
||||
edgecolor="white", linewidth=1.2, label=st["label"], zorder=4)
|
||||
off = (7, -15) if d == "xiiotid" else (9, 6)
|
||||
for a, x, y in zip(ALPHAS, xs, ys):
|
||||
ax.annotate(f"\u03b1={a:.2f}", (x, y), textcoords="offset points",
|
||||
xytext=off, fontsize=8.8, color=st["color"])
|
||||
ax.set_xlabel("realized mean class-support CV of N_ij")
|
||||
ax.set_ylabel(r"mean prototype displacement $\||\bar{C}_j(1)-\bar{C}_j(0)\||$")
|
||||
ax.set_title("Realized support imbalance vs prototype displacement (beta=1 vs beta=0)")
|
||||
ax.legend(title="dataset", loc="upper left", frameon=True, edgecolor="#CCCCCC")
|
||||
_footnote(fig,
|
||||
"Each point is one (dataset, alpha) condition. Descriptive association only;\n"
|
||||
"the imbalance-displacement link is mechanical under the beta formula and this plot is not causal evidence.",
|
||||
y=-0.13)
|
||||
_save(fig, "figE_support_cv_vs_displacement")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Figure F: selected-beta support_weighted vs ORIGINAL PROTEAN (test)
|
||||
# ---------------------------------------------------------------------------
|
||||
def figure_f() -> None:
|
||||
# SOURCE: artifacts/analysis/final__six_condition_master.csv
|
||||
rows = _read_csv(ANALYSIS / "final__six_condition_master.csv")
|
||||
master = {(r["dataset"], float(r["alpha"])): r for r in rows}
|
||||
conditions = [(d, a) for d in DATASETS for a in ALPHAS]
|
||||
labels = [f"{DATASET_NAME[d]}\n\u03b1={a:.2f}" for d, a in conditions]
|
||||
|
||||
orig = np.array([float(master[c]["original_rare_macro_f1_mean"]) for c in conditions])
|
||||
orig_e = np.array([float(master[c]["original_rare_macro_f1_std"]) for c in conditions])
|
||||
sel = np.array([float(master[c]["selected_rare_macro_f1_mean"]) for c in conditions])
|
||||
sel_e = np.array([float(master[c]["selected_rare_macro_f1_std"]) for c in conditions])
|
||||
betas = [float(master[c]["selected_beta"]) for c in conditions]
|
||||
delta = sel - orig
|
||||
|
||||
fig, ax = plt.subplots(figsize=(11.4, 6.0), constrained_layout=True)
|
||||
ax.set_axisbelow(True)
|
||||
ax.grid(axis="y")
|
||||
x = np.arange(len(conditions))
|
||||
w = 0.36
|
||||
ax.bar(x - w / 2, orig, w, yerr=orig_e, capsize=3.5, color=CHARCOAL,
|
||||
edgecolor="#1A1A1A", linewidth=0.7, label="original_protean", zorder=3)
|
||||
ax.bar(x + w / 2, sel, w, yerr=sel_e, capsize=3.5, color=TEAL,
|
||||
edgecolor="#0A4F4E", linewidth=0.7, hatch="///",
|
||||
label="support_weighted (validation-selected beta)", zorder=3)
|
||||
|
||||
for xi, db, sb in zip(x, delta, betas):
|
||||
if sb == 0.0:
|
||||
colour = GREY
|
||||
elif db > 0:
|
||||
colour = GREEN
|
||||
else:
|
||||
colour = RED
|
||||
sign = "+" if db >= 0 else "\u2212"
|
||||
top = max(orig[xi] + orig_e[xi], sel[xi] + sel_e[xi])
|
||||
ax.annotate(f"{sign}{abs(db):.4f}", (xi, top + 0.012), ha="center",
|
||||
fontsize=9.5, weight="bold", color=colour)
|
||||
ax.annotate(f"beta={sb:g}", (xi, top + 0.030), ha="center", fontsize=8.5,
|
||||
color="#444444")
|
||||
ax.set_xticks(x)
|
||||
ax.set_xticklabels(labels)
|
||||
ax.set_ylabel("test rare-class macro-F1\n(mean over seeds {0,1,2})")
|
||||
ax.set_ylim(0.5, max(max(orig + orig_e), max(sel + sel_e)) + 0.10)
|
||||
ax.set_title("Test rare-class macro-F1: original PROTEAN vs validation-selected beta")
|
||||
ax.legend(loc="upper left", frameon=True, edgecolor="#CCCCCC")
|
||||
_footnote(fig,
|
||||
"Bar height = mean test rare-class macro-F1 over seeds {0,1,2}; error bars = \u00b11 sample std across seeds.\n"
|
||||
"Delta (top) = selected-beta minus original_protean; green = positive, red = negative, grey = selected beta 0\n"
|
||||
"(floating-point ~0). No statistical significance test is applied (3 seeds). Gains are condition-dependent;\n"
|
||||
"the proposed weighting does NOT consistently outperform original PROTEAN.",
|
||||
y=-0.13)
|
||||
_save(fig, "figF_selected_vs_original_test_rare_macro_f1")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Figure G: per-client effect of selected beta relative to beta=0 control
|
||||
# ---------------------------------------------------------------------------
|
||||
def figure_g() -> None:
|
||||
# SOURCE: artifacts/analysis/final__client_effect_summary.csv
|
||||
rows = _read_csv(ANALYSIS / "final__client_effect_summary.csv")
|
||||
rec = {(r["dataset"], float(r["alpha"]), int(r["seed"]), float(r["beta"])): r
|
||||
for r in rows}
|
||||
imp_col, deg_col, unc_col = "#2E8B57", "#C0392B", "#BFBFBF"
|
||||
|
||||
fig, axes = plt.subplots(1, 2, figsize=(12.6, 5.4), constrained_layout=True,
|
||||
sharey=True)
|
||||
for ax, d in zip(axes, DATASETS):
|
||||
ax.set_axisbelow(True)
|
||||
ax.grid(axis="y")
|
||||
xticks, xticklabels = [], []
|
||||
pos = 0.0
|
||||
for a in ALPHAS:
|
||||
for s in (0, 1, 2):
|
||||
r = rec[(d, a, s, SELECTED[(d, a)])]
|
||||
imp, deg, unc = (int(r["n_improve_vs_beta0"]),
|
||||
int(r["n_degrade_vs_beta0"]),
|
||||
int(r["n_unchanged"]))
|
||||
ax.bar([pos], [imp], width=0.82, color=imp_col, edgecolor="#1F5C39",
|
||||
linewidth=0.6, hatch="///", zorder=3)
|
||||
ax.bar([pos], [deg], bottom=[imp], width=0.82, color=deg_col,
|
||||
edgecolor="#7A1F14", linewidth=0.6, hatch="\\\\\\", zorder=3)
|
||||
ax.bar([pos], [unc], bottom=[imp + deg], width=0.82, color=unc_col,
|
||||
edgecolor="#7A7A7A", linewidth=0.6, hatch="..", zorder=3)
|
||||
xticks.append(pos)
|
||||
xticklabels.append(f"seed {s}")
|
||||
pos += 1
|
||||
pos += 0.8
|
||||
ax.set_xticks(xticks)
|
||||
ax.set_xticklabels(xticklabels, fontsize=8.2)
|
||||
for k, a in enumerate(ALPHAS):
|
||||
center = k * 3 + 1 + k * 0.8
|
||||
ax.text(center, 10.8, f"\u03b1={a:.2f}", ha="center", fontsize=10,
|
||||
weight="bold")
|
||||
if k < len(ALPHAS) - 1:
|
||||
ax.axvline(k * 3 + 2.5 + k * 0.8 + 0.4, color="#DDDDDD", lw=0.8)
|
||||
ax.set_ylim(0, 11.9)
|
||||
ax.set_yticks([0, 2, 4, 6, 8, 10])
|
||||
ax.set_title(DATASET_NAME[d])
|
||||
ax.set_xlabel("seed within each alpha condition")
|
||||
|
||||
axes[0].set_ylabel("clients (out of 10)")
|
||||
handles = [plt.Rectangle((0, 0), 1, 1, facecolor=imp_col, edgecolor="#1F5C39", hatch="///"),
|
||||
plt.Rectangle((0, 0), 1, 1, facecolor=deg_col, edgecolor="#7A1F14", hatch="\\\\\\"),
|
||||
plt.Rectangle((0, 0), 1, 1, facecolor=unc_col, edgecolor="#7A7A7A", hatch="..")]
|
||||
fig.legend(handles, ["improved", "degraded", "unchanged"], title="vs beta=0 control",
|
||||
loc="upper center", bbox_to_anchor=(0.5, 1.04), ncols=3,
|
||||
frameon=True, edgecolor="#CCCCCC")
|
||||
fig.suptitle("Per-client effect of the validation-selected beta relative to the beta=0 control (validation split)",
|
||||
fontsize=12.5, weight="bold", y=1.12)
|
||||
_footnote(fig,
|
||||
"Comparison is the selected-beta support_weighted run vs the support_weighted beta=0 run (CONTROL), using per-client validation\n"
|
||||
"rare-class macro-F1; it is NOT a comparison against original_protean. At both alpha=0.25 conditions the selected beta is 0,\n"
|
||||
"so all clients are unchanged. Client ids are per-seed realizations and are not pooled.",
|
||||
y=-0.11)
|
||||
_save(fig, "figG_client_effect_at_selected_beta")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Supplemental Figure H: paired test confusion matrices
|
||||
# ---------------------------------------------------------------------------
|
||||
def figure_h(dataset="5gnidd", alpha=0.75, seed=0, selected_beta=0.5) -> bool:
|
||||
astr = f"{alpha:g}"
|
||||
runs = {
|
||||
"original_protean": f"{dataset}__alpha{astr}__seed{seed}__original_protean",
|
||||
"selected_beta": f"{dataset}__alpha{astr}__seed{seed}__support_weighted__beta{selected_beta:g}",
|
||||
}
|
||||
paths = {k: RESULTS / v / "test_confusion_matrix.json" for k, v in runs.items()}
|
||||
meta_path = RESULTS / runs["original_protean"] / "metadata.json"
|
||||
if not all(p.exists() for p in paths.values()) or not meta_path.exists():
|
||||
print("figH skipped: missing inputs")
|
||||
return False
|
||||
|
||||
meta = json.loads(meta_path.read_text())
|
||||
meta_sel = json.loads((RESULTS / runs["selected_beta"] / "metadata.json").read_text())
|
||||
same_partition = (meta["dataset"] == meta_sel["dataset"] == dataset
|
||||
and abs(float(meta["alpha"]) - float(meta_sel["alpha"])) < 1e-9
|
||||
and int(meta["seed"]) == int(meta_sel["seed"]) == seed
|
||||
and meta["manifest_content_hash"] == meta_sel["manifest_content_hash"]
|
||||
and abs(float(meta_sel["beta"]) - selected_beta) < 1e-9)
|
||||
if not same_partition or selected_beta != SELECTED[(dataset, alpha)]:
|
||||
print("figH skipped: paired-condition verification failed")
|
||||
return False
|
||||
|
||||
class_names = meta["class_names"]
|
||||
mats = {k: np.array(json.loads(p.read_text())) for k, p in paths.items()}
|
||||
|
||||
from matplotlib.colors import LogNorm
|
||||
vmax = max(m.max() for m in mats.values())
|
||||
vmin = 1
|
||||
fig, axes = plt.subplots(1, 2, figsize=(13.2, 6.2), constrained_layout=True)
|
||||
titles = {
|
||||
"original_protean": "ORIGINAL PROTEAN",
|
||||
"selected_beta": f"SELECTED-BETA PROTEAN (beta={selected_beta:g})",
|
||||
}
|
||||
im = None
|
||||
for ax, key in zip(axes, ["original_protean", "selected_beta"]):
|
||||
m = mats[key]
|
||||
im = ax.imshow(m, cmap="YlGnBu", norm=LogNorm(vmin=vmin, vmax=vmax),
|
||||
aspect="equal")
|
||||
ax.set_xticks(range(len(class_names)))
|
||||
ax.set_yticks(range(len(class_names)))
|
||||
ax.set_xticklabels(class_names, rotation=45, ha="right", fontsize=8.2)
|
||||
ax.set_yticklabels(class_names, fontsize=8.2)
|
||||
ax.set_xlabel("predicted class")
|
||||
ax.set_title(titles[key], fontsize=11)
|
||||
thr = m.max() * 0.06
|
||||
for i in range(m.shape[0]):
|
||||
for j in range(m.shape[1]):
|
||||
v = int(m[i, j])
|
||||
if v == 0:
|
||||
continue
|
||||
txt = f"{v/1000:.1f}k" if v >= 10000 else str(v)
|
||||
ax.text(j, i, txt, ha="center", va="center", fontsize=6.4,
|
||||
color="white" if m[i, j] > thr else "#333333")
|
||||
axes[0].set_ylabel("true class")
|
||||
cbar = fig.colorbar(im, ax=axes, fraction=0.025, pad=0.02)
|
||||
cbar.set_label("test samples (raw counts, log colour scale)")
|
||||
dname = DATASET_NAME.get(dataset, dataset)
|
||||
fig.suptitle(
|
||||
f"Paired test confusion matrices - {dname}, alpha={alpha:g}, seed={seed} "
|
||||
f"(same manifest, split, seed, class order)",
|
||||
fontsize=11.5, weight="bold", y=1.05)
|
||||
_footnote(fig,
|
||||
"Left = original_protean; right = support_weighted at the validation-selected beta. Raw counts; identical class order and colour scale.\n"
|
||||
"Dominant off-diagonal error: Benign predicted as UDPFlood, essentially unchanged by beta. No statistical significance is inferred.",
|
||||
y=-0.20)
|
||||
_save(fig, f"figH_confusion_matrix_{dataset}_a{alpha:g}_seed{seed}")
|
||||
return True
|
||||
|
||||
|
||||
def main() -> None:
|
||||
_apply_style()
|
||||
for d in DATASETS:
|
||||
figure_a(d)
|
||||
figure_b()
|
||||
figure_c()
|
||||
figure_d()
|
||||
figure_e()
|
||||
figure_f()
|
||||
figure_g()
|
||||
figure_h()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||