docs: improve readability of PROTEAN comparison figures
This commit is contained in:
willtanoe committed
1 parent
ad8b2d7671
commit
34bab45865
22 files changed
+398
-212
No files matched your search
@@ -473,8 +473,10 @@ Representative confusion-matrix files:
|
||||
## 16. Final Figures and Visualizations
|
||||
|
||||
All figures live in `artifacts/figures/final/` and are regenerated from the audited
|
||||
`artifacts/analysis/final__*` tables by `scripts/make_final_figures.py` (no training; every number is
|
||||
loaded from an artifact). Each figure is saved as a high-resolution PNG (220 dpi) and a vector PDF.
|
||||
`artifacts/analysis/final__*` tables by `scripts/make_final_figures.py` and
|
||||
`scripts/make_comparison_figures.py` (no training; every number is loaded from an artifact). Each
|
||||
figure is saved as a high-resolution PNG (220 dpi) **and** a matching vector PDF, and its supporting
|
||||
source table is linked where useful.
|
||||
|
||||
**Naming and legend conventions used throughout.**
|
||||
|
||||
@@ -494,9 +496,12 @@ loaded from an artifact). Each figure is saved as a high-resolution PNG (220 dpi
|
||||
| B | Beta selection surface | none (selection, not performance) |
|
||||
| C | Dataset heterogeneity | none (data distribution) |
|
||||
| D, E | Fixed-snapshot comparison vs `beta = 0` | `beta = 0` control |
|
||||
| F | Direct comparison vs `original_protean` | `original_protean` |
|
||||
| G | Per-client comparison vs `beta = 0` | `beta = 0` control |
|
||||
| H | Paired `original_protean` vs selected beta | `original_protean` |
|
||||
| F | Direct comparison vs `original_protean` (test) | `original_protean` |
|
||||
| G | Per-client comparison vs `beta = 0` (validation) | `beta = 0` control |
|
||||
| H | Paired `original_protean` vs selected beta confusion matrices | `original_protean` |
|
||||
| I1, I2 | Per-client test comparison vs `original_protean` | `original_protean` |
|
||||
| J1, J2 | Per-class test comparison vs `original_protean` | `original_protean` |
|
||||
| K | Ranked confusion change vs `original_protean` (+ full matrix supplement) | `original_protean` |
|
||||
|
||||
### Fig A1 / A2 — Validation-based beta selection (support_weighted)
|
||||
|
||||
@@ -548,18 +553,23 @@ the imbalance-displacement link is mechanical under the beta formula and is not
|
||||
|
||||
|
||||
|
||||
### Fig F — Selected-beta PROTEAN vs ORIGINAL PROTEAN (test)
|
||||
### Fig F — Selected-beta PROTEAN vs ORIGINAL PROTEAN (test), two panels
|
||||
|
||||
**Compared:** `original_protean` (charcoal) against `support_weighted` at the validation-selected
|
||||
beta (teal), per condition. **Level/split:** test, mean over seeds `{0,1,2}`. **Metric:** test
|
||||
rare-class macro-F1; error bars = ±1 sample std across seeds. **Observation:** signed deltas are
|
||||
`+0.0230`, `+0.0014`, `+0.0168` (positive), `−0.0086` (negative), and `−0.0007`/`−0.0010` at the two
|
||||
`beta = 0` conditions (floating-point ~0). **Limits:** no statistical significance test (three
|
||||
seeds); the largest gain is within seed noise; the weighting does not consistently outperform
|
||||
`original_protean`.
|
||||
**Compared:** `original_protean` against `support_weighted` at the validation-selected beta, per
|
||||
condition, test split. **How to read:** Panel A shows the absolute mean test rare-class macro-F1
|
||||
(±1 sample std across seeds) and is deliberately near-flat; Panel B shows the signed difference
|
||||
`proposed − original` with the exact value printed at each bar and the selected beta annotated.
|
||||
**Main result:** three positive deltas (X-IIoTID a0.50 +0.0230, X-IIoTID a0.75 +0.0014, 5G-NIDD a0.75
|
||||
+0.0168), one negative delta (5G-NIDD a0.50 −0.0086), and two `beta=0` conditions that are
|
||||
effectively unchanged (−0.0007 / −0.0010). **Limitation:** no significance test (three seeds); a
|
||||
positive mean delta is not a proven improvement.
|
||||
|
||||
|
||||
|
||||
Links: [PNG](artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.png) ·
|
||||
[PDF](artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.pdf) · source
|
||||
[`final__six_condition_master.csv`](artifacts/analysis/final__six_condition_master.csv).
|
||||
|
||||
### Fig G — Per-client effect of selected beta relative to the beta=0 control
|
||||
|
||||
**Compared:** selected-beta `support_weighted` against the `beta = 0` control (not
|
||||
@@ -571,17 +581,25 @@ client ids are per-seed realizations and are not pooled.
|
||||
|
||||
|
||||
|
||||
### Fig H (supplemental) — Paired test confusion matrices
|
||||
### Fig H — Paired test confusion matrices (original vs beta-tuned)
|
||||
|
||||
**Compared:** `original_protean` (left) against selected-beta `support_weighted` (right) for a valid
|
||||
paired condition (5G-NIDD, `alpha = 0.75`, `seed = 0`). **Level/split:** test; global counts over the
|
||||
evaluation split. **Metric:** raw 9×9 counts (log colour scale), identical class order and colour
|
||||
scale. **Observation:** the dominant off-diagonal error is `Benign → UDPFlood` (~56.9k vs ~57.1k) and
|
||||
is essentially unchanged by beta. **Limits:** no statistical significance is inferred from the visual
|
||||
difference; the matrices come from the same manifest, split, seed, and class order.
|
||||
paired condition (5G-NIDD, `alpha = 0.75`, `seed = 0`), test split. **How to read:** rows = actual
|
||||
class, columns = predicted class; the diagonal is correct and off-diagonal is misclassified. Raw
|
||||
counts, identical class order and colour scale; both matrices contain the same 243,177 test samples.
|
||||
**Main result:** the dominant residual error `Benign → UDPFlood` is essentially unchanged (56,899 →
|
||||
57,054, +155). The summary below the figure lists the most important changes (`HTTPFlood →
|
||||
SlowrateDoS` −547; `Benign → TCPConnectScan` −520; `SlowrateDoS → HTTPFlood` +229; diagonal
|
||||
`HTTPFlood` +547, `Benign` +288, `SlowrateDoS` −239). **Limitation:** no significance test; the
|
||||
`Benign → UDPFlood` increase is **not** an improvement.
|
||||
|
||||
|
||||
|
||||
Links: [PNG](artifacts/figures/final/figH_confusion_matrix_5gnidd_a0.75_seed0.png) ·
|
||||
[PDF](artifacts/figures/final/figH_confusion_matrix_5gnidd_a0.75_seed0.pdf) · source
|
||||
`results/5gnidd__alpha0.75__seed0__original_protean/test_confusion_matrix.json` and
|
||||
`results/5gnidd__alpha0.75__seed0__support_weighted__beta0.5/test_confusion_matrix.json`.
|
||||
|
||||
|
||||
## 17. Deep Comparison: Original PROTEAN vs Proposed (Result)
|
||||
|
||||
@@ -609,34 +627,71 @@ Both `alpha=0.25` conditions selected `beta=0` and are numerically null. Among t
|
||||
with it, while rare-class recall usually falls (precision–recall trade-off). 5G-NIDD a0.50 is
|
||||
seed-dominated. No statistical significance is claimed.
|
||||
|
||||
### Fig I — Paired per-client test rare-class macro-F1 (proposed vs ORIGINAL PROTEAN)
|
||||
### Fig I1 / I2 — Paired per-client test rare-class macro-F1 (proposed vs ORIGINAL PROTEAN)
|
||||
|
||||
Each point is one (client, seed) test observation, faceted by dataset and alpha; charcoal/teal/red/gray
|
||||
and markers encode outcome and seed. At `beta>0` conditions the response is client-specific (e.g.
|
||||
X-IIoTID a0.50: 12/30 improved, 1 degraded, 17 unchanged), with individual changes larger than the
|
||||
condition mean; at `beta=0` conditions almost all observations are unchanged.
|
||||
One figure per dataset, three alpha panels. **Compared:** `original_protean` against selected-beta
|
||||
`support_weighted` on the test split, each client on its own evaluation partition. **How to read:**
|
||||
x = client id, one marker per seed (circle = seed 0, square = seed 1, triangle = seed 2),
|
||||
y = Δ rare-class macro-F1; the shaded band is the ±0.02 effectively-unchanged threshold; the strongest
|
||||
positive and negative observation in each panel is labelled `c<client> s<seed>`. **Main result:**
|
||||
X-IIoTID α=0.25 essentially unchanged (1 / 1 / 28 improved/degraded/unchanged), α=0.50 12 / 1 / 17
|
||||
(mean +0.030, range −0.257 … +0.324), α=0.75 8 / 2 / 20; 5G-NIDD α=0.25 1 / 0 / 29, α=0.50 15 / 6 / 9
|
||||
(seed-dominated), α=0.75 8 / 1 / 21. **Limitation:** client ids are per-seed realizations (id `k` is
|
||||
not the same entity across seeds); at α=0.25 the selected beta is 0 so the residual spread is
|
||||
numerical, not a weighting effect.
|
||||
|
||||
|
||||
|
||||
|
||||
### Fig J — Per-class test F1 change (beta>0 conditions)
|
||||
|
||||
|
||||
Signed Δ F1 per class, panels per dataset-alpha; class support is annotated. Largest gains are
|
||||
X-IIoTID `Tampering` (+0.14) and `Reconnaissance` (+0.05) and 5G-NIDD `TCPConnectScan` (+0.03) /
|
||||
`ICMPFlood` (+0.10, tiny support); largest losses are X-IIoTID `C&C` (−0.06) and 5G-NIDD `SYNFlood`
|
||||
(−0.04). Benign detection is essentially unchanged.
|
||||
Links: [I1 PNG](artifacts/figures/final/figI1_xiiotid_per_client_original_vs_proposed.png) ·
|
||||
[I1 PDF](artifacts/figures/final/figI1_xiiotid_per_client_original_vs_proposed.pdf) ·
|
||||
[I2 PNG](artifacts/figures/final/figI2_5gnidd_per_client_original_vs_proposed.png) ·
|
||||
[I2 PDF](artifacts/figures/final/figI2_5gnidd_per_client_original_vs_proposed.pdf) · source
|
||||
[`deepcmp__per_client.csv`](artifacts/analysis/deepcmp__per_client.csv).
|
||||
|
||||
|
||||
### Fig J1 / J2 — Per-class test F1 change (beta>0 conditions)
|
||||
|
||||
### Fig K — Confusion-matrix difference (proposed − original)
|
||||
One figure per dataset, two alpha panels. **Compared:** `original_protean` against selected-beta
|
||||
`support_weighted`, test split. **How to read:** horizontal bars are the mean ΔF1 over seeds `{0,1,2}`
|
||||
(teal = increased, red = decreased) with the exact signed ΔF1 printed at each bar; test support `n≈`
|
||||
is on a second label line; both panels share one x-scale so bar lengths are comparable. **Main
|
||||
result:** X-IIoTID largest gains `Tampering` +0.1438 and `Reconnaissance` +0.0542, largest loss `C&C`
|
||||
−0.0556; 5G-NIDD largest gain `TCPConnectScan` +0.0342 / `ICMPFlood` +0.0970 (tiny support), largest
|
||||
loss `SYNFlood` −0.0426. **Limitation:** the delta is **F1**, not precision or recall (e.g. Tampering
|
||||
+0.1438 F1 corresponds to +0.2120 precision and −0.0091 recall); no significance test.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Links: [J1 PNG](artifacts/figures/final/figJ1_xiiotid_per_class_original_vs_proposed.png) ·
|
||||
[J1 PDF](artifacts/figures/final/figJ1_xiiotid_per_class_original_vs_proposed.pdf) ·
|
||||
[J2 PNG](artifacts/figures/final/figJ2_5gnidd_per_class_original_vs_proposed.png) ·
|
||||
[J2 PDF](artifacts/figures/final/figJ2_5gnidd_per_class_original_vs_proposed.pdf) · source
|
||||
[`deepcmp__per_class.csv`](artifacts/analysis/deepcmp__per_class.csv).
|
||||
|
||||
### Fig K — Ranked confusion change vs ORIGINAL PROTEAN (+ full-matrix supplement)
|
||||
|
||||
For the paired condition 5G-NIDD, `alpha=0.75`, `seed=0` (identical manifest, split, class order,
|
||||
243,177 test samples): left = raw count difference, right = row-normalized rate difference. The
|
||||
dominant error `Benign → UDPFlood` is essentially unchanged (56,899 → 57,054, +155). The main reduced
|
||||
errors are `HTTPFlood → SlowrateDoS` (−547) and `Benign → TCPConnectScan` (−520); the main increased
|
||||
error is `SlowrateDoS → HTTPFlood` (+229). Red = positive delta, blue = negative; no significance is
|
||||
inferred.
|
||||
243,177 test samples). **How to read:** the ranked view lists the largest off-diagonal changes as raw
|
||||
test counts; the upper panel shows *more* errors after weighting, the lower panel *fewer*.
|
||||
**Main result:** the largest reduced errors are `HTTPFlood → SlowrateDoS` (−547; 759 → 212) and
|
||||
`Benign → TCPConnectScan` (−520; 2,015 → 1,495); the largest increased errors are `SlowrateDoS →
|
||||
HTTPFlood` (+229) and `Benign → UDPFlood` (+155). The dominant error `Benign → UDPFlood` is essentially
|
||||
unchanged and is **not** an improvement. **Limitation:** no significance test; the figure is a single
|
||||
paired condition, not a condition average. The full difference matrices (raw count and row-rate) are
|
||||
kept as a supplemental figure for traceability.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Links: [ranked PNG](artifacts/figures/final/figK_confusion_error_changes_5gnidd_a0.75_seed0.png) ·
|
||||
[ranked PDF](artifacts/figures/final/figK_confusion_error_changes_5gnidd_a0.75_seed0.pdf) ·
|
||||
[full-matrix PNG](artifacts/figures/final/figK_confusion_delta_5gnidd_a0.75_seed0.png) ·
|
||||
[full-matrix PDF](artifacts/figures/final/figK_confusion_delta_5gnidd_a0.75_seed0.pdf) · source
|
||||
[`deepcmp__confusion_delta.csv`](artifacts/analysis/deepcmp__confusion_delta.csv).
|
||||
|
||||
**Per-client findings.** The proposed method redistributes benefit: within a condition some clients
|
||||
improve and others degrade, and the low-rare-support quartile is not systematically disadvantaged
|
||||
@@ -792,7 +847,8 @@ Analysis outputs (`artifacts/analysis/`), by function:
|
||||
| Prototype snapshots & weights | `*__prototype_snapshot.json`, `*__prototype_weights.csv`, `*__prototype_effect*.csv` |
|
||||
|
||||
Partition manifests: `artifacts/partitions/<dataset>__alpha<alpha>__seed<seed>.json` (18 files).
|
||||
Figures: `artifacts/figures/final/` — 12 figures (figA–figK) as PNG (220 dpi) plus matching vector PDF;
|
||||
Figures: `artifacts/figures/final/` — 15 figures (figA–figK, including the split I1/I2 and J1/J2 and
|
||||
the K full-matrix supplement) as PNG (220 dpi) plus matching vector PDF;
|
||||
regenerated reproducibly by `scripts/make_final_figures.py`.
|
||||
|
||||
Key documents:
|
||||
|
||||
Reference in new issue
Block a user