research: finalize experimental synthesis
This commit is contained in:
willtanoe committed
1 parent
d30e9831e6
commit
8a571a2387
21 files changed
+1325
-69
No files matched your search
@@ -0,0 +1,491 @@
|
||||
# Final Experimental Synthesis
|
||||
|
||||
Session 12 synthesis of the complete PROTEAN support-weighted prototype aggregation study.
|
||||
This document is **read-only with respect to experiments**: every number is loaded from the
|
||||
audited machine-readable artifacts under `artifacts/analysis/`. No training was run, no
|
||||
partition was regenerated, no source/model/aggregation code was modified, and no beta was
|
||||
re-selected from test metrics.
|
||||
|
||||
- Repository HEAD at synthesis: `d30e9831e6c17fc6b28383ae4592070ccd83657e` (branch `main`, in sync with `origin/main`).
|
||||
- Experimental grid: **2 datasets × 3 alpha levels × 3 seeds × (5 betas + original baseline) = 108 full runs** (plus one unrelated `smoke` directory, excluded).
|
||||
- Tests: `75 passed, 0 failed, 0 skipped`.
|
||||
- Beta candidates (frozen): `{0.00, 0.25, 0.50, 0.75, 1.00}`.
|
||||
- Selection rule (frozen): maximize **pooled validation per-client-mean rare-class macro-F1**, validation only; per-seed selection is diagnostic only; test is never used for selection.
|
||||
- `planningawal/` and both raw datasets unchanged (re-verified).
|
||||
|
||||
Terminology used throughout:
|
||||
|
||||
- **Measured**: directly read or recomputed from an audited artifact.
|
||||
- **Descriptive interpretation**: a neutral reading of measured values.
|
||||
- **Unsupported / not established**: a claim that the design cannot defend (causal, universal, superiority).
|
||||
|
||||
---
|
||||
|
||||
## 1. Scope and protocol
|
||||
|
||||
The thesis modifies **only** the server-side prototype aggregation of PROTEAN (Chennoufi et al.,
|
||||
2025). Each contributing client `i` provides a class-`j` local prototype `C_ij` (local class
|
||||
embedding mean) and a class-specific training support `N_ij`. The candidate aggregation is
|
||||
|
||||
```
|
||||
Cbar_j(beta) = ( sum_{i in S_j} N_ij^beta * C_ij ) / ( sum_{i in S_j} N_ij^beta )
|
||||
S_j = { i | N_ij > 0 and a valid local prototype C_ij exists }
|
||||
```
|
||||
|
||||
`S_j` is an explicit **reproduction assumption** (no verified official PROTEAN code is
|
||||
available); `N_ij` is class-specific **sample support, not prototype quality**. The frozen
|
||||
`original_protean` mode uses the equal-mean reference. `beta=0` is arithmetically the equal mean
|
||||
over the shared `S_j` but is **not bitwise identical** to the baseline (floating-point
|
||||
divergence; equivalence class **B** in every condition).
|
||||
|
||||
Protocol (frozen, `docs/research/EXPERIMENT_PROTOCOL.md`): datasets X-IIoTID (`class2`,
|
||||
benign `Normal`, K=9) and 5G-NIDD (`Attack Type`, benign `Benign`, K=9); 10 clients; symmetric
|
||||
Dirichlet non-IID with `alpha ∈ {0.25, 0.50, 0.75}`; seeds `{0,1,2}`; stratified train/validation/test
|
||||
= 70/10/20 (validation is a project decision; the paper uses 80/20); 10 rounds; 3 local epochs;
|
||||
batch 256; Adam; `lr=1e-3`, `mu=0.1`, `lambda=1.0` (project decisions). All compared configurations
|
||||
within a `(dataset, alpha, seed)` reuse exactly the same manifest, split and initialization
|
||||
(verified by content and partition hashes).
|
||||
|
||||
---
|
||||
|
||||
## 2. Completed experimental grid
|
||||
|
||||
| dataset | alpha | seeds | betas + baseline | runs | audit |
|
||||
|---|---|---|---|---|---|
|
||||
| X-IIoTID | 0.25 | 0,1,2 | 5 + 1 | 18 | PASS WITH MINOR CORRECTIONS (Session 7) |
|
||||
| X-IIoTID | 0.50 | 0,1,2 | 5 + 1 | 18 | PASS WITH MINOR CORRECTIONS (Session 6) |
|
||||
| X-IIoTID | 0.75 | 0,1,2 | 5 + 1 | 18 | PASS (Session 8) |
|
||||
| 5G-NIDD | 0.25 | 0,1,2 | 5 + 1 | 18 | PASS (Session 11) |
|
||||
| 5G-NIDD | 0.50 | 0,1,2 | 5 + 1 | 18 | PASS (Session 9) |
|
||||
| 5G-NIDD | 0.75 | 0,1,2 | 5 + 1 | 18 | PASS (Session 10) |
|
||||
| **total** | | | | **108** | |
|
||||
|
||||
---
|
||||
|
||||
## 3. Six-condition master result
|
||||
|
||||
Authoritative table: `artifacts/analysis/final__six_condition_master.csv` / `.json`.
|
||||
|
||||
| dataset | alpha | beta* | pooled val (std) | sel test rare-F1 | orig test rare-F1 | sel rare-recall | orig rare-recall | sel macro-F1 | sel acc | sel wF1 |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| X-IIoTID | 0.25 | **0.00** | 0.5917 (0.0895) | 0.6321 | 0.6328 | 0.8213 | 0.8220 | 0.6813 | 0.8740 | 0.9069 |
|
||||
| X-IIoTID | 0.50 | **0.75** | 0.5794 (0.0751) | 0.6655 | 0.6425 | 0.8626 | 0.8770 | 0.7312 | 0.9064 | 0.9336 |
|
||||
| X-IIoTID | 0.75 | **0.50** | 0.4767 (0.0466) | 0.6052 | 0.6039 | 0.8488 | 0.8565 | 0.7576 | 0.9281 | 0.9452 |
|
||||
| 5G-NIDD | 0.25 | **0.00** | 0.7615 (0.0219) | 0.8412 | 0.8423 | 0.9612 | 0.9606 | 0.8166 | 0.7302 | 0.7045 |
|
||||
| 5G-NIDD | 0.50 | **1.00** | 0.6924 (0.1391) | 0.8720 | 0.8805 | 0.9671 | 0.9729 | 0.8386 | 0.7433 | 0.7182 |
|
||||
| 5G-NIDD | 0.75 | **0.50** | 0.8091 (0.0587) | 0.8967 | 0.8799 | 0.9597 | 0.9612 | 0.8531 | 0.7523 | 0.7275 |
|
||||
|
||||
All selected betas are validation-only; `test_used_for_selection = false` in every run;
|
||||
`beta0_equivalence_class = B` in every condition. The selected beta takes values
|
||||
`{0.00, 0.50, 1.00}` across the six conditions: **not universal, not monotone, not a function of
|
||||
alpha alone**.
|
||||
|
||||
Selected-beta test metrics are **post-selection descriptive summaries**, not selection inputs.
|
||||
|
||||
---
|
||||
|
||||
## 4. Realized non-IID characteristics
|
||||
|
||||
Dirichlet `alpha` is *not* a realized-heterogeneity measure. Authoritative realized table:
|
||||
`artifacts/analysis/final__realized_heterogeneity.csv`.
|
||||
|
||||
| dataset | alpha | `|S_j|<M` / 27 | missing client-class cells | client-size ratio (mean / max) | mean support CV | max support CV | max per-class max/min |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| X-IIoTID | 0.25 | 19 | 38 | 14.75 / 20.84 | 1.442 | 2.134 | 122,800 |
|
||||
| X-IIoTID | 0.50 | 7 | 9 | 7.41 / 9.89 | 1.183 | 2.095 | 12,215 |
|
||||
| X-IIoTID | 0.75 | 3 | 4 | 4.67 / 5.07 | 0.881 | 1.322 | 1,903 |
|
||||
| 5G-NIDD | 0.25 | 16 | 30 | 17.60 / 27.83 | 1.302 | 2.139 | 185,504 |
|
||||
| 5G-NIDD | 0.50 | 4 | 4 | 9.59 / 12.72 | 1.225 | 1.964 | 88,943 |
|
||||
| 5G-NIDD | 0.75 | 4 | 5 | 6.31 / 11.36 | 0.916 | 1.265 | 7,642.5 |
|
||||
|
||||
Measured facts:
|
||||
|
||||
- Realized support imbalance, missing-cell frequency and client-size ratio **decrease as alpha
|
||||
rises** within each dataset (alpha orders heterogeneity only on average).
|
||||
- Per-seed realized imbalance is **not monotone** even at fixed alpha: for 5G-NIDD alpha=0.50,
|
||||
seed 0 has client-size ratio 11.63 while alpha=0.25 seed 0 has 9.93.
|
||||
- All 10 clients are eligible in every condition (exactly two rare attack classes each).
|
||||
- Rare-class **union differs across seeds** in several conditions (e.g. 5G-NIDD alpha=0.25:
|
||||
union rare 6/6/8, union unseen 6/5/5), so test rare support is not constant across seeds and
|
||||
absolute cross-seed metric levels are not directly comparable.
|
||||
- The `S_j` reproduction assumption is materially exercised in 19/27 (X-IIoTID alpha=0.25),
|
||||
16/27 (5G-NIDD alpha=0.25) and ≤7/27 cells elsewhere.
|
||||
|
||||
**Interpretation**: `alpha` is a design knob; the realized partition (support CV, missing cells,
|
||||
client-size ratio) is the variable that actually differs across conditions.
|
||||
|
||||
---
|
||||
|
||||
## 5. Pooled beta-selection behavior
|
||||
|
||||
Authoritative curve table: `artifacts/analysis/final__pooled_validation_curves.csv` (30 rows).
|
||||
Figure: `artifacts/figures/final/figA_*_pooled_validation_beta_curve.png`.
|
||||
|
||||
Pooled validation rare-class macro-F1 (selected in bold):
|
||||
|
||||
| beta | X-IIoTID a0.25 | X-IIoTID a0.50 | X-IIoTID a0.75 | 5G-NIDD a0.25 | 5G-NIDD a0.50 | 5G-NIDD a0.75 |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 0.00 | **0.5917** | 0.5444 | 0.4717 | **0.7615** | 0.6872 | 0.7781 |
|
||||
| 0.25 | 0.5887 | 0.5554 | 0.4744 | 0.7493 | 0.6451 | 0.7905 |
|
||||
| 0.50 | 0.5859 | 0.5673 | **0.4767** | 0.7257 | 0.6747 | **0.8091** |
|
||||
| 0.75 | 0.5795 | **0.5794** | 0.4725 | 0.7222 | 0.6884 | 0.7915 |
|
||||
| 1.00 | 0.5321 | 0.5735 | 0.4686 | 0.7376 | **0.6924** | 0.8043 |
|
||||
|
||||
Measured facts:
|
||||
|
||||
- Response is **not monotone** in beta in any condition:
|
||||
X-IIoTID alpha=0.25 decreases monotonically (0.5917→0.5321);
|
||||
X-IIoTID alpha=0.50 peaks at 0.75; X-IIoTID alpha=0.75 peaks at 0.50;
|
||||
5G-NIDD alpha=0.25 falls to a minimum at 0.75 then rises at 1.00;
|
||||
5G-NIDD alpha=0.50 peaks at 1.00; 5G-NIDD alpha=0.75 peaks at 0.50.
|
||||
- Between-beta spreads (0.008 – 0.060) are of the same order as, or smaller than, across-seed
|
||||
sample std (0.003 – 0.139). See section 13.
|
||||
|
||||
---
|
||||
|
||||
## 6. Selected-beta test results (descriptive only)
|
||||
|
||||
Table: `artifacts/analysis/final__selected_vs_original.csv`.
|
||||
Figure: `artifacts/figures/final/figF_selected_vs_original_test_rare_macro_f1.png`.
|
||||
|
||||
Descriptive difference `selected beta − original_protean` (mean over 3 seeds), **not** a test of
|
||||
statistical superiority:
|
||||
|
||||
| dataset | alpha | beta* | Δ rare-F1 | Δ rare-recall | Δ macro-F1 | Δ accuracy | Δ wF1 |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| X-IIoTID | 0.25 | 0.00 | −0.0007 | −0.0007 | −0.0007 | −0.0014 | −0.0008 |
|
||||
| X-IIoTID | 0.50 | 0.75 | **+0.0230** | −0.0144 | +0.0222 | +0.0069 | +0.0089 |
|
||||
| X-IIoTID | 0.75 | 0.50 | +0.0014 | −0.0077 | +0.0058 | +0.0075 | +0.0061 |
|
||||
| 5G-NIDD | 0.25 | 0.00 | −0.0010 | +0.0007 | −0.0028 | −0.0007 | −0.0010 |
|
||||
| 5G-NIDD | 0.50 | 1.00 | −0.0086 | −0.0058 | −0.0038 | −0.0007 | −0.0005 |
|
||||
| 5G-NIDD | 0.75 | 0.50 | +0.0168 | −0.0015 | +0.0117 | +0.0013 | +0.0014 |
|
||||
|
||||
Measured facts:
|
||||
|
||||
- The largest positive descriptive rare-F1 difference (+0.0230, X-IIoTID alpha=0.50) is still
|
||||
smaller than that condition's between-seed sample std (0.0267) and than the original_protean
|
||||
cross-seed std (0.0287).
|
||||
- At both alpha=0.25 conditions the selected beta is 0, so the "selected" run **is** the beta=0
|
||||
run; the tiny differences from `original_protean` are floating-point divergence (class B), not
|
||||
a weighting effect.
|
||||
- Selected rare-recall is ≤ original rare-recall in 5 of 6 conditions: emphasis on high-support
|
||||
classes trades a little rare-class recall for rare-class precision.
|
||||
- 5G-NIDD test weighted-F1 sits below accuracy (hard, high-support Benign/UDPFlood); X-IIoTID
|
||||
weighted-F1 exceeds accuracy. This is a dataset characteristic, not a beta effect.
|
||||
|
||||
---
|
||||
|
||||
## 7. Direct prototype displacement
|
||||
|
||||
Authoritative tables: `final__prototype_displacement_by_beta.csv`,
|
||||
`final__prototype_displacement_by_class.csv`.
|
||||
Figures: `figD_prototype_displacement_beta0_vs_beta1.png`,
|
||||
`figE_support_cv_vs_displacement.png`.
|
||||
|
||||
Mean `||Cbar_j(beta) − Cbar_j(0)||` across classes, fixed round-0 snapshots (no retraining):
|
||||
|
||||
| dataset | alpha | β=0.25 | β=0.50 | β=0.75 | β=1.00 |
|
||||
|---|---|---|---|---|---|
|
||||
| X-IIoTID | 0.25 | 1.476 | 2.403 | 3.003 | **3.422** |
|
||||
| X-IIoTID | 0.50 | 0.826 | 1.374 | 1.780 | **2.099** |
|
||||
| X-IIoTID | 0.75 | 0.477 | 0.829 | 1.097 | **1.305** |
|
||||
| 5G-NIDD | 0.25 | 1.295 | 2.055 | 2.559 | **2.919** |
|
||||
| 5G-NIDD | 0.50 | 0.953 | 1.567 | 2.003 | **2.340** |
|
||||
| 5G-NIDD | 0.75 | 0.497 | 0.875 | 1.163 | **1.385** |
|
||||
|
||||
Measured facts:
|
||||
|
||||
- For a fixed snapshot, displacement is **monotone increasing in beta** in all six conditions
|
||||
(mechanically expected: larger beta shifts weight toward high-`N_ij` clients).
|
||||
- Displacement at beta=1 **decreases as alpha rises** in both datasets, tracking realized
|
||||
support imbalance.
|
||||
- Displacement is a **representation-aggregation** metric only; it is not detection quality.
|
||||
|
||||
---
|
||||
|
||||
## 8. Client-level effects
|
||||
|
||||
Authoritative table: `artifacts/analysis/final__client_effect_summary.csv` (90 rows:
|
||||
6 conditions × 3 seeds × 5 betas, `delta` = validation per-client rare-class macro-F1 relative
|
||||
to beta=0). Figure: `figG_client_effect_at_selected_beta.png`.
|
||||
|
||||
Aggregate at **beta=1.00** (mean of per-client mean delta across 3 seeds):
|
||||
|
||||
| dataset | alpha | mean Δ | improve / degrade / unchanged (clients, summed over seeds) |
|
||||
|---|---|---|---|
|
||||
| X-IIoTID | 0.25 | **−0.0596** | 12 / 15 / 3 |
|
||||
| X-IIoTID | 0.50 | +0.0291 | 19 / 9 / 2 |
|
||||
| X-IIoTID | 0.75 | −0.0031 | 14 / 15 / 1 |
|
||||
| 5G-NIDD | 0.25 | **−0.0238** | 10 / 11 / 9 |
|
||||
| 5G-NIDD | 0.50 | +0.0051 | 16 / 10 / 4 |
|
||||
| 5G-NIDD | 0.75 | +0.0262 | 14 / 10 / 6 |
|
||||
|
||||
Measured facts:
|
||||
|
||||
- Support emphasis at beta=1 is most damaging at alpha=0.25 in both datasets (most negative
|
||||
mean delta, most client degradation), consistent with those conditions selecting beta=0.
|
||||
- Within a single condition, some clients improve while others degrade at every beta
|
||||
(both datasets, all alphas). `N_ij` vs ΔF1 is non-monotonic: the worst-degrading client is
|
||||
not always the lowest-support client.
|
||||
- Client identities are per-seed realizations and are **not pooled** across seeds.
|
||||
|
||||
---
|
||||
|
||||
## 9. RQ1 answer
|
||||
|
||||
> **RQ1.** How are class-support imbalance, differences between local prototypes, and rare-class
|
||||
> detection performance related in PROTEAN?
|
||||
|
||||
Evidence chain (all three alphas, both datasets):
|
||||
|
||||
*Mechanically established.*
|
||||
- Changing beta changes the normalized contribution weights `N_ij^beta / Σ_k N_kj^beta` and
|
||||
therefore the position of the global prototype `Cbar_j`. For a fixed snapshot, mean
|
||||
displacement grows monotonically with beta in all six conditions.
|
||||
|
||||
*Empirically observed (descriptive).*
|
||||
- Larger realized support imbalance (higher support CV, more missing cells, higher client-size
|
||||
ratio) tends to accompany larger beta=1 displacement, within both datasets:
|
||||
X-IIoTID (CV, disp) = (1.442, 3.422) → (1.183, 2.099) → (0.881, 1.305);
|
||||
5G-NIDD = (1.302, 2.919) → (1.225, 2.340) → (0.916, 1.385).
|
||||
- Imbalance also co-varies with selection behavior: both alpha=0.25 conditions are the most
|
||||
imbalanced and both selected beta=0.
|
||||
|
||||
*Not established.*
|
||||
- Larger displacement does **not** automatically imply better rare-class detection: at beta=1
|
||||
(maximal displacement) rare-class performance is worse than at the selected beta in most
|
||||
conditions, and validation response is non-monotone in beta.
|
||||
- The imbalance → displacement link is mechanical under the beta formula; the link from
|
||||
displacement to detection is confounded by end-to-end retraining and cannot be isolated with
|
||||
this design.
|
||||
|
||||
RQ1 can therefore be answered **descriptively**: support imbalance and prototype displacement
|
||||
co-vary, but displacement is not a proxy for rare-class detection quality.
|
||||
|
||||
---
|
||||
|
||||
## 10. RQ2 answer
|
||||
|
||||
> **RQ2.** How does changing beta affect the global prototype and rare-class detection
|
||||
> performance under different non-IID conditions?
|
||||
|
||||
*Direct aggregation effect (fixed snapshot, no retraining).*
|
||||
- Displacement from beta=0 is monotone increasing in beta in every condition; magnitude differs
|
||||
by condition (largest at alpha=0.25).
|
||||
- This is a pure weighting/geometry effect and is measured independently of training.
|
||||
|
||||
*End-to-end training effect (iterative).*
|
||||
- Pooled validation rare-class macro-F1 is **not monotone** in beta. Preference varies by
|
||||
condition: high beta is preferred at 5G-NIDD alpha=0.50 (beta=1.0), moderate beta at both
|
||||
alpha=0.75 conditions (beta=0.5), and the smallest beta at both alpha=0.25 conditions
|
||||
(beta=0.0).
|
||||
- Weighting helps some clients and hurts others within the same condition (section 8), so the
|
||||
condition-level preference is an average over heterogeneous client responses.
|
||||
|
||||
The two effects must not be merged: the geometry changes monotonically, the task-level response
|
||||
does not.
|
||||
|
||||
---
|
||||
|
||||
## 11. RQ3 answer
|
||||
|
||||
> **RQ3.** Does the validation-selected beta change across non-IID levels, and what is the
|
||||
> effect on overall and per-client performance?
|
||||
|
||||
Authoritative selected betas (validation only):
|
||||
|
||||
| dataset | alpha=0.25 | alpha=0.50 | alpha=0.75 |
|
||||
|---|---|---|---|
|
||||
| X-IIoTID | 0.00 | 0.75 | 0.50 |
|
||||
| 5G-NIDD | 0.00 | 1.00 | 0.50 |
|
||||
|
||||
Established conclusions:
|
||||
|
||||
- The selected beta **changes by condition**: it is not a single universal value and not a
|
||||
monotone function of alpha.
|
||||
- Both datasets select beta=0 at alpha=0.25 and beta=0.50 at alpha=0.75; they differ at
|
||||
alpha=0.50 (X-IIoTID 0.75 vs 5G-NIDD 1.00).
|
||||
- Only five discrete candidates were tested; no global optimum is claimed, and each value is
|
||||
"best among the tested candidates under pooled validation for that condition".
|
||||
|
||||
Overall effect (section 6): selected-beta test rare-F1 differences are small and positive in
|
||||
three conditions, small and negative in three; the largest positive difference is smaller than
|
||||
between-seed variability. Per-client effect (section 8): the selected beta improves the
|
||||
validation rare-class metric for a majority of clients in four conditions, but at both alpha=0.25
|
||||
conditions (where the selected beta is 0) there is trivially no weighting change, and at higher
|
||||
beta the effect is client-dependent.
|
||||
|
||||
---
|
||||
|
||||
## 12. Cross-dataset synthesis
|
||||
|
||||
- **Agreement**: matched-alpha selections coincide at alpha=0.25 (both 0.00) and alpha=0.75
|
||||
(both 0.50); they differ at alpha=0.50 (0.75 vs 1.00).
|
||||
- **Displacement ordering is consistent**: beta=1 displacement falls with alpha in both datasets,
|
||||
and ranking of conditions by displacement mirrors ranking by realized support CV.
|
||||
- **Rare-class objects are not comparable across datasets**: 5G-NIDD rare classes are low-volume
|
||||
flood/scan types; X-IIoTID rare classes are low-volume behavioural categories. Absolute metric
|
||||
levels must not be compared across datasets; only within-condition differences are meaningful.
|
||||
- **Metric asymmetry**: 5G-NIDD weighted-F1 < accuracy; X-IIoTID weighted-F1 > accuracy.
|
||||
- No cross-dataset causality, no dataset superiority, no dataset-independent beta rule.
|
||||
|
||||
---
|
||||
|
||||
## 13. Selection uncertainty
|
||||
|
||||
Table: `artifacts/analysis/final__selection_strength.csv`.
|
||||
|
||||
| dataset | alpha | beta* | best | 2nd best | abs gap | rel gap | top-2 gap / selected std | per-seed betas |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| X-IIoTID | 0.25 | 0.00 | 0.5917 | 0.5887 (0.25) | 0.0030 | 0.51% | 0.034 | 0.75 / 0.5 / 0.0 |
|
||||
| X-IIoTID | 0.50 | 0.75 | 0.5794 | 0.5735 (1.0) | 0.0059 | 1.02% | 0.078 | 0.5 / 1.0 / 0.75 |
|
||||
| X-IIoTID | 0.75 | 0.50 | 0.4767 | 0.4744 (0.25) | 0.0024 | 0.50% | 0.051 | 0.5 / 0.5 / 0.0 |
|
||||
| 5G-NIDD | 0.25 | 0.00 | 0.7615 | 0.7493 (0.25) | 0.0122 | 1.60% | 0.556 | 0.25 / 0.0 / 0.0 |
|
||||
| 5G-NIDD | 0.50 | 1.00 | 0.6924 | 0.6884 (0.75) | 0.0040 | 0.58% | 0.029 | 1.0 / 0.0 / 1.0 |
|
||||
| 5G-NIDD | 0.75 | 0.50 | 0.8091 | 0.8043 (1.0) | 0.0048 | 0.59% | 0.081 | 0.5 / 1.0 / 1.0 |
|
||||
|
||||
Quantitative basis for the (descriptive) classification: the top-two gap is compared with the
|
||||
selected beta's across-seed sample std. **In all six conditions the top-two gap is smaller than
|
||||
the selected-beta sample std** (ratios 0.03 – 0.56), so every selection is classified as
|
||||
*small numerical separation relative to seed variability*. No tie-break was ever exercised. The
|
||||
per-seed diagnostic beta agrees with the pooled selection in only 1–2 of 3 seeds per condition.
|
||||
|
||||
No inferential test (t-test, ANOVA, Wilcoxon, Friedman, regression) is applied. With three
|
||||
project-defined seeds per condition, such tests would be underpowered and would overstate the
|
||||
evidence; mean, sample std, per-seed values and the top-two gap are reported instead.
|
||||
|
||||
---
|
||||
|
||||
## 14. Threats to validity
|
||||
|
||||
### Internal validity
|
||||
- **`S_j` reproduction assumption**: the participant set / denominator rule is not verified
|
||||
against official code and is materially exercised in up to 19/27 cells. All results are
|
||||
conditional on it.
|
||||
- **beta=0 floating-point divergence**: beta=0 is semantically the equal mean but not bitwise
|
||||
identical to `original_protean`; selected-vs-original differences at alpha=0.25 are this
|
||||
divergence, not an effect. Equivalence class B everywhere.
|
||||
- **Frozen implementation**: aggregation and training code unchanged across all 108 runs;
|
||||
the same manifest/split/init is reused within each `(dataset, alpha, seed)`.
|
||||
- **Paired partitions/seeds**: verified by content, partition and split hashes.
|
||||
- **Validation/test separation**: beta selected from validation only; `test_used_for_selection`
|
||||
is false in all runs; test read only after selection.
|
||||
|
||||
### Construct validity
|
||||
- `N_ij` is **sample support, not prototype quality**; a larger `N_ij` does not mean a better
|
||||
prototype.
|
||||
- Rare-class definition is operational (two smallest positive-support attack classes per client
|
||||
from the training split); rare and unseen are never merged.
|
||||
- Prototype displacement is a **representation** metric, not detection quality.
|
||||
|
||||
### External validity
|
||||
- Only two IDS datasets (X-IIoTID, 5G-NIDD), 10 clients, three alpha levels, three
|
||||
project-defined seeds, one PROTEAN architecture and training configuration.
|
||||
- Rare-class compositions are dataset-specific and not comparable across datasets.
|
||||
|
||||
### Conclusion validity
|
||||
- Small seed count (3); between-beta spreads are comparable to across-seed std.
|
||||
- Discrete beta grid `{0,0.25,0.5,0.75,1.0}`; the true response between candidates is unobserved.
|
||||
- Weak numerical separation at all six conditions (section 13).
|
||||
- No inferential significance testing was performed or is claimed.
|
||||
|
||||
---
|
||||
|
||||
## 15. Contribution supported by evidence
|
||||
|
||||
1. A **controlled, beta-parameterized support-weighting** operator applied to the server-side
|
||||
prototype aggregation of PROTEAN, with an explicit participant rule and missing-class
|
||||
semantics.
|
||||
2. **Condition-wise, validation-based beta selection** (no test leakage) across six paired
|
||||
conditions.
|
||||
3. A **paired multi-seed analysis** across two IDS datasets and three non-IID levels, with
|
||||
identical partitions within each comparison cell.
|
||||
4. A **direct linkage** between realized support imbalance, weight redistribution, prototype
|
||||
displacement, rare-class detection and per-client consequences — separated into direct
|
||||
aggregation effects (fixed snapshots) and end-to-end training effects.
|
||||
5. Evidence of **when stronger support weighting helps, has little effect, or degrades**
|
||||
rare-class behavior, including the identification that both evaluated alpha=0.25 conditions
|
||||
select beta=0 (no weighting benefit) and that response is not monotone in beta.
|
||||
|
||||
**Novelty caveat.** Sample-count prototype weighting is already discussed in related work, e.g.
|
||||
**FedProto** (Tan et al., 2022). The contribution cannot rest on using `N_ij` or adding beta
|
||||
alone; it rests on the controlled parameterization, the condition-wise validation selection, the
|
||||
paired multi-seed setup, and the resulting behavioral findings.
|
||||
|
||||
---
|
||||
|
||||
## 16. Claims NOT supported by evidence
|
||||
|
||||
The following must not appear as established results:
|
||||
|
||||
- "Support weighting improves rare-class detection." (Mixed and condition-dependent; largest
|
||||
positive difference is within seed noise.)
|
||||
- "Selected-beta beats original PROTEAN." (Only small descriptive differences; no significance
|
||||
test; at alpha=0.25 the difference is floating-point divergence.)
|
||||
- "alpha causally determines the optimal beta." (Only two datasets, three seeds; selection is
|
||||
not monotone in alpha.)
|
||||
- "A larger `N_ij` means a better prototype / greater prototype quality."
|
||||
- "Larger prototype displacement means better (or worse) rare-class detection."
|
||||
- "beta=0 reproduces original PROTEAN" (only semantically equivalent under the shared `S_j`
|
||||
assumption; class B divergence).
|
||||
- "There is a universal / globally optimal beta."
|
||||
- "The method generalizes" or "dataset X is superior".
|
||||
- Any metric interpreted as improvement without the qualifier "numerical/descriptive".
|
||||
|
||||
---
|
||||
|
||||
## 17. Recommended thesis conclusions
|
||||
|
||||
1. Under the validated design, support weighting is **condition-dependent**: the best tested beta
|
||||
differs across non-IID conditions, is not monotone in alpha, and no single beta dominates.
|
||||
2. Changing beta **monotonically displaces** global prototypes at a fixed snapshot, while the
|
||||
rare-class detection response is **non-monotone** and weak relative to seed variability.
|
||||
3. Realized support imbalance and prototype displacement **co-vary**; displacement alone is not
|
||||
evidence of detection quality.
|
||||
4. Weighting **redistributes benefit across clients**; within a condition some clients gain and
|
||||
others lose.
|
||||
5. In both evaluated datasets the most imbalanced condition (alpha=0.25) selected beta=0, i.e.
|
||||
equal weighting over the shared participant set, and realized the largest prototype
|
||||
displacement — a descriptive pattern that motivates broader validation, not a rule.
|
||||
|
||||
---
|
||||
|
||||
## 18. Scientific-gap assessment
|
||||
|
||||
**Classification: A — NO MATERIAL GAP** for the defined thesis scope.
|
||||
|
||||
- RQ1, RQ2 and RQ3 are all answerable descriptively from the 108-run grid and the existing
|
||||
audited artifacts.
|
||||
- The remaining limitations (S_j assumption, three seeds, discrete grid, two datasets, no
|
||||
inferential power) are **scope/validity** boundaries, not missing experiments that prevent the
|
||||
stated research questions from being answered.
|
||||
- A **causal** isolation of the displacement→detection link (e.g. evaluating recomputed global
|
||||
prototypes on trained embeddings at a fixed snapshot, or interpolating beta in prototype space)
|
||||
would strengthen RQ1/RQ2, but it is not required by the defined RQs and would change the design;
|
||||
it is therefore out of scope rather than a blocking gap.
|
||||
- No new training is required or recommended.
|
||||
|
||||
---
|
||||
|
||||
## Artifact index (Session 12 deliverables)
|
||||
|
||||
| artifact | path |
|
||||
|---|---|
|
||||
| master table | `artifacts/analysis/final__six_condition_master.{csv,json}` |
|
||||
| pooled validation curves | `artifacts/analysis/final__pooled_validation_curves.csv` |
|
||||
| realized heterogeneity | `artifacts/analysis/final__realized_heterogeneity.csv` |
|
||||
| displacement by beta | `artifacts/analysis/final__prototype_displacement_by_beta.csv` |
|
||||
| displacement by class (beta=1) | `artifacts/analysis/final__prototype_displacement_by_class.csv` |
|
||||
| client-level effect | `artifacts/analysis/final__client_effect_summary.csv` |
|
||||
| selected vs original | `artifacts/analysis/final__selected_vs_original.csv` |
|
||||
| selection strength | `artifacts/analysis/final__selection_strength.csv` |
|
||||
| figures | `artifacts/figures/final/figA–G*.png` |
|
||||
| Chapter 4 outline | `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md` |
|
||||
| claims register | `docs/thesis/CLAIMS_REGISTER.md` |
|
||||
|
||||
**Environment note.** Figures require `matplotlib`, which is not in the frozen `research`
|
||||
dependency set. It was installed into the git-ignored `.venv` only
|
||||
(`uv pip install --python .venv/bin/python matplotlib`, matplotlib 3.11.2); no tracked
|
||||
dependency file was modified. The synthesis generator is `scripts`-style analysis code kept
|
||||
outside the repository at `/tmp/opencode/s12/synthesize_final_analysis.py`, consistent with the
|
||||
Session-11 convention for ad-hoc analysis helpers.
|
||||
Reference in new issue
Block a user