research: finalize experimental synthesis

docs/thesis/CHAPTER4_RESULTS_OUTLINE.md

@@ -0,0 +1,148 @@
+# Chapter 4 — Results and Discussion (Blueprint)
+
+This is a **blueprint**, not a finished chapter. Each subsection states its purpose, the exact
+table/figure to use, the key numbers to quote (loaded from audited artifacts), what may be
+interpreted, and what must not be interpreted. All numbers trace to
+`docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md` and the `artifacts/analysis/final__*` files.
+
+Global wording rules: prefer "numerical/descriptive difference", "best among the tested
+candidates under pooled validation", "co-varies"; never "improvement", "optimal", "causal",
+"superior", or "generalizes" unless explicitly qualified.
+
+---
+
+## 4.1 Experimental setup recap
+
+- **Purpose**: fix the protocol so later numbers are reproducible and comparable.
+- **Use**: text; table of grid (Section 2 of synthesis); `docs/research/EXPERIMENT_PROTOCOL.md`.
+- **Key numbers**: 2 datasets × 3 alpha × 3 seeds × (5 betas + `original_protean`) = **108 runs**;
+  betas `{0, 0.25, 0.5, 0.75, 1}`; 10 clients; 10 rounds; 3 local epochs; batch 256; Adam lr=1e-3,
+  mu=0.1, lambda=1.0; train/val/test 70/10/20; validation-only selection.
+- **Allowed**: state that `S_j` is a reproduction assumption; state that beta=0 is not bitwise
+  identical to the baseline.
+- **Avoid**: describing the design as new training beyond the frozen grid; implying the paper's
+  split was 70/10/20.
+
+## 4.2 Realized non-IID characteristics
+
+- **Purpose**: show that alpha is a design knob and realized heterogeneity is the measured
+  variable.
+- **Use**: Table `final__realized_heterogeneity.csv`; Figure `figC_realized_support_cv.png`.
+- **Key numbers**: support CV 1.442/1.183/0.881 (X-IIoTID) and 1.302/1.225/0.916 (5G-NIDD) for
+  alpha 0.25/0.50/0.75; `|S_j|<M` 19/7/3 and 16/4/4 of 27; missing cells 38/9/4 and 30/4/5;
+  client-size ratio 14.75/7.41/4.67 and 17.60/9.59/6.31.
+- **Allowed**: realized imbalance generally decreases as alpha rises; per-seed imbalance is not
+  monotone at fixed alpha; rare-class union differs across seeds.
+- **Avoid**: treating Dirichlet alpha itself as the observed heterogeneity; comparing rare-class
+  metrics across datasets.
+
+## 4.3 Beta tuning results
+
+- **Purpose**: present the pooled validation curves and the six selections.
+- **Use**: Table `final__pooled_validation_curves.csv`; Figures
+  `figA_xiiotid_pooled_validation_beta_curve.png`,
+  `figA_5gnidd_pooled_validation_beta_curve.png`, `figB_selected_beta_by_condition.png`;
+  selection-strength table `final__selection_strength.csv`.
+- **Key numbers**: selections X-IIoTID 0.00 / 0.75 / 0.50 and 5G-NIDD 0.00 / 1.00 / 0.50 for
+  alpha 0.25 / 0.50 / 0.75; response non-monotone in beta; all top-two gaps < selected-beta
+  across-seed std (relative gaps 0.50% – 1.60%).
+- **Allowed**: selected beta changes by condition; selection is not monotone in alpha; no global
+  optimum; separation is small relative to seed variability.
+- **Avoid**: calling the selected beta "optimal"; drawing a continuous optimum; reporting
+  per-seed diagnostic betas as the primary result.
+
+## 4.4 Effect on global prototypes
+
+- **Purpose**: separate the direct (fixed-snapshot) aggregation effect from training.
+- **Use**: Table `final__prototype_displacement_by_beta.csv`; Figure
+  `figD_prototype_displacement_beta0_vs_beta1.png`; per-class file
+  `final__prototype_displacement_by_class.csv`.
+- **Key numbers**: mean `||Cbar(1)−Cbar(0)||` = 3.422 / 2.099 / 1.305 (X-IIoTID) and
+  2.919 / 2.340 / 1.385 (5G-NIDD); monotone increasing in beta within each condition.
+- **Allowed**: displacement grows monotonically with beta for a fixed snapshot; magnitude differs
+  by realized condition; larger beta moves the global prototype toward high-support clients.
+- **Avoid**: interpreting displacement as improved/degraded detection; implying a causal link
+  from displacement to rare-class performance.
+
+## 4.5 Rare-class detection results
+
+- **Purpose**: report selected-beta test metrics against `original_protean` descriptively.
+- **Use**: master table `final__six_condition_master.csv`; table
+  `final__selected_vs_original.csv`; Figure
+  `figF_selected_vs_original_test_rare_macro_f1.png`.
+- **Key numbers**: Δ test rare-F1 = −0.0007 / +0.0230 / +0.0014 (X-IIoTID) and
+  −0.0010 / −0.0086 / +0.0168 (5G-NIDD); largest positive difference (+0.0230) below that
+  condition's between-seed std (0.0267); alpha=0.25 differences are floating-point divergence.
+- **Allowed**: differences are numerical/descriptive only; mixed sign; rare-recall is usually
+  slightly lower under weighting.
+- **Avoid**: significance markers; "beats baseline"; merging rare and unseen; comparing absolute
+  levels across conditions or datasets.
+
+## 4.6 Per-client effects
+
+- **Purpose**: show redistribution of benefit across clients.
+- **Use**: Table `final__client_effect_summary.csv`; Figure
+  `figG_client_effect_at_selected_beta.png`.
+- **Key numbers**: beta=1 mean Δ = −0.0596 / +0.0291 / −0.0031 (X-IIoTID) and
+  −0.0238 / +0.0051 / +0.0262 (5G-NIDD); improve/degrade/unchanged counts per condition.
+- **Allowed**: within a condition some clients gain and others lose; most damaging at alpha=0.25;
+  `N_ij` vs ΔF1 is non-monotonic.
+- **Avoid**: pooling client identities across seeds; claiming weighting helps "all" or
+  "low-support" clients.
+
+## 4.7 Cross-dataset analysis
+
+- **Purpose**: compare the two datasets at matched alpha without over-claiming.
+- **Use**: synthesis Section 12; cross-dataset artifacts `cross_dataset__matched_alpha_summary.csv`.
+- **Key numbers**: matched selections agree at alpha=0.25 (0.00) and alpha=0.75 (0.50), differ at
+  alpha=0.50 (0.75 vs 1.00).
+- **Allowed**: descriptive agreement/disagreement; both datasets' displacement falls with alpha.
+- **Avoid**: cross-dataset metric comparison; dataset causality; universal beta rule; claiming a
+  dataset-independent law.
+
+## 4.8 RQ1 discussion
+
+- **Purpose**: answer the support-imbalance ↔ prototype-difference ↔ rare-class relation.
+- **Use**: synthesis Section 9; sections 4.2, 4.4, 4.5.
+- **Key numbers**: imbalance ranking and displacement ranking coincide within each dataset.
+- **Allowed**: mechanical weight/prototype effect is established; imbalance and displacement
+  co-vary descriptively; displacement is not detection quality.
+- **Avoid**: causal mediation claims; "displacement improves detection".
+
+## 4.9 RQ2 discussion
+
+- **Purpose**: answer how beta affects global prototypes and rare-class performance.
+- **Use**: synthesis Section 10; sections 4.3, 4.4, 4.6.
+- **Key numbers**: monotone displacement vs non-monotone validation response.
+- **Allowed**: distinguish direct aggregation effect from end-to-end training effect; weighting
+  helps some clients and harms others.
+- **Avoid**: treating the monotonically growing displacement as a performance trend.
+
+## 4.10 RQ3 discussion
+
+- **Purpose**: answer whether the selected beta changes across non-IID levels and its effect.
+- **Use**: synthesis Section 11; sections 4.3, 4.5, 4.6.
+- **Key numbers**: selected betas `{0.00, 0.75, 0.50}` (X-IIoTID) and `{0.00, 1.00, 0.50}`
+  (5G-NIDD); only five candidate betas tested.
+- **Allowed**: selection changes by condition; not monotone in alpha; no universal beta; both
+  datasets select 0 at alpha=0.25 and 0.5 at alpha=0.75.
+- **Avoid**: "alpha causally determines beta"; "global optimum"; extrapolating to untested betas.
+
+## 4.11 Threats / limitations
+
+- **Purpose**: bound the conclusions honestly.
+- **Use**: synthesis Section 14.
+- **Key numbers**: S_j assumption material up to 19/27 cells; three seeds; discrete grid; two
+  datasets; no inferential test.
+- **Allowed**: internal/construct/external/conclusion validity bullets as written.
+- **Avoid**: omitting the reproduction assumption; implying statistical significance.
+
+## 4.12 Chapter summary
+
+- **Purpose**: one-paragraph takeaway.
+- **Use**: synthesis Section 17.
+- **Key numbers**: condition-dependent beta, non-monotone response, displacement co-varies with
+  imbalance, client-level heterogeneity, both alpha=0.25 conditions select beta=0.
+- **Allowed**: descriptive summary; explicit statement that no global optimum or universal beta
+  was found and that only tested candidates were compared.
+- **Avoid**: any claim of improvement, causality, superiority or generalization.

docs/thesis/CLAIMS_REGISTER.md

@@ -0,0 +1,149 @@
+# Thesis Claims Register
+
+Every claim considered for Chapter 4/5, with its evidence, status, limitation and a
+thesis-safe wording. Statuses:
+
+- **supported** — directly established by audited machine-readable artifacts or by the frozen
+  operator definition.
+- **descriptive only** — observable in the artifacts but not causally or inferentially established.
+- **unsupported** — the design cannot defend the claim; must not appear as a result.
+
+Artifacts referenced live under `artifacts/analysis/final__*`, `artifacts/analysis/*__multiseed__*`,
+and `docs/research/*`. Numbers are cross-checked in
+`docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md`.
+
+---
+
+## 1. Beta changes prototype positions
+
+- **Status**: supported (mechanically established).
+- **Evidence**: `final__prototype_displacement_by_beta.csv`; frozen aggregator definition
+  `Cbar_j(beta) = Σ N_ij^beta C_ij / Σ N_ij^beta`.
+- **Limitation**: conditional on the shared `S_j` reproduction assumption; fixed round-0 snapshot.
+- **Safe wording**: "Changing beta changes the normalized contribution weights and therefore the
+  position of each global prototype."
+
+## 2. Prototype displacement increases with beta
+
+- **Status**: supported (monotone at fixed snapshot).
+- **Evidence**: `final__prototype_displacement_by_beta.csv` — mean displacement increases from
+  beta=0.25 to beta=1.00 in all six conditions.
+- **Limitation**: measured on fixed round-0 snapshots; no retraining; not a detection metric.
+- **Safe wording**: "For a fixed snapshot, mean `||Cbar(beta)−Cbar(0)||` increases monotonically
+  with beta in every evaluated condition."
+
+## 3. Realized imbalance and prototype displacement co-vary
+
+- **Status**: descriptive only.
+- **Evidence**: `final__realized_heterogeneity.csv` and `final__prototype_displacement_by_beta.csv`;
+  both datasets show support CV and beta=1 displacement falling together as alpha rises.
+- **Limitation**: six conditions, three seeds; correlation not quantified with inferential
+  statistics; displacement is a representation metric.
+- **Safe wording**: "Within both datasets, conditions with larger realized support imbalance also
+  exhibit larger beta=1 prototype displacement; this is a descriptive co-variation."
+
+## 4. Beta selection changes by condition
+
+- **Status**: supported (measured, validation-only).
+- **Evidence**: `final__six_condition_master.csv`; `final__selection_strength.csv`.
+- **Limitation**: only five discrete candidates; selection is "best among tested candidates".
+- **Safe wording**: "The validation-selected beta differs across conditions and is not monotone in
+  alpha; it is best among the tested candidates for each condition."
+
+## 5. Beta is not universal
+
+- **Status**: supported.
+- **Evidence**: selected betas `{0.00,0.75,0.50}` (X-IIoTID) and `{0.00,1.00,0.50}` (5G-NIDD).
+- **Limitation**: only two datasets and three alpha levels.
+- **Safe wording**: "The selected beta takes three distinct values across the six conditions and
+  is not a single universal value; no global optimum is claimed."
+
+## 6. Beta=0 selected at alpha=0.25 on both datasets
+
+- **Status**: supported (measured).
+- **Evidence**: `final__six_condition_master.csv` (X-IIoTID and 5G-NIDD alpha=0.25 both select 0.00).
+- **Limitation**: two datasets, three seeds; descriptive pattern, not a rule. Because the selected
+  beta is 0, these conditions cannot demonstrate any support-weighting benefit.
+- **Safe wording**: "In both evaluated datasets the alpha=0.25 condition selected beta=0 among the
+  tested candidates; these conditions also showed the strongest realized support imbalance and the
+  largest prototype displacement."
+
+## 7. Selected beta beats original PROTEAN
+
+- **Status**: unsupported.
+- **Evidence**: `final__selected_vs_original.csv` — differences are mixed in sign and small; the
+  largest positive rare-F1 difference (+0.0230, X-IIoTID alpha=0.50) is below that condition's
+  between-seed std (0.0267); alpha=0.25 differences are floating-point divergence (class B).
+- **Limitation**: no significance test; three seeds; selected beta is determined post hoc per
+  condition.
+- **Safe wording**: "Selected-beta test metrics show only small, mixed-sign numerical differences
+  from `original_protean`; no statistical superiority is established."
+
+## 8. Support weighting improves rare-class detection
+
+- **Status**: unsupported.
+- **Evidence**: `final__selected_vs_original.csv`, `final__client_effect_summary.csv` — response is
+  condition-dependent, non-monotone in beta, and some clients degrade at every beta; at beta=1 the
+  mean client delta is negative at both alpha=0.25 conditions and at X-IIoTID alpha=0.75.
+- **Limitation**: end-to-end results confound the weighting with retraining; small seed count.
+- **Safe wording**: "Support weighting changes rare-class behavior condition-dependently: it helps
+  a majority of clients in some conditions and degrades most clients at the most imbalanced
+  conditions; no uniform improvement is demonstrated."
+
+## 9. Alpha causally determines the optimal beta
+
+- **Status**: unsupported.
+- **Evidence**: `final__selection_strength.csv` — selection is not monotone in alpha; selected beta
+  is `{0.00,0.75,0.50}` and `{0.00,1.00,0.50}`, differing at alpha=0.50.
+- **Limitation**: only three alpha levels, two datasets, three seeds; no trend test performed.
+- **Safe wording**: "The selected beta changes with non-IID level but is not a monotone function of
+  alpha; no causal role of alpha is established."
+
+## 10. Beta=1 represents prototype quality
+
+- **Status**: unsupported.
+- **Evidence**: spec and protocol define `N_ij` as class-specific sample support, not prototype
+  quality; beta=1 weights by support only.
+- **Limitation**: no ground-truth prototype-quality measure exists in the design.
+- **Safe wording**: "`N_ij` is class-specific sample support; a larger `N_ij` does not imply a
+  higher-quality prototype, and beta weights support, not quality."
+
+## 11. Prototype displacement implies detection quality
+
+- **Status**: unsupported.
+- **Evidence**: `final__prototype_displacement_by_beta.csv` vs `final__pooled_validation_curves.csv`
+  — displacement is monotone in beta while detection response is non-monotone; at beta=1 detection
+  is usually worse than at the selected beta.
+- **Limitation**: displacement measured on untreated round-0 embeddings; detection measured
+  end-to-end.
+- **Safe wording**: "Prototype displacement is a representation-aggregation metric; it is not
+  interpreted as improved or degraded detection."
+
+## 12. Beta=0 reproduces original PROTEAN
+
+- **Status**: unsupported (conditional equivalence only).
+- **Evidence**: `*__multiseed__beta0_equivalence.csv` — same manifest/snapshot/support but
+  independent-aggregation differences ~2e-16 and non-zero history/test metric differences;
+  classification **B** in every condition.
+- **Limitation**: conditional on the shared `S_j` reproduction assumption; no official code.
+- **Safe wording**: "Under the shared participant-set assumption, beta=0 is semantically the equal
+  mean over `S_j` and is numerically close to `original_protean`; it is not a proof that beta=0
+  reproduces the original method."
+
+## 13. Rare-class and unseen-class performance can be merged
+
+- **Status**: unsupported.
+- **Evidence**: protocol defines rare (`N_ij>0`, two smallest positive-support attack classes) and
+  unseen (`N_ij=0`) as distinct categories from the training split.
+- **Limitation**: none; definitional.
+- **Safe wording**: "Rare-class and unseen-class results are reported separately and are never
+  merged into one metric."
+
+## 14. Results generalize beyond the tested setting
+
+- **Status**: unsupported.
+- **Evidence**: two datasets, 10 clients, three alpha levels, three project-defined seeds, one
+  architecture/training configuration.
+- **Limitation**: external validity.
+- **Safe wording**: "Findings are specific to the two evaluated IDS datasets and the frozen
+  PROTEAN configuration; no generalization is claimed."