research: add post-hoc statistical audit and defense preparation aids
- scripts/posthoc_statistical_audit.py + docs/research/POSTHOC_STATISTICAL_AUDIT.md: read-only paired audit of the completed grid (exact sign-flip / Wilcoxon, selection stability, per-class/per-client summaries); writes only posthoc__*. - kisikisithesis/: robustness audit and examiner-preparation document with deterministic read-only checks (robustness_checks.py). - artifacts/analysis/posthoc__*.csv: audit outputs (6 tables). - artifacts/analysis: track the 4 remaining beta0-snapshot effect/weight CSVs (xiiotid a0.5 seeds 1-2), completing the tracked beta0 set. - .gitignore: codify that non-beta0 prototype snapshots stay untracked.
This commit is contained in:
willtanoe committed
1 parent
34bab45865
commit
111adb272e
15 files changed
+2431
No files matched your search
@@ -0,0 +1,335 @@
|
||||
# Post-hoc Statistical Audit of the PROTEAN Experiment Grid
|
||||
|
||||
Status: **read-only post-hoc audit of completed experiments.** No training, no partition
|
||||
regeneration, no modification of any existing result, manifest, dataset, configuration, source
|
||||
file, or test. All numbers below are recomputed from the audited artifacts by
|
||||
`scripts/posthoc_statistical_audit.py`, which writes new `posthoc__*.csv` tables under
|
||||
`artifacts/analysis/` and refuses to overwrite existing files.
|
||||
|
||||
Scope: determine what the completed 108-run grid can and cannot support statistically, and
|
||||
provide transparent, reusable wording for the thesis limitations.
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive findings
|
||||
|
||||
1. **The comparison is valid and paired.** All 18 `(dataset, alpha, seed)` cells pair the
|
||||
`original_protean` run with the validation-selected `support_weighted` run on an identical
|
||||
`manifest_content_hash`, split, and seed (`posthoc__paired_global_per_seed.csv`,
|
||||
`pairing_verified = True` for every row).
|
||||
2. **The paired differences are small and mostly seed-inconsistent.** For test rare-class
|
||||
macro-F1, only **X-IIoTID α=0.50** has all three seed differences in the same direction
|
||||
(**+0.0230** mean; paired-difference sample std **0.0058**); the other five conditions have
|
||||
mixed-sign seed differences.
|
||||
3. **No condition can reach statistical significance with three seeds.** The smallest attainable
|
||||
two-sided exact sign-flip p-value with three nonzero differences is **0.25**, and only
|
||||
X-IIoTID α=0.50 attains it. This is a **power/design limit**, not evidence of no effect.
|
||||
4. **The paired-difference variability is not the group variability.** At X-IIoTID α=0.50 the
|
||||
baseline and proposed cross-seed std are **0.0287** and **0.0267**, while the paired-difference
|
||||
std is **0.0058** — shared seed-to-seed variation cancels in the paired difference.
|
||||
5. **Selected beta is weakly separated and unstable across seeds.** The best-vs-second margin is
|
||||
only **0.0024–0.0122**, and the pooled selection agrees with at most **2 of 3** seeds in every
|
||||
condition (`posthoc__selection_stability.csv`).
|
||||
6. **The `S_j` participant rule is materially exercised**: `|S_j| < M` in **53 of 162**
|
||||
class-condition cells (verified). This is an implementation/reproduction assumption, not an
|
||||
ordinary statistical limitation.
|
||||
7. **Publication metrics could not be verified** because the referenced `journal/main.tex`
|
||||
(and the paper's Table IV) is **not present in this repository**. Reproduced baseline
|
||||
accuracies are reported instead; the discrepancy figures remain unverified.
|
||||
|
||||
---
|
||||
|
||||
## 2. Data sources and pairing validation
|
||||
|
||||
Primary sources (all committed, read-only):
|
||||
|
||||
| Purpose | Artifact |
|
||||
|---|---|
|
||||
| Paired per-seed global metrics | `artifacts/analysis/deepcmp__global_per_seed.csv` |
|
||||
| Per-run source of truth | `results/<run>/test_global_metrics.json`, `metadata.json` |
|
||||
| Validation per-seed / pooled | `artifacts/analysis/<ds>__alpha*__multiseed__validation_per_seed.csv`, `..._validation_pooled.csv`, `..._selection_summary.json` |
|
||||
| Selection margins | `artifacts/analysis/final__selection_strength.csv`, `final__pooled_validation_curves.csv` |
|
||||
| Per-class | `results/<run>/test_per_class.csv` → `deepcmp__per_class.csv` |
|
||||
| Per-client | `results/<run>/per_client.csv` → `deepcmp__per_client.csv` |
|
||||
| `S_j` rule | `artifacts/analysis/sj_participant_rule_analysis.csv` |
|
||||
| Reproduced baseline metrics | `docs/baseline/BASELINE_VALIDATION.md` |
|
||||
|
||||
Pairing rule enforced in the script: same `dataset`, `alpha`, `seed`, identical
|
||||
`manifest_content_hash`, and recorded `beta == validation-selected beta`. Result: **18/18 cells
|
||||
verified**, written with `pairing_verified` and the manifest hash into
|
||||
`posthoc__paired_global_per_seed.csv`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Descriptive paired results by condition
|
||||
|
||||
Test split, `delta = proposed_selected_beta − original_protean`, three paired seeds per condition.
|
||||
Source: `posthoc__paired_global_by_condition.csv`. The standard deviations below are three distinct
|
||||
quantities: `base_std` and `prop_std` are the cross-seed spread within each group, `paired_std` is
|
||||
computed directly from the three matched differences.
|
||||
|
||||
### 3.1 Rare-class macro-F1 (primary comparison)
|
||||
|
||||
| Dataset | alpha | beta* | Δ seed 0 | Δ seed 1 | Δ seed 2 | mean Δ | paired std | base std | prop std | +/−/0 | same sign |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| X-IIoTID | 0.25 | 0.00 | +0.0016 | −0.0020 | −0.0016 | −0.0007 | 0.0019 | 0.0472 | 0.0476 | 1/2/0 | no |
|
||||
| X-IIoTID | 0.50 | 0.75 | +0.0266 | +0.0163 | +0.0261 | **+0.0230** | 0.0058 | 0.0287 | 0.0267 | 3/0/0 | **yes** |
|
||||
| X-IIoTID | 0.75 | 0.50 | −0.0097 | +0.0101 | +0.0036 | +0.0014 | 0.0101 | 0.0778 | 0.0855 | 2/1/0 | no |
|
||||
| 5G-NIDD | 0.25 | 0.00 | −0.0050 | −0.0010 | +0.0029 | −0.0010 | 0.0039 | 0.0254 | 0.0242 | 1/2/0 | no |
|
||||
| 5G-NIDD | 0.50 | 1.00 | +0.0090 | −0.0811 | +0.0464 | −0.0086 | 0.0655 | 0.0371 | 0.0285 | 2/1/0 | no |
|
||||
| 5G-NIDD | 0.75 | 0.50 | −0.0011 | +0.0450 | +0.0067 | +0.0168 | 0.0247 | 0.0231 | 0.0145 | 2/1/0 | no |
|
||||
|
||||
**Verification of the reported `+0.0230`.** Source `final__six_condition_master.csv`
|
||||
(row X-IIoTID α=0.50): original rare-F1 mean 0.642520 (std **0.028683**), selected-beta mean
|
||||
0.665503 (std **0.026685**), difference **+0.022983**. The paired-difference std computed directly
|
||||
from the three matched seed differences is **0.0058**. The value is exact.
|
||||
|
||||
**What this does and does not establish.** X-IIoTID α=0.50 is the only condition where the
|
||||
improvement appears at every seed, and only there is the paired difference small relative to its own
|
||||
mean. It does **not** establish statistical significance (see §4), and it does not generalize: the
|
||||
same beta direction fails at 5G-NIDD α=0.50 and is mixed at four other conditions.
|
||||
|
||||
### 3.2 Other metrics
|
||||
|
||||
All five metrics (rare-class macro-F1, rare-class recall, macro-F1, weighted-F1, accuracy) with the
|
||||
same columns, including per-metric positive/negative/zero counts and sign agreement, are in
|
||||
`posthoc__paired_global_by_condition.csv`. Highlights:
|
||||
|
||||
- **Rare-class recall** is negative in **5 of 6** conditions (and in **all four** `beta>0`
|
||||
conditions): the rare-class F1 gains are precision-driven.
|
||||
- **Overall macro-F1** moves in the same direction as rare-class F1 in every condition, but its
|
||||
paired differences are equally seed-inconsistent.
|
||||
- **5G-NIDD α=0.50** has a paired-difference std of **0.0655**, larger than either group std,
|
||||
because the three seed differences have mixed sign (one large negative).
|
||||
|
||||
---
|
||||
|
||||
## 4. Inferential analysis and limitations
|
||||
|
||||
Sources: `posthoc__paired_test_results.csv`. Only three paired seeds exist per condition.
|
||||
|
||||
| Dataset | alpha | beta* | exact sign-flip p (2-sided) | exact Wilcoxon p (2-sided) | paired-t 95% CI | min attainable sign-flip p |
|
||||
|---|---|---|---|---|---|---|
|
||||
| X-IIoTID | 0.25 | 0.00 | 0.75 | 1.00 | [−0.0054, +0.0041] | 0.25 |
|
||||
| X-IIoTID | 0.50 | 0.75 | **0.25** | 0.50 | [+0.0085, +0.0374] | 0.25 |
|
||||
| X-IIoTID | 0.75 | 0.50 | 0.75 | 1.00 | [−0.0236, +0.0264] | 0.25 |
|
||||
| 5G-NIDD | 0.25 | 0.00 | 0.75 | 1.00 | [−0.0108, +0.0087] | 0.25 |
|
||||
| 5G-NIDD | 0.50 | 1.00 | 1.00 | 1.00 | [−0.1713, +0.1542] | 0.25 |
|
||||
| 5G-NIDD | 0.75 | 0.50 | 0.50 | 1.00 | [−0.0444, +0.0781] | 0.25 |
|
||||
|
||||
Test definitions and assumptions:
|
||||
|
||||
- **Exact sign-flip permutation test** on the mean of the paired differences. Assumption: under the
|
||||
null the sign of each paired difference is exchangeable. With `n = 3`, the null has `2^3 = 8`
|
||||
sign vectors, so the smallest attainable two-sided p-value is `2/8 = 0.25`; this was verified by
|
||||
enumeration, not assumed.
|
||||
- **Exact Wilcoxon signed-rank** (average ranks for ties, zero differences dropped, exact null by
|
||||
enumerating `2^n` sign assignments, two-sided min-rank convention). At `n = 3` the exact p-values
|
||||
are coarse (0.50 or 1.00), so this test adds little.
|
||||
- **Paired-t 95% CI** using `t = 4.3027` (df = 2). This is an explicitly assumption-dependent,
|
||||
underpowered summary; it is not a reliable interval estimate for a population.
|
||||
|
||||
Interpretation rules applied:
|
||||
|
||||
- **No condition is statistically significant**, and none can be with three seeds.
|
||||
- **Non-significance is not evidence of no effect**; it reflects a sample far too small to resolve
|
||||
effects of this size.
|
||||
- Tests are **not** pooled across the six conditions (they are not replicates of one condition) nor
|
||||
across clients (clients are nested in the same trained model and seed).
|
||||
- Bootstrap CIs are deliberately **not** used: three seeds cannot support a reliable resampling
|
||||
distribution.
|
||||
|
||||
Conclusion: **prioritize descriptive reporting** (values, direction, seed consistency) over
|
||||
inferential claims for this grid.
|
||||
|
||||
---
|
||||
|
||||
## 5. Beta-selection stability
|
||||
|
||||
Source: `posthoc__selection_stability.csv`; per-seed selections are computed from each condition's
|
||||
`validation_per_seed.csv` (per-client-mean validation rare-class macro-F1).
|
||||
|
||||
| Dataset | alpha | pooled beta | per-seed beta (s0/s1/s2) | agree with pooled | best score | 2nd-best (beta, score) | margin | candidate range |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| X-IIoTID | 0.25 | 0.00 | 0.75 / 0.50 / 0.00 | 1/3 | 0.5917 | 0.25, 0.5887 | 0.0030 | 0.0596 |
|
||||
| X-IIoTID | 0.50 | 0.75 | 0.50 / 1.00 / 0.75 | 1/3 | 0.5794 | 1.00, 0.5735 | 0.0059 | 0.0350 |
|
||||
| X-IIoTID | 0.75 | 0.50 | 0.50 / 0.50 / 0.00 | 2/3 | 0.4767 | 0.25, 0.4744 | 0.0024 | 0.0081 |
|
||||
| 5G-NIDD | 0.25 | 0.00 | 0.25 / 0.00 / 0.00 | 2/3 | 0.7615 | 0.25, 0.7493 | 0.0122 | 0.0392 |
|
||||
| 5G-NIDD | 0.50 | 1.00 | 1.00 / 0.00 / 1.00 | 2/3 | 0.6924 | 0.75, 0.6884 | 0.0040 | 0.0473 |
|
||||
| 5G-NIDD | 0.75 | 0.50 | 0.50 / 1.00 / 1.00 | 1/3 | 0.8091 | 1.00, 0.8043 | 0.0048 | 0.0310 |
|
||||
|
||||
Verification: the reported selection margins of **0.0024–0.0122** are confirmed exactly.
|
||||
|
||||
**Interpretation.** A margin of 0.002–0.012 is small relative to both the candidate-score range
|
||||
(0.008–0.060) and the across-seed std of the selected beta (0.022–0.139 in earlier analyses), and the
|
||||
pooled selection agrees with only 1–2 of 3 seeds. This makes the selection **fragile / weakly
|
||||
separated**: it should be described as "best among the tested candidates under the pooled validation
|
||||
procedure", **not** as an optimal or robust choice. A small margin does **not** prove the selected
|
||||
beta is wrong; it means the data cannot discriminate among close candidates. All selections remain
|
||||
**validation-only** (`test_used_for_selection = False`); test metrics are never used to select beta.
|
||||
|
||||
---
|
||||
|
||||
## 6. `S_j` participant-rule audit
|
||||
|
||||
Source: `artifacts/analysis/sj_participant_rule_analysis.csv` (162 class-condition cells = 18
|
||||
condition-seeds × 9 classes). Verified: **`|S_j| < M` in 53 / 162 cells** (X-IIoTID 19/7/3 and
|
||||
5G-NIDD 16/4/4 for α = 0.25/0.50/0.75); all 53 are exactly the cells with ≥1 missing client
|
||||
(`missing_clients` non-empty). `S_j` is the set of clients with `N_ij > 0` and a valid prototype.
|
||||
|
||||
**Why this is an implementation/reproduction assumption, not an ordinary statistical limitation.**
|
||||
Eq. (2)/(5.5) of the source paper sums over `M` clients but is silent on `N_ij = 0`; no official
|
||||
implementation exists (see `docs/baseline/PROTEAN_BASELINE_INVESTIGATION.md`). The project therefore
|
||||
*froze* `S_j = {i : N_ij > 0 and C_ij exists}` and used equal `1/|S_j|` weights in the baseline. In
|
||||
`53/162` cells the literal `1/M` reading (e.g. substituting a zero vector for an absent prototype)
|
||||
would differ from the `S_j` reading, so all results are **conditional on this rule**. More seeds
|
||||
would not remove this; only the (unavailable) official code, or a documented threat/robustness
|
||||
analysis across alternative participant rules, could.
|
||||
|
||||
---
|
||||
|
||||
## 7. Baseline reproduction discrepancy audit
|
||||
|
||||
**Reproduced baseline metrics available in this repository**
|
||||
(`docs/baseline/BASELINE_VALIDATION.md`, and `final__six_condition_master.csv`):
|
||||
|
||||
| Evidence | Value |
|
||||
|---|---|
|
||||
| X-IIoTID α=0.50 seed 0 accuracy | 0.8904 (macro-F1 0.7077, rare-F1 0.6096) |
|
||||
| 5G-NIDD α=0.50 seed 0 accuracy | 0.7344 (macro-F1 0.8369, rare-F1 0.8720) |
|
||||
| X-IIoTID original accuracy means (α=0.25/0.50/0.75) | 0.8754 / 0.8995 / 0.9206 |
|
||||
| 5G-NIDD original accuracy means (α=0.25/0.50/0.75) | 0.7310 / 0.7440 / 0.7510 |
|
||||
|
||||
**Unresolved data gap.** The task's comparison uses the paper's reported metrics ("Table IV") in
|
||||
`journal/main.tex`. That file and directory **do not exist in this repository**, and no file records
|
||||
the paper's numeric accuracy/F1 table (the baseline investigation records only that the paper reports
|
||||
"macro accuracy, accuracy, F1, precision", without values). Therefore the claimed deviations
|
||||
(X-IIoTID ≈ 5.1–9.7 percentage points below the paper; 5G-NIDD ≈ 4.6–8.4 percentage points above the
|
||||
paper) **could not be verified** here. Per the constraints, this is documented as a gap rather than
|
||||
reconstructed.
|
||||
|
||||
**Classification.** Requires the external source (`journal/main.tex` / the published paper and
|
||||
thesis). Even once sourced, such deviations must be reported as **reproduction differences**, not as
|
||||
automatic evidence of an implementation bug: metric definitions, train/test split (the project uses
|
||||
70/10/20 stratified, the paper 80/20), class handling (one dropped class on X-IIoTID), seed
|
||||
initialization, and nearest-prototype inference all differ from the paper and can move accuracy
|
||||
independently of any defect. No defect is claimed.
|
||||
|
||||
---
|
||||
|
||||
## 8. Metric-selection versus headline-metric audit
|
||||
|
||||
Traced in code:
|
||||
|
||||
- **Selection metric** (`scripts/run_baseline.py`, line ~303): `val_rare_class_macro_f1_per_client_mean`
|
||||
— the mean over eligible clients of each client's own validation rare-class macro-F1. Pooled
|
||||
selection across seeds uses this (`scripts/analyze_multiseed_beta.py`).
|
||||
- **Headline test metric**: `test_rare_class_macro_f1` from `rare_class_metrics(test_labels, test_pred,
|
||||
num_classes, union_rare)` — a **global** rare-class macro-F1 pooled over the union of client rare
|
||||
classes.
|
||||
|
||||
**Consequence.** The method is *selected* on a **per-client-mean validation** rare-class macro-F1 but
|
||||
*reported* on a **global test** rare-class macro-F1. These are not the same estimator: the per-client
|
||||
mean weights every client equally (and can be dominated by small clients), whereas the global metric
|
||||
weights by evaluation support. A beta that is best under the selection criterion need not be best
|
||||
under the reported criterion. The two quantities are therefore reported separately in this thesis
|
||||
(validation selection tables vs test evaluation tables) and must never be conflated; neither uses the
|
||||
test set for selection.
|
||||
|
||||
---
|
||||
|
||||
## 9. What can and cannot be concluded
|
||||
|
||||
**Supported by the evidence.**
|
||||
|
||||
- The comparison is a valid paired design with verified common manifests and splits.
|
||||
- Test rare-class macro-F1 changes are **small (≤ ~0.023)** and **condition-dependent**; only
|
||||
X-IIoTID α=0.50 improves in all three seeds.
|
||||
- Rare-class recall tends to fall when rare-class F1 rises (precision–recall trade-off).
|
||||
- The validation-selected beta is **weakly separated** and its pooled choice agrees with at most 2/3
|
||||
seeds; it is "best among tested candidates", not robust or optimal.
|
||||
- The `S_j` rule is materially exercised (53/162 cells) and conditions all results.
|
||||
|
||||
**Not supported / cannot be concluded.**
|
||||
|
||||
- Statistical superiority or significance (min attainable sign-flip p = 0.25).
|
||||
- That the method has *no* effect (non-significance ≠ equivalence).
|
||||
- A universal or globally optimal beta.
|
||||
- That prototype displacement implies better detection.
|
||||
- Any cross-condition pooling as if the 18 cells were independent replicates.
|
||||
- The paper-comparison deviation magnitudes (source absent).
|
||||
- Any privacy claim from the `N_ij` flow without a threat model.
|
||||
|
||||
---
|
||||
|
||||
## 10. Recommended replacement wording for thesis section A.7 (Indonesian)
|
||||
|
||||
> **Keterbatasan bukti statistik.** Grid eksperimen yang telah diselesaikan terdiri atas enam
|
||||
> kondisi (dua dataset × tiga nilai `α`) dengan tiga seed per kondisi, sehingga setiap perbandingan
|
||||
> berpasangan antara `original_protean` dan metode usulan hanya memiliki tiga selisih. Dengan tiga
|
||||
> selisih tersebut, uji permutasi tanda dua sisi yang eksak sekalipun tidak dapat menghasilkan nilai
|
||||
> p di bawah 0,25; karena itu hasil eksperimen ini dilaporkan secara deskriptif (nilai tengah,
|
||||
> arah perubahan, dan konsistensi antar-seed), bukan sebagai klaim signifikansi. Ketidaksignifikanan
|
||||
> tidak boleh ditafsirkan sebagai bukti bahwa metode tidak berpengaruh, melainkan sebagai
|
||||
> keterbatasan daya uji akibat jumlah seed yang kecil. Perbedaan hasil yang teramati juga kecil dan
|
||||
> bergantung pada kondisi; hanya kondisi X-IIoTID `α`=0,50 yang konsisten membaik pada ketiga seed.
|
||||
> Pemilihan `β` dilakukan sepenuhnya pada data validasi, dengan margin antara kandidat terbaik dan
|
||||
> kedua terbaik yang kecil (0,0024–0,0122), sehingga nilai `β` terpilih dinyatakan sebagai yang
|
||||
> terbaik di antara kandidat yang diuji pada prosedur validasi, bukan sebagai nilai optimal global
|
||||
> maupun pilihan yang kokoh. Selain itu, aturan peserta agregasi `S_j` merupakan asumsi reproduksi
|
||||
> yang secara material berpengaruh pada 53 dari 162 sel kelas-kondisi; seluruh hasil bersifat
|
||||
> kondisional terhadap asumsi tersebut. Generalisasi juga terbatas pada dua dataset, sepuluh
|
||||
> client, satu arsitektur, tiga tingkat `α`, tiga seed, dan lima kandidat `β` diskret.
|
||||
|
||||
*(This is proposed wording for review; no existing thesis document was edited.)*
|
||||
|
||||
---
|
||||
|
||||
## 11. Reproducibility
|
||||
|
||||
```bash
|
||||
# post-hoc audit (read-only; no training; writes posthoc__*.csv only)
|
||||
.venv/bin/python scripts/posthoc_statistical_audit.py
|
||||
```
|
||||
|
||||
The script:
|
||||
1. re-verifies pairing (dataset, alpha, seed, `manifest_content_hash`, recorded beta) for all 18
|
||||
cells and aborts on any mismatch;
|
||||
2. writes `posthoc__paired_global_per_seed.csv`, `posthoc__paired_global_by_condition.csv`,
|
||||
`posthoc__paired_test_results.csv`, `posthoc__selection_stability.csv`,
|
||||
`posthoc__per_class_summary.csv`, `posthoc__per_client_summary.csv`;
|
||||
3. refuses to overwrite any existing artifact.
|
||||
|
||||
Inputs are the committed `deepcmp__*`, `final__*`, `*__multiseed__validation_per_seed.csv`, and
|
||||
`sj_participant_rule_analysis.csv` tables plus per-run `metadata.json`. No network, no training, no
|
||||
partition regeneration.
|
||||
|
||||
---
|
||||
|
||||
## 12. Unresolved data gaps
|
||||
|
||||
1. **`journal/main.tex` / paper Table IV absent** — the paper's reported accuracy/F1 values cannot
|
||||
be verified from the repository; the reproduction-deviation magnitudes remain unconfirmed.
|
||||
2. **No official PROTEAN code** — the `S_j` participant rule and missing-class handling remain a
|
||||
reproduction assumption (material in 53/162 cells).
|
||||
3. **Three seeds only** — inferential power is fundamentally limited; no significant result is
|
||||
attainable and bootstrap/interval estimates would be unreliable.
|
||||
4. **One missing validation artifact set** for `xiiotid__alpha0.5__seed0__original_protean`
|
||||
(documented in the README) — does not affect the six selections but limits that cell's
|
||||
validation-side diagnostics.
|
||||
5. **Metric mismatch** between the per-client validation selection criterion and the global test
|
||||
headline metric cannot be removed post-hoc; it is a design/protocol property.
|
||||
|
||||
---
|
||||
|
||||
## Appendix — Limitations classification (task Phase 5)
|
||||
|
||||
| Item | Classification |
|
||||
|---|---|
|
||||
| A. Limited seeds / statistical evidence | Computable (done: descriptive + exact tests) and a **design-scope limitation** for significance |
|
||||
| B. `S_j` participant rule (53/162) | **Requires source-code/protocol verification** (no official code) → documented reproduction assumption |
|
||||
| C. Baseline reproduction discrepancy | **Requires the external source** (`journal/main.tex` / paper table; absent here); report as reproduction difference, not a defect |
|
||||
| D. Selection margins and seed agreement | **Computable** (done; margins 0.0024–0.0122 verified) |
|
||||
| E. Selection metric vs headline metric | **Computable/traceable in code** (confirmed: validation per-client mean vs global test rare-F1) |
|
||||
| F. Experimental scope (2 datasets, 10 clients, 1 architecture, 3 α, 3 seeds, 5 betas) | **Design-scope limitation** — cannot be removed post-hoc; needs additional experiments |
|
||||
| G. Support information / privacy (`N_ij` at server) | **Requires protocol verification (done: `N_ij` is sent) + threat-model analysis** for any privacy claim; no vulnerability is claimed here |
|
||||
Reference in new issue
Block a user