docs: record 1-thread seed-0 sweep, noise checks and grid comparison
This commit is contained in:
willtanoe committed
1 parent
91b0e371f7
commit
ad203eb2ee
2 files changed
+245
No files matched your search
@@ -0,0 +1,166 @@
|
||||
# 1-thread seed-0 re-run: validation selection vs. the 108-run grid
|
||||
|
||||
Status: **exploratory consistency check**, single seed. It does not replace, correct or
|
||||
re-select anything in `FINAL_EXPERIMENTAL_SYNTHESIS.md`. The primary selection of the thesis
|
||||
remains the pooled validation across clients and seeds of the 108-run grid.
|
||||
|
||||
## 1. What was run
|
||||
|
||||
- 30 runs: 2 datasets (X-IIoTID, 5G-NIDD) x 3 alpha (0.25, 0.5, 0.75) x 5 beta
|
||||
(0, 0.25, 0.5, 0.75, 1), `seed = 0`, `support_weighted` mode.
|
||||
- 1 thread per training process, 3 processes in parallel. 2026-10-27T01:13Z to
|
||||
2026-10-28T18:00Z (2026-10-27 08:13 to 2026-10-29 01:00 WIB), about 40.8 h.
|
||||
- All 30 runs ended `completed exit 0` with a `COMPLETE` marker; 0 failed, retry pass was empty.
|
||||
- Outputs live in the git-ignored `results/sweep/` (runs, `summary.csv`, `analysis/`). They are
|
||||
not committed. The orchestration record (including a gate-deadlock fix) is
|
||||
`results/sweep/relaunch_report.md`, also untracked.
|
||||
|
||||
Before the main queue continued, `reference_check.py` compared the first two runs (X-IIoTID
|
||||
alpha 0.25 seed 0, beta 0 and 1) with the audit values; the maximum difference to the manual
|
||||
verdict was 4.3e-10 and every |z| was at most 0.12.
|
||||
|
||||
## 2. Method
|
||||
|
||||
`results/sweep/analyze_sweep.py` (read-only, deterministic; two executions gave byte-identical
|
||||
CSVs):
|
||||
|
||||
1. **Stage 1, validation only.** For each (dataset, alpha) choose the beta with the highest
|
||||
`val_rare_class_macro_f1_per_client_mean`; ties go to the smaller beta. `selection.csv` is
|
||||
written before any test file is opened.
|
||||
2. **Stage 2.** Test rare-class macro-F1 at every beta, read only after stage 1.
|
||||
(`test_at_selected_beta.csv`.) Descriptive; not used to choose anything.
|
||||
3. **Stage 3.** Absolute differences to the earlier grid run of the same
|
||||
(dataset, alpha, seed 0, beta). (`repro_vs_previous.csv`.)
|
||||
|
||||
`results/sweep/compare_with_grid.py` then compares the sweep's choice with the grid's seed-0
|
||||
choice and the grid's pooled choice, using validation metrics only.
|
||||
|
||||
## 3. Results
|
||||
|
||||
### 3.1 Run-to-run difference (30 pairs, same configuration, earlier run vs. 1-thread run)
|
||||
|
||||
| metric | median abs. difference | max abs. difference |
|
||||
|---|---|---|
|
||||
| test rare-class macro-F1 | 0.0022 | 0.0127 (5G-NIDD, alpha 0.5, beta 0.75) |
|
||||
| validation selection metric | 0.0029 | 0.0328 |
|
||||
|
||||
None of the 30 pairs is exactly equal. The cause is not established. Thread count is a
|
||||
hypothesis only; no experiment isolates it.
|
||||
|
||||
### 3.2 Seed-0 selection
|
||||
|
||||
| dataset | alpha | beta (sweep) | beta (grid, seed 0) | beta (grid, pooled) | sweep margin | grid s0 margin |
|
||||
|---|---|---|---|---|---|---|
|
||||
| X-IIoTID | 0.25 | 0.75 | 0.75 | 0 | 0.0413 | 0.0377 |
|
||||
| X-IIoTID | 0.50 | 0.50 | 0.50 | 0.75 | 0.0074 | 0.0078 |
|
||||
| X-IIoTID | 0.75 | 0.50 | 0.50 | 0.50 | 0.0004 | 0.0075 |
|
||||
| 5G-NIDD | 0.25 | 0.25 | 0.25 | 0 | 0.0155 | 0.0109 |
|
||||
| 5G-NIDD | 0.50 | 0.75 | 1.00 | 1.00 | 0.0053 | 0.0044 |
|
||||
| 5G-NIDD | 0.75 | 0.25 | 0.50 | 0.50 | 0.0132 | 0.0021 |
|
||||
|
||||
"Margin" is the validation score of the best beta minus that of the runner-up.
|
||||
|
||||
- The sweep and grid seed-0 choices coincide in **4 of 6** conditions.
|
||||
- The two disagreements are both 5G-NIDD. At alpha 0.5 the sweep's beta is rank 3 in the grid's
|
||||
seed-0 ranking; at alpha 0.75 it is rank 4. These two conditions also show the largest
|
||||
per-beta validation differences between the two runs (0.0328 and 0.0262).
|
||||
- The sweep beta equals the grid **pooled** beta in 1 of 6, but the grid's own seed-0 beta
|
||||
equals its pooled beta in only 3 of 6. A seed-0 versus pooled mismatch is therefore expected
|
||||
and is not a measure of reproducibility.
|
||||
- X-IIoTID alpha 0.75 has a sweep margin of 0.0004, far below the median validation run-to-run
|
||||
difference (0.0029), yet its choice coincides with the grid's. The narrow margin did not
|
||||
produce a different choice here.
|
||||
|
||||
### 3.3 Test rare-class macro-F1 at the validation-selected beta (descriptive, seed 0)
|
||||
|
||||
| dataset | alpha | selected beta | test @ selected | test @ beta 0 |
|
||||
|---|---|---|---|---|
|
||||
| X-IIoTID | 0.25 | 0.75 | 0.6692 | 0.6420 |
|
||||
| X-IIoTID | 0.50 | 0.50 | 0.6439 | 0.6131 |
|
||||
| X-IIoTID | 0.75 | 0.50 | 0.5068 | 0.5263 |
|
||||
| 5G-NIDD | 0.25 | 0.25 | 0.8876 | 0.8612 |
|
||||
| 5G-NIDD | 0.50 | 0.75 | 0.8793 | 0.8694 |
|
||||
| 5G-NIDD | 0.75 | 0.25 | 0.9065 | 0.9024 |
|
||||
|
||||
Selected minus beta 0: +0.0272, +0.0308, -0.0195 (X-IIoTID alpha 0.25 / 0.5 / 0.75) and
|
||||
+0.0264, +0.0099, +0.0042 (5G-NIDD alpha 0.25 / 0.5 / 0.75). In 5 of 6 conditions the test value
|
||||
at the validation-selected beta is above that at beta 0; in X-IIoTID alpha 0.75 it is below.
|
||||
|
||||
Four gaps (absolute value 0.0195 to 0.0308) exceed the largest test run-to-run difference seen
|
||||
in section 3.1 (0.0127). The two 5G-NIDD gaps at alpha 0.5 and 0.75 (0.0099 and 0.0042) lie
|
||||
within the range of that difference (median 0.0022, max 0.0127) and should not be read as
|
||||
improvements. The run-to-run difference compares two runs of one configuration; it is a rough
|
||||
yardstick, not a noise bound for a difference between two betas.
|
||||
|
||||
### 3.4 Noise checks (`results/sweep/check_noise.py` -> `analysis/noise_checks.json`)
|
||||
|
||||
Read-only, no training. The 30 pairs are not independent (the five betas of a condition share
|
||||
one partition), so the sign counts below are descriptive only.
|
||||
|
||||
- **Where the runs first differ.** All 30 pairs already differ at round 0 (max absolute
|
||||
difference in rare-class macro-F1 1.8e-3; 0 of 30 exactly equal). The number of evaluated
|
||||
samples is identical in every round of every pair, so data and partitions are the same. The
|
||||
difference is small at round 0 (median 3.1e-4) and **grows over the rounds** (median 2.1e-3 and
|
||||
max 1.5e-2 at the last round; larger at the last round than at round 0 in 24 of 30 pairs).
|
||||
A tiny first-round difference that is amplified by later rounds fits the pattern; it does not
|
||||
identify what produced the first-round difference.
|
||||
- **Direction.** Test rare-F1, sweep minus grid: 14 higher, 16 lower, mean signed -0.0004 against
|
||||
mean absolute 0.0034. Validation selection metric: 15 higher, 15 lower, mean signed -0.0004
|
||||
against mean absolute 0.0057. The differences scatter around zero; there is no sign of a
|
||||
systematic offset. (A two-sided sign test gives p = 0.86 and 1.0, but the pairs are dependent,
|
||||
so this is not a significance claim.)
|
||||
- **By dataset.** Mean absolute validation difference: X-IIoTID 0.0031, 5G-NIDD 0.0082 (about
|
||||
2.7 times larger). Test: 0.0026 vs 0.0042 (about 1.6 times). 5G-NIDD is the noisier dataset
|
||||
in this comparison, consistent with both disagreements sitting there.
|
||||
- **Ranking stability** (Spearman over the five betas, sweep vs grid, validation): X-IIoTID
|
||||
1.00 / 0.90 / 1.00 and 5G-NIDD 0.90 / 0.50 / 0.30 for alpha 0.25 / 0.5 / 0.75. Betas whose
|
||||
sweep validation score lies within the condition's own maximum run-to-run difference of the
|
||||
best beta (a crude yardstick): X-IIoTID alpha 0.25 {0.75}; alpha 0.5 {0.5, 0.75}; alpha 0.75
|
||||
{0, 0.25, 0.5}; 5G-NIDD alpha 0.25 {0.25}; alpha 0.5 {0, 0.75, 1}; alpha 0.75 {0, 0.25, 0.5}.
|
||||
Selections with one beta in that set (X-IIoTID alpha 0.25, 5G-NIDD alpha 0.25) are the least
|
||||
ambiguous; the others cannot separate their top candidates from the noise.
|
||||
- **Third reference point** (X-IIoTID alpha 0.25 seed 0, test rare-F1; from the earlier
|
||||
5-thread smoke run recorded in `reference_check.json`): beta 0: smoke 0.6402, grid 0.6414,
|
||||
sweep 0.6420; beta 1: smoke 0.5721, grid 0.5841, sweep 0.5931. For beta 1 the three setups
|
||||
span 0.021, larger than the sweep-vs-grid difference alone (0.0089).
|
||||
|
||||
### 3.5 What the thread hypothesis rests on
|
||||
|
||||
- The earlier grid used several cores per process (CPU 741 to 746 % in the X-IIoTID alpha 0.25
|
||||
runtime table, about 7.4 cores); the sweep forces 1 thread (`thread1_launcher.py`,
|
||||
`NLWP = 1` observed). Multi-threaded reductions can change floating-point summation order, so
|
||||
a thread effect is plausible. `BASELINE_VALIDATION.md` (F.2) reports that two invocations of
|
||||
`run_federated` with the same seed gave exactly equal results in a regression test on a small
|
||||
fixture, and that no nondeterminism source was observed at the tested scales on this CPU
|
||||
backend. That says a same-setup repeat was deterministic there; it does not show that full-size
|
||||
runs are bit-reproducible, and it does not test the thread hypothesis.
|
||||
- Confounders that were **not** controlled: the planning document recorded 8 CPU threads and
|
||||
about 29 GiB RAM for the earlier work, whereas this host has `nproc = 12` and 48 GiB; the old
|
||||
run metadata records a different repository path (`.../projects/perdalam-protean`); the
|
||||
metadata stores no PyTorch version, so a library difference cannot be excluded (current:
|
||||
2.14.1+cpu). Whether the grid ran on a different machine is inferred, not verified.
|
||||
|
||||
## 4. Interpretation limits
|
||||
|
||||
- One seed. The grid pools three. Nothing here is a significance test.
|
||||
- Where the sweep and the grid disagree (5G-NIDD alpha 0.5 and 0.75), a different beta is the
|
||||
best of the tested candidates under a slightly different realization of training; this does
|
||||
not show that either beta is better.
|
||||
- Unverified: that thread count causes the run-to-run difference (it is confounded with hardware
|
||||
and possibly library version, section 3.5). Two cheap runs would separate them: (a) repeat one
|
||||
1-thread run and compare it bit-for-bit with the sweep run (is a repeat identical?); (b) run
|
||||
the same configuration multi-threaded on this host and compare with the grid. Neither has
|
||||
been run.
|
||||
- No claim is made that support weighting improves detection, and the selected beta is "best
|
||||
among the tested candidates under the defined validation procedure", not a global optimum.
|
||||
- `S_j` handling remains the documented reproduction assumption.
|
||||
|
||||
## 5. Reproduce
|
||||
|
||||
```bash
|
||||
.venv/bin/python results/sweep/analyze_sweep.py
|
||||
.venv/bin/python results/sweep/compare_with_grid.py
|
||||
```
|
||||
|
||||
Both only read; they write into `results/sweep/analysis/`. The scripts and CSVs are untracked
|
||||
because `results/` is git-ignored.
|
||||
Reference in new issue
Block a user