diff --git a/docs/handoff/SESSION_HANDOFF.md b/docs/handoff/SESSION_HANDOFF.md index 0ab48f3..434cc8b 100644 --- a/docs/handoff/SESSION_HANDOFF.md +++ b/docs/handoff/SESSION_HANDOFF.md @@ -336,11 +336,18 @@ Validation-selected beta (seed 0) and test rare-class macro-F1 (global) afterwar confounded** with host (8 threads / 29 GiB documented earlier vs 12 / 48 GiB now), possibly a different PyTorch version (not recorded in metadata), and a different checkout path. A sign test (p = 0.86 / 1.0) is reported only descriptively because the pairs are not independent. -- **Write-up:** `docs/research/SWEEP_1THREAD_SEED0_CHECK.md` (untracked, not yet committed) - summarizes the sweep, the run-to-run difference, the noise checks and the comparison above. +- **Write-up:** `docs/research/SWEEP_1THREAD_SEED0_CHECK.md` (committed in `ad203eb`; section 3.6 + was added afterwards and is uncommitted) summarizes the sweep, the grid-vs-sweep difference, + the noise checks and the comparison above. +- **Repeat / multi-thread test** (`results/sweep_repro/`, X-IIoTID alpha 0.25 seed 0 beta 0 only): + a 1-thread repeat is bit-identical to the sweep run (all result JSON and the prototype + snapshot), so the 1-thread sweep has no run-to-run noise. An 8-thread run on this host differs + from the 1-thread run (test rare-F1 1.9e-3) and also does not reproduce the old grid (2.5e-3). + Thread count changes results but does not explain the whole grid-vs-sweep difference; host and + PyTorch version remain uncontrolled. 5G-NIDD and an 8-thread repeat were not tested. - **Exact next task:** decide with the user whether the 5G-NIDD alpha 0.5 / 0.75 disagreement - warrants investigating the thread-count hypothesis (needs extra runs; not started) and whether - to commit the handoff and the write-up. Do not change the primary pooled selection on the basis of this single-seed re-run. + warrants a 5G-NIDD repeat / multi-thread test (needs extra runs; not started) and whether to + commit section 3.6. Do not change the primary pooled selection on the basis of this single-seed re-run. - **Housekeeping:** the tmux session `sweep` still exists (windows `control`, `pool`); the `heartbeat` window was stopped on 2026-10-29. Backups of edited sweep scripts are `*.bak` in `results/sweep/`; orphaned attempt-1 artifacts are in `results/sweep/_orphans_20261027T011213Z/`. diff --git a/docs/research/SWEEP_1THREAD_SEED0_CHECK.md b/docs/research/SWEEP_1THREAD_SEED0_CHECK.md index 397915d..1f42098 100644 --- a/docs/research/SWEEP_1THREAD_SEED0_CHECK.md +++ b/docs/research/SWEEP_1THREAD_SEED0_CHECK.md @@ -44,8 +44,10 @@ choice and the grid's pooled choice, using validation metrics only. | test rare-class macro-F1 | 0.0022 | 0.0127 (5G-NIDD, alpha 0.5, beta 0.75) | | validation selection metric | 0.0029 | 0.0328 | -None of the 30 pairs is exactly equal. The cause is not established. Thread count is a -hypothesis only; no experiment isolates it. +None of the 30 pairs is exactly equal. The cause is not established (section 3.6 tests thread +count for one configuration). Because the 1-thread setup turned out to be exactly reproducible +(section 3.6), "run-to-run" in sections 3.1 to 3.4 means earlier grid versus 1-thread sweep: a +difference between setups, not noise inside one setup. ### 3.2 Seed-0 selection @@ -140,17 +142,46 @@ one partition), so the sign counts below are descriptive only. metadata stores no PyTorch version, so a library difference cannot be excluded (current: 2.14.1+cpu). Whether the grid ran on a different machine is inferred, not verified. +### 3.6 Repeat and multi-thread test (X-IIoTID alpha 0.25 seed 0 beta 0; `results/sweep_repro/`) + +Two extra runs of one configuration on this host: same partition (hash unchanged), same +hyperparameters, same 11 GiB memory scope. A: 1 thread through `thread1_launcher.py`, identical +to the sweep command (CPU 99 %, wall 9 704 s). B: 8 threads through `run_baseline.py` directly +(CPU 758 %, wall 4 546 s). Script: `results/sweep_repro/compare_repro.py`. + +| pair | test rare-F1 abs diff | val selection metric abs diff | last-round rare-F1 diff | all result JSON equal | prototype snapshot byte-identical | +|---|---|---|---|---|---| +| A (1 thread repeat) vs sweep | 0 | 0 | 0 | yes | yes | +| A (= sweep) vs grid | 6.2e-4 | 1.27e-3 | 3.4e-4 | no | no | +| B (8 threads) vs A (= sweep) | 1.86e-3 | 4.6e-4 | 1.9e-3 | no | no | +| B (8 threads) vs grid | 2.47e-3 | 8.0e-4 | 2.2e-3 | no | no | + +- **The 1-thread setup is exactly reproducible** (one configuration, one repeat): every compared + result file and the prototype snapshot of A equal the sweep run. The sweep therefore contains no + random run-to-run noise; the grid-vs-sweep differences are deterministic consequences of the + setup, not luck of a single run. +- **Thread count changes the result.** B differs from A by 1.86e-3 on test rare-F1, the same order + as the median grid-vs-sweep difference (2.2e-3). One pair only. +- **Thread count alone does not reproduce the grid.** B is not identical to the grid either. On + test rare-F1 it is farther from the grid (2.47e-3) than the 1-thread run is (6.2e-4); on the + validation metric it is slightly closer (8.0e-4 vs 1.27e-3). Host, PyTorch version and the grid's + exact thread count (CPU 741 to 746 % suggests about 7 to 8, not verified) remain uncontrolled. +- **Not tested:** whether the 8-thread setup reproduces itself (needs an 8-thread repeat); any + other configuration, in particular 5G-NIDD, where the two beta disagreements occur. This + configuration's grid-vs-sweep difference (6.2e-4) is below the median over all 30 pairs, so it + is a favourable case for agreement and says little about the 5G-NIDD cases. + ## 4. Interpretation limits - One seed. The grid pools three. Nothing here is a significance test. - Where the sweep and the grid disagree (5G-NIDD alpha 0.5 and 0.75), a different beta is the best of the tested candidates under a slightly different realization of training; this does not show that either beta is better. -- Unverified: that thread count causes the run-to-run difference (it is confounded with hardware - and possibly library version, section 3.5). Two cheap runs would separate them: (a) repeat one - 1-thread run and compare it bit-for-bit with the sweep run (is a repeat identical?); (b) run - the same configuration multi-threaded on this host and compare with the grid. Neither has - been run. +- Thread count as the cause of the grid-vs-sweep difference is partly supported but not + established (section 3.6): threading changes the result, yet 8 threads on this host does not + reproduce the grid. Hardware, PyTorch version and the grid's exact thread count stay + uncontrolled. One configuration (X-IIoTID alpha 0.25 beta 0) was tested; an 8-thread repeat and + 5G-NIDD were not. - No claim is made that support weighting improves detection, and the selected beta is "best among the tested candidates under the defined validation procedure", not a global optimum. - `S_j` handling remains the documented reproduction assumption.