diff --git a/docs/handoff/SESSION_HANDOFF.md b/docs/handoff/SESSION_HANDOFF.md index 434cc8b..b148bef 100644 --- a/docs/handoff/SESSION_HANDOFF.md +++ b/docs/handoff/SESSION_HANDOFF.md @@ -344,10 +344,13 @@ Validation-selected beta (seed 0) and test rare-class macro-F1 (global) afterwar snapshot), so the 1-thread sweep has no run-to-run noise. An 8-thread run on this host differs from the 1-thread run (test rare-F1 1.9e-3) and also does not reproduce the old grid (2.5e-3). Thread count changes results but does not explain the whole grid-vs-sweep difference; host and - PyTorch version remain uncontrolled. 5G-NIDD and an 8-thread repeat were not tested. + PyTorch version remain uncontrolled. The same test was also run on 5G-NIDD alpha 0.5 beta 0.75 + (the largest disagreement): the 1-thread repeat is again bit-identical to the sweep, and the + 8-thread run again reproduces neither (closer to the sweep on test rare-F1, closer to the grid + on the validation metric). Still untested: an 8-thread repeat (its own stability). - **Exact next task:** decide with the user whether the 5G-NIDD alpha 0.5 / 0.75 disagreement - warrants a 5G-NIDD repeat / multi-thread test (needs extra runs; not started) and whether to - commit section 3.6. Do not change the primary pooled selection on the basis of this single-seed re-run. + warrants an 8-thread repeat (is the multi-thread setup itself stable?) and whether to commit + section 3.7. Do not change the primary pooled selection on the basis of this single-seed re-run. - **Housekeeping:** the tmux session `sweep` still exists (windows `control`, `pool`); the `heartbeat` window was stopped on 2026-10-29. Backups of edited sweep scripts are `*.bak` in `results/sweep/`; orphaned attempt-1 artifacts are in `results/sweep/_orphans_20261027T011213Z/`. diff --git a/docs/research/SWEEP_1THREAD_SEED0_CHECK.md b/docs/research/SWEEP_1THREAD_SEED0_CHECK.md index 1f42098..c72a512 100644 --- a/docs/research/SWEEP_1THREAD_SEED0_CHECK.md +++ b/docs/research/SWEEP_1THREAD_SEED0_CHECK.md @@ -171,6 +171,40 @@ to the sweep command (CPU 99 %, wall 9 704 s). B: 8 threads through `run_baselin configuration's grid-vs-sweep difference (6.2e-4) is below the median over all 30 pairs, so it is a favourable case for agreement and says little about the 5G-NIDD cases. +### 3.7 Repeat and multi-thread test, 5G-NIDD alpha 0.5 beta 0.75 (the largest disagreement) + +Same protocol as section 3.6, on the configuration where the grid and the sweep disagreed most +about which beta to choose. Partition hash unchanged. A: 1 thread, CPU 99 %, wall 16 752 s. +B: 8 threads, CPU 762 %, wall 9 722 s. Outputs in `results/sweep_repro/5gnidd_a0.5_b0.75/`. + +| run | test rare-F1 | test macro-F1 | test accuracy | validation selection metric | +|---|---|---|---|---| +| sweep (1 thread) | 0.879315 | 0.853592 | 0.745404 | 0.771921 | +| A, 1-thread repeat | 0.879315 | 0.853592 | 0.745404 | 0.771921 | +| B, 8 threads | 0.875942 | 0.853482 | 0.746436 | 0.744348 | +| grid (earlier) | 0.866627 | 0.848043 | 0.746765 | 0.750661 | + +| pair | test rare-F1 abs diff | validation metric abs diff | result JSON equal | snapshot identical | +|---|---|---|---|---| +| A vs sweep | 0 | 0 | 4/4 | yes | +| A vs grid | 1.27e-2 | 2.13e-2 | 0/4 | no | +| B vs sweep | 3.37e-3 | 2.76e-2 | 0/4 | no | +| B vs grid | 9.32e-3 | 6.31e-3 | 0/4 | no | + +- **The 1-thread setup is exactly reproducible on this dataset too.** Every compared result file + and the prototype snapshot of A equal the sweep run. Together with section 3.6 this holds for + the two configurations tested (one X-IIoTID, one 5G-NIDD). +- **Eight threads do not reproduce the grid here either.** B sits between the sweep and the grid, + and which one it is closer to depends on the metric: on test rare-F1 it is closer to the sweep + (3.4e-3 vs 9.3e-3), on the validation metric it is closer to the grid (6.3e-3 vs 2.8e-2). No + single conclusion follows from this metric split. +- **Thread count changes this configuration more than X-IIoTID**: B differs from the sweep by + 3.4e-3 on test rare-F1 here, against 1.9e-3 there. +- **Consequence for the beta disagreement.** Since the 1-thread result is deterministic and the + grid result is not reproduced by 8 threads, the grid-vs-sweep difference in this condition is a + reproducible consequence of a setup difference whose exact ingredient is not identified. It is + not run-to-run randomness of the sweep. + ## 4. Interpretation limits - One seed. The grid pools three. Nothing here is a significance test. @@ -178,10 +212,12 @@ to the sweep command (CPU 99 %, wall 9 704 s). B: 8 threads through `run_baselin best of the tested candidates under a slightly different realization of training; this does not show that either beta is better. - Thread count as the cause of the grid-vs-sweep difference is partly supported but not - established (section 3.6): threading changes the result, yet 8 threads on this host does not - reproduce the grid. Hardware, PyTorch version and the grid's exact thread count stay - uncontrolled. One configuration (X-IIoTID alpha 0.25 beta 0) was tested; an 8-thread repeat and - 5G-NIDD were not. + established (sections 3.6 and 3.7): threading changes the result, yet 8 threads on this host + does not reproduce the grid, on either of the two configurations tested (X-IIoTID alpha 0.25 + beta 0; 5G-NIDD alpha 0.5 beta 0.75). Hardware, PyTorch version and the grid's exact thread + count stay uncontrolled. +- The 1-thread determinism result rests on one repeat per configuration (two configurations). The + 8-thread setup has not been repeated, so its own reproducibility is unknown. - No claim is made that support weighting improves detection, and the selected beta is "best among the tested candidates under the defined validation procedure", not a global optimum. - `S_j` handling remains the documented reproduction assumption.