docs: add 5G-NIDD repeat and multi-thread test (section 3.7)

This commit is contained in:
willtanoe committed 2026-10-29 17:56:16 +01:00
1 parent b40c928051
commit e66ecc4b59
2 files changed
+46 -7

No files matched your search

+6 -3
View File
@@ -344,10 +344,13 @@ Validation-selected beta (seed 0) and test rare-class macro-F1 (global) afterwar
snapshot), so the 1-thread sweep has no run-to-run noise. An 8-thread run on this host differs
from the 1-thread run (test rare-F1 1.9e-3) and also does not reproduce the old grid (2.5e-3).
Thread count changes results but does not explain the whole grid-vs-sweep difference; host and
PyTorch version remain uncontrolled. 5G-NIDD and an 8-thread repeat were not tested.
PyTorch version remain uncontrolled. The same test was also run on 5G-NIDD alpha 0.5 beta 0.75
(the largest disagreement): the 1-thread repeat is again bit-identical to the sweep, and the
8-thread run again reproduces neither (closer to the sweep on test rare-F1, closer to the grid
on the validation metric). Still untested: an 8-thread repeat (its own stability).
- **Exact next task:** decide with the user whether the 5G-NIDD alpha 0.5 / 0.75 disagreement
warrants a 5G-NIDD repeat / multi-thread test (needs extra runs; not started) and whether to
commit section 3.6. Do not change the primary pooled selection on the basis of this single-seed re-run.
warrants an 8-thread repeat (is the multi-thread setup itself stable?) and whether to commit
section 3.7. Do not change the primary pooled selection on the basis of this single-seed re-run.
- **Housekeeping:** the tmux session `sweep` still exists (windows `control`, `pool`); the
`heartbeat` window was stopped on 2026-10-29. Backups of edited sweep scripts are `*.bak` in
`results/sweep/`; orphaned attempt-1 artifacts are in `results/sweep/_orphans_20261027T011213Z/`.
+40 -4
View File
@@ -171,6 +171,40 @@ to the sweep command (CPU 99 %, wall 9 704 s). B: 8 threads through `run_baselin
configuration's grid-vs-sweep difference (6.2e-4) is below the median over all 30 pairs, so it
is a favourable case for agreement and says little about the 5G-NIDD cases.
### 3.7 Repeat and multi-thread test, 5G-NIDD alpha 0.5 beta 0.75 (the largest disagreement)
Same protocol as section 3.6, on the configuration where the grid and the sweep disagreed most
about which beta to choose. Partition hash unchanged. A: 1 thread, CPU 99 %, wall 16 752 s.
B: 8 threads, CPU 762 %, wall 9 722 s. Outputs in `results/sweep_repro/5gnidd_a0.5_b0.75/`.
| run | test rare-F1 | test macro-F1 | test accuracy | validation selection metric |
|---|---|---|---|---|
| sweep (1 thread) | 0.879315 | 0.853592 | 0.745404 | 0.771921 |
| A, 1-thread repeat | 0.879315 | 0.853592 | 0.745404 | 0.771921 |
| B, 8 threads | 0.875942 | 0.853482 | 0.746436 | 0.744348 |
| grid (earlier) | 0.866627 | 0.848043 | 0.746765 | 0.750661 |
| pair | test rare-F1 abs diff | validation metric abs diff | result JSON equal | snapshot identical |
|---|---|---|---|---|
| A vs sweep | 0 | 0 | 4/4 | yes |
| A vs grid | 1.27e-2 | 2.13e-2 | 0/4 | no |
| B vs sweep | 3.37e-3 | 2.76e-2 | 0/4 | no |
| B vs grid | 9.32e-3 | 6.31e-3 | 0/4 | no |
- **The 1-thread setup is exactly reproducible on this dataset too.** Every compared result file
and the prototype snapshot of A equal the sweep run. Together with section 3.6 this holds for
the two configurations tested (one X-IIoTID, one 5G-NIDD).
- **Eight threads do not reproduce the grid here either.** B sits between the sweep and the grid,
and which one it is closer to depends on the metric: on test rare-F1 it is closer to the sweep
(3.4e-3 vs 9.3e-3), on the validation metric it is closer to the grid (6.3e-3 vs 2.8e-2). No
single conclusion follows from this metric split.
- **Thread count changes this configuration more than X-IIoTID**: B differs from the sweep by
3.4e-3 on test rare-F1 here, against 1.9e-3 there.
- **Consequence for the beta disagreement.** Since the 1-thread result is deterministic and the
grid result is not reproduced by 8 threads, the grid-vs-sweep difference in this condition is a
reproducible consequence of a setup difference whose exact ingredient is not identified. It is
not run-to-run randomness of the sweep.
## 4. Interpretation limits
- One seed. The grid pools three. Nothing here is a significance test.
@@ -178,10 +212,12 @@ to the sweep command (CPU 99 %, wall 9 704 s). B: 8 threads through `run_baselin
best of the tested candidates under a slightly different realization of training; this does
not show that either beta is better.
- Thread count as the cause of the grid-vs-sweep difference is partly supported but not
established (section 3.6): threading changes the result, yet 8 threads on this host does not
reproduce the grid. Hardware, PyTorch version and the grid's exact thread count stay
uncontrolled. One configuration (X-IIoTID alpha 0.25 beta 0) was tested; an 8-thread repeat and
5G-NIDD were not.
established (sections 3.6 and 3.7): threading changes the result, yet 8 threads on this host
does not reproduce the grid, on either of the two configurations tested (X-IIoTID alpha 0.25
beta 0; 5G-NIDD alpha 0.5 beta 0.75). Hardware, PyTorch version and the grid's exact thread
count stay uncontrolled.
- The 1-thread determinism result rests on one repeat per configuration (two configurations). The
8-thread setup has not been repeated, so its own reproducibility is unknown.
- No claim is made that support weighting improves detection, and the selected beta is "best
among the tested candidates under the defined validation procedure", not a global optimum.
- `S_j` handling remains the documented reproduction assumption.