docs: add 5G-NIDD repeat and multi-thread test (section 3.7)
This commit is contained in:
1 parent
b40c928051
commit
e66ecc4b59
2 files changed
+46
-7
No files matched your search
@@ -344,10 +344,13 @@ Validation-selected beta (seed 0) and test rare-class macro-F1 (global) afterwar
|
||||
snapshot), so the 1-thread sweep has no run-to-run noise. An 8-thread run on this host differs
|
||||
from the 1-thread run (test rare-F1 1.9e-3) and also does not reproduce the old grid (2.5e-3).
|
||||
Thread count changes results but does not explain the whole grid-vs-sweep difference; host and
|
||||
PyTorch version remain uncontrolled. 5G-NIDD and an 8-thread repeat were not tested.
|
||||
PyTorch version remain uncontrolled. The same test was also run on 5G-NIDD alpha 0.5 beta 0.75
|
||||
(the largest disagreement): the 1-thread repeat is again bit-identical to the sweep, and the
|
||||
8-thread run again reproduces neither (closer to the sweep on test rare-F1, closer to the grid
|
||||
on the validation metric). Still untested: an 8-thread repeat (its own stability).
|
||||
- **Exact next task:** decide with the user whether the 5G-NIDD alpha 0.5 / 0.75 disagreement
|
||||
warrants a 5G-NIDD repeat / multi-thread test (needs extra runs; not started) and whether to
|
||||
commit section 3.6. Do not change the primary pooled selection on the basis of this single-seed re-run.
|
||||
warrants an 8-thread repeat (is the multi-thread setup itself stable?) and whether to commit
|
||||
section 3.7. Do not change the primary pooled selection on the basis of this single-seed re-run.
|
||||
- **Housekeeping:** the tmux session `sweep` still exists (windows `control`, `pool`); the
|
||||
`heartbeat` window was stopped on 2026-10-29. Backups of edited sweep scripts are `*.bak` in
|
||||
`results/sweep/`; orphaned attempt-1 artifacts are in `results/sweep/_orphans_20261027T011213Z/`.
|
||||
|
||||
@@ -171,6 +171,40 @@ to the sweep command (CPU 99 %, wall 9 704 s). B: 8 threads through `run_baselin
|
||||
configuration's grid-vs-sweep difference (6.2e-4) is below the median over all 30 pairs, so it
|
||||
is a favourable case for agreement and says little about the 5G-NIDD cases.
|
||||
|
||||
### 3.7 Repeat and multi-thread test, 5G-NIDD alpha 0.5 beta 0.75 (the largest disagreement)
|
||||
|
||||
Same protocol as section 3.6, on the configuration where the grid and the sweep disagreed most
|
||||
about which beta to choose. Partition hash unchanged. A: 1 thread, CPU 99 %, wall 16 752 s.
|
||||
B: 8 threads, CPU 762 %, wall 9 722 s. Outputs in `results/sweep_repro/5gnidd_a0.5_b0.75/`.
|
||||
|
||||
| run | test rare-F1 | test macro-F1 | test accuracy | validation selection metric |
|
||||
|---|---|---|---|---|
|
||||
| sweep (1 thread) | 0.879315 | 0.853592 | 0.745404 | 0.771921 |
|
||||
| A, 1-thread repeat | 0.879315 | 0.853592 | 0.745404 | 0.771921 |
|
||||
| B, 8 threads | 0.875942 | 0.853482 | 0.746436 | 0.744348 |
|
||||
| grid (earlier) | 0.866627 | 0.848043 | 0.746765 | 0.750661 |
|
||||
|
||||
| pair | test rare-F1 abs diff | validation metric abs diff | result JSON equal | snapshot identical |
|
||||
|---|---|---|---|---|
|
||||
| A vs sweep | 0 | 0 | 4/4 | yes |
|
||||
| A vs grid | 1.27e-2 | 2.13e-2 | 0/4 | no |
|
||||
| B vs sweep | 3.37e-3 | 2.76e-2 | 0/4 | no |
|
||||
| B vs grid | 9.32e-3 | 6.31e-3 | 0/4 | no |
|
||||
|
||||
- **The 1-thread setup is exactly reproducible on this dataset too.** Every compared result file
|
||||
and the prototype snapshot of A equal the sweep run. Together with section 3.6 this holds for
|
||||
the two configurations tested (one X-IIoTID, one 5G-NIDD).
|
||||
- **Eight threads do not reproduce the grid here either.** B sits between the sweep and the grid,
|
||||
and which one it is closer to depends on the metric: on test rare-F1 it is closer to the sweep
|
||||
(3.4e-3 vs 9.3e-3), on the validation metric it is closer to the grid (6.3e-3 vs 2.8e-2). No
|
||||
single conclusion follows from this metric split.
|
||||
- **Thread count changes this configuration more than X-IIoTID**: B differs from the sweep by
|
||||
3.4e-3 on test rare-F1 here, against 1.9e-3 there.
|
||||
- **Consequence for the beta disagreement.** Since the 1-thread result is deterministic and the
|
||||
grid result is not reproduced by 8 threads, the grid-vs-sweep difference in this condition is a
|
||||
reproducible consequence of a setup difference whose exact ingredient is not identified. It is
|
||||
not run-to-run randomness of the sweep.
|
||||
|
||||
## 4. Interpretation limits
|
||||
|
||||
- One seed. The grid pools three. Nothing here is a significance test.
|
||||
@@ -178,10 +212,12 @@ to the sweep command (CPU 99 %, wall 9 704 s). B: 8 threads through `run_baselin
|
||||
best of the tested candidates under a slightly different realization of training; this does
|
||||
not show that either beta is better.
|
||||
- Thread count as the cause of the grid-vs-sweep difference is partly supported but not
|
||||
established (section 3.6): threading changes the result, yet 8 threads on this host does not
|
||||
reproduce the grid. Hardware, PyTorch version and the grid's exact thread count stay
|
||||
uncontrolled. One configuration (X-IIoTID alpha 0.25 beta 0) was tested; an 8-thread repeat and
|
||||
5G-NIDD were not.
|
||||
established (sections 3.6 and 3.7): threading changes the result, yet 8 threads on this host
|
||||
does not reproduce the grid, on either of the two configurations tested (X-IIoTID alpha 0.25
|
||||
beta 0; 5G-NIDD alpha 0.5 beta 0.75). Hardware, PyTorch version and the grid's exact thread
|
||||
count stay uncontrolled.
|
||||
- The 1-thread determinism result rests on one repeat per configuration (two configurations). The
|
||||
8-thread setup has not been repeated, so its own reproducibility is unknown.
|
||||
- No claim is made that support weighting improves detection, and the selected beta is "best
|
||||
among the tested candidates under the defined validation procedure", not a global optimum.
|
||||
- `S_j` handling remains the documented reproduction assumption.
|
||||
|
||||
Reference in new issue
Block a user