docs: add 5G-NIDD repeat and multi-thread test (section 3.7)

This commit is contained in:
willtanoe committed 2026-10-29 17:56:16 +01:00
1 parent b40c928051
commit e66ecc4b59
2 files changed
+46 -7

No files matched your search

+40 -4
View File
@@ -171,6 +171,40 @@ to the sweep command (CPU 99 %, wall 9 704 s). B: 8 threads through `run_baselin
configuration's grid-vs-sweep difference (6.2e-4) is below the median over all 30 pairs, so it
is a favourable case for agreement and says little about the 5G-NIDD cases.
### 3.7 Repeat and multi-thread test, 5G-NIDD alpha 0.5 beta 0.75 (the largest disagreement)
Same protocol as section 3.6, on the configuration where the grid and the sweep disagreed most
about which beta to choose. Partition hash unchanged. A: 1 thread, CPU 99 %, wall 16 752 s.
B: 8 threads, CPU 762 %, wall 9 722 s. Outputs in `results/sweep_repro/5gnidd_a0.5_b0.75/`.
| run | test rare-F1 | test macro-F1 | test accuracy | validation selection metric |
|---|---|---|---|---|
| sweep (1 thread) | 0.879315 | 0.853592 | 0.745404 | 0.771921 |
| A, 1-thread repeat | 0.879315 | 0.853592 | 0.745404 | 0.771921 |
| B, 8 threads | 0.875942 | 0.853482 | 0.746436 | 0.744348 |
| grid (earlier) | 0.866627 | 0.848043 | 0.746765 | 0.750661 |
| pair | test rare-F1 abs diff | validation metric abs diff | result JSON equal | snapshot identical |
|---|---|---|---|---|
| A vs sweep | 0 | 0 | 4/4 | yes |
| A vs grid | 1.27e-2 | 2.13e-2 | 0/4 | no |
| B vs sweep | 3.37e-3 | 2.76e-2 | 0/4 | no |
| B vs grid | 9.32e-3 | 6.31e-3 | 0/4 | no |
- **The 1-thread setup is exactly reproducible on this dataset too.** Every compared result file
and the prototype snapshot of A equal the sweep run. Together with section 3.6 this holds for
the two configurations tested (one X-IIoTID, one 5G-NIDD).
- **Eight threads do not reproduce the grid here either.** B sits between the sweep and the grid,
and which one it is closer to depends on the metric: on test rare-F1 it is closer to the sweep
(3.4e-3 vs 9.3e-3), on the validation metric it is closer to the grid (6.3e-3 vs 2.8e-2). No
single conclusion follows from this metric split.
- **Thread count changes this configuration more than X-IIoTID**: B differs from the sweep by
3.4e-3 on test rare-F1 here, against 1.9e-3 there.
- **Consequence for the beta disagreement.** Since the 1-thread result is deterministic and the
grid result is not reproduced by 8 threads, the grid-vs-sweep difference in this condition is a
reproducible consequence of a setup difference whose exact ingredient is not identified. It is
not run-to-run randomness of the sweep.
## 4. Interpretation limits
- One seed. The grid pools three. Nothing here is a significance test.
@@ -178,10 +212,12 @@ to the sweep command (CPU 99 %, wall 9 704 s). B: 8 threads through `run_baselin
best of the tested candidates under a slightly different realization of training; this does
not show that either beta is better.
- Thread count as the cause of the grid-vs-sweep difference is partly supported but not
established (section 3.6): threading changes the result, yet 8 threads on this host does not
reproduce the grid. Hardware, PyTorch version and the grid's exact thread count stay
uncontrolled. One configuration (X-IIoTID alpha 0.25 beta 0) was tested; an 8-thread repeat and
5G-NIDD were not.
established (sections 3.6 and 3.7): threading changes the result, yet 8 threads on this host
does not reproduce the grid, on either of the two configurations tested (X-IIoTID alpha 0.25
beta 0; 5G-NIDD alpha 0.5 beta 0.75). Hardware, PyTorch version and the grid's exact thread
count stay uncontrolled.
- The 1-thread determinism result rests on one repeat per configuration (two configurations). The
8-thread setup has not been repeated, so its own reproducibility is unknown.
- No claim is made that support weighting improves detection, and the selected beta is "best
among the tested candidates under the defined validation procedure", not a global optimum.
- `S_j` handling remains the documented reproduction assumption.