docs: add 5G-NIDD repeat and multi-thread test (section 3.7)
This commit is contained in:
1 parent
b40c928051
commit
e66ecc4b59
2 files changed
+46
-7
No files matched your search
@@ -171,6 +171,40 @@ to the sweep command (CPU 99 %, wall 9 704 s). B: 8 threads through `run_baselin
|
||||
configuration's grid-vs-sweep difference (6.2e-4) is below the median over all 30 pairs, so it
|
||||
is a favourable case for agreement and says little about the 5G-NIDD cases.
|
||||
|
||||
### 3.7 Repeat and multi-thread test, 5G-NIDD alpha 0.5 beta 0.75 (the largest disagreement)
|
||||
|
||||
Same protocol as section 3.6, on the configuration where the grid and the sweep disagreed most
|
||||
about which beta to choose. Partition hash unchanged. A: 1 thread, CPU 99 %, wall 16 752 s.
|
||||
B: 8 threads, CPU 762 %, wall 9 722 s. Outputs in `results/sweep_repro/5gnidd_a0.5_b0.75/`.
|
||||
|
||||
| run | test rare-F1 | test macro-F1 | test accuracy | validation selection metric |
|
||||
|---|---|---|---|---|
|
||||
| sweep (1 thread) | 0.879315 | 0.853592 | 0.745404 | 0.771921 |
|
||||
| A, 1-thread repeat | 0.879315 | 0.853592 | 0.745404 | 0.771921 |
|
||||
| B, 8 threads | 0.875942 | 0.853482 | 0.746436 | 0.744348 |
|
||||
| grid (earlier) | 0.866627 | 0.848043 | 0.746765 | 0.750661 |
|
||||
|
||||
| pair | test rare-F1 abs diff | validation metric abs diff | result JSON equal | snapshot identical |
|
||||
|---|---|---|---|---|
|
||||
| A vs sweep | 0 | 0 | 4/4 | yes |
|
||||
| A vs grid | 1.27e-2 | 2.13e-2 | 0/4 | no |
|
||||
| B vs sweep | 3.37e-3 | 2.76e-2 | 0/4 | no |
|
||||
| B vs grid | 9.32e-3 | 6.31e-3 | 0/4 | no |
|
||||
|
||||
- **The 1-thread setup is exactly reproducible on this dataset too.** Every compared result file
|
||||
and the prototype snapshot of A equal the sweep run. Together with section 3.6 this holds for
|
||||
the two configurations tested (one X-IIoTID, one 5G-NIDD).
|
||||
- **Eight threads do not reproduce the grid here either.** B sits between the sweep and the grid,
|
||||
and which one it is closer to depends on the metric: on test rare-F1 it is closer to the sweep
|
||||
(3.4e-3 vs 9.3e-3), on the validation metric it is closer to the grid (6.3e-3 vs 2.8e-2). No
|
||||
single conclusion follows from this metric split.
|
||||
- **Thread count changes this configuration more than X-IIoTID**: B differs from the sweep by
|
||||
3.4e-3 on test rare-F1 here, against 1.9e-3 there.
|
||||
- **Consequence for the beta disagreement.** Since the 1-thread result is deterministic and the
|
||||
grid result is not reproduced by 8 threads, the grid-vs-sweep difference in this condition is a
|
||||
reproducible consequence of a setup difference whose exact ingredient is not identified. It is
|
||||
not run-to-run randomness of the sweep.
|
||||
|
||||
## 4. Interpretation limits
|
||||
|
||||
- One seed. The grid pools three. Nothing here is a significance test.
|
||||
@@ -178,10 +212,12 @@ to the sweep command (CPU 99 %, wall 9 704 s). B: 8 threads through `run_baselin
|
||||
best of the tested candidates under a slightly different realization of training; this does
|
||||
not show that either beta is better.
|
||||
- Thread count as the cause of the grid-vs-sweep difference is partly supported but not
|
||||
established (section 3.6): threading changes the result, yet 8 threads on this host does not
|
||||
reproduce the grid. Hardware, PyTorch version and the grid's exact thread count stay
|
||||
uncontrolled. One configuration (X-IIoTID alpha 0.25 beta 0) was tested; an 8-thread repeat and
|
||||
5G-NIDD were not.
|
||||
established (sections 3.6 and 3.7): threading changes the result, yet 8 threads on this host
|
||||
does not reproduce the grid, on either of the two configurations tested (X-IIoTID alpha 0.25
|
||||
beta 0; 5G-NIDD alpha 0.5 beta 0.75). Hardware, PyTorch version and the grid's exact thread
|
||||
count stay uncontrolled.
|
||||
- The 1-thread determinism result rests on one repeat per configuration (two configurations). The
|
||||
8-thread setup has not been repeated, so its own reproducibility is unknown.
|
||||
- No claim is made that support weighting improves detection, and the selected beta is "best
|
||||
among the tested candidates under the defined validation procedure", not a global optimum.
|
||||
- `S_j` handling remains the documented reproduction assumption.
|
||||
|
||||
Reference in new issue
Block a user