docs: record 1-thread seed-0 sweep, noise checks and grid comparison
This commit is contained in:
willtanoe committed
1 parent
91b0e371f7
commit
ad203eb2ee
2 files changed
+245
No files matched your search
@@ -268,3 +268,82 @@ and artifacts/analysis/final__*. Obey CLAIMS_REGISTER.md wording, keep all limit
|
||||
and do not run training or add inferential tests by default. Use .venv/bin/python. Do not commit
|
||||
or push unless asked.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 15. Addendum (2026-10-29, WIB / UTC+7): 1-thread seed-0 sweep in `results/sweep/`
|
||||
|
||||
This addendum is **separate from the 108-run grid above**, which remains the primary evidence. It
|
||||
records an additional deterministic-threading re-run of seed 0 only. Everything lives in the
|
||||
git-ignored `results/sweep/`; no tracked source, model or aggregation file was changed.
|
||||
|
||||
- **What ran:** 30 runs = 2 datasets x 3 alpha (0.25/0.5/0.75) x 5 beta, seed 0, 1 thread per
|
||||
process, pool P=3. Relaunched 2026-10-27T01:13Z (08:13 WIB); last run finished
|
||||
2026-10-28T18:00Z (2026-10-29 01:00 WIB). 30/30 `completed exit 0`, 30 `COMPLETE` markers,
|
||||
0 failed, retry pass was a no-op, `PHASE=done`.
|
||||
- **Gate deadlock (fixed):** `reference_check.py` read a wrong JSON key
|
||||
(`val_rare_macro_f1_per_client_mean` instead of `val_rare_class_macro_f1_per_client_mean`), so the
|
||||
gate never resolved. After the fix the reference check PASSED (max difference to the manual verdict
|
||||
4.3e-10). Details, diffs and raw numbers: `results/sweep/relaunch_report.md` (untracked).
|
||||
- **One journal line to keep open:** `Oct 28 12:11:37 systemd[2118]: -.slice: A process of this unit
|
||||
has been killed by the OOM killer` (host-local time, UTC+1 inferred from file mtimes, i.e.
|
||||
18:11 WIB). The killed process is unidentified; no run scope failed and all 30 runs completed, so
|
||||
it is probably not a training process (inference, not verified).
|
||||
- **Analysis script:** `results/sweep/analyze_sweep.py` (read-only, deterministic; re-running it
|
||||
reproduced all three CSVs byte-identically). Stage 1 selects beta from validation only
|
||||
(`val_rare_class_macro_f1_per_client_mean`, tie -> smaller beta) and is written before any test
|
||||
file is opened. Outputs in `results/sweep/analysis/`: `selection.csv`,
|
||||
`test_at_selected_beta.csv`, `repro_vs_previous.csv`.
|
||||
- **Status of this selection:** it is a **single-seed, per-seed diagnostic**. The primary thesis
|
||||
selection remains pooled validation across clients and seeds (section 6, item 9).
|
||||
|
||||
Validation-selected beta (seed 0) and test rare-class macro-F1 (global) afterwards:
|
||||
|
||||
| dataset | alpha | selected beta | val margin to runner-up | test @ selected | test @ beta 0 |
|
||||
|---|---|---|---|---|---|
|
||||
| X-IIoTID | 0.25 | 0.75 | 0.0413 | 0.6692 | 0.6420 |
|
||||
| X-IIoTID | 0.50 | 0.50 | 0.0074 | 0.6439 | 0.6131 |
|
||||
| X-IIoTID | 0.75 | 0.50 | 0.0004 | 0.5068 | 0.5263 |
|
||||
| 5G-NIDD | 0.25 | 0.25 | 0.0155 | 0.8876 | 0.8612 |
|
||||
| 5G-NIDD | 0.50 | 0.75 | 0.0053 | 0.8793 | 0.8694 |
|
||||
| 5G-NIDD | 0.75 | 0.25 | 0.0132 | 0.9065 | 0.9024 |
|
||||
|
||||
- **Reproducibility caveat (important):** the 1-thread re-run is **not bit-identical** to the earlier
|
||||
grid for the same (dataset, alpha, seed 0, beta). Over 30 pairs the absolute test rare-F1
|
||||
difference has median 0.0022 and max 0.0127 (5G-NIDD alpha 0.5 beta 0.75); the validation
|
||||
selection metric differs by median 0.0029, max 0.0328; 0 of 30 pairs are exactly equal. Compared
|
||||
with the median validation difference (0.0029), only X-IIoTID alpha 0.75 (margin 0.0004) is
|
||||
clearly inside the noise. X-IIoTID alpha 0.5 (0.0074) and 5G-NIDD alpha 0.5 (0.0053) are above
|
||||
the median but well below the max, so they are also not clearly separated. The other three
|
||||
margins (0.0132 to 0.0413) are larger than the median difference; the max difference (0.0328)
|
||||
still exceeds all but the X-IIoTID alpha 0.25 margin. None of this is a significance test.
|
||||
- **Comparison with the 108-run grid** (`results/sweep/compare_with_grid.py` ->
|
||||
`results/sweep/analysis/compare_with_grid.csv`, validation only, no test file opened):
|
||||
the sweep's seed-0 beta equals the grid's seed-0 beta in 4/6 conditions (all three X-IIoTID,
|
||||
plus 5G-NIDD alpha 0.25). The two differences are both 5G-NIDD: alpha 0.5 (sweep 0.75, grid
|
||||
seed-0 1.0; sweep pick is rank 3 in the grid ranking) and alpha 0.75 (sweep 0.25, grid 0.5;
|
||||
rank 4). These two conditions also have the largest per-beta validation differences
|
||||
(0.0328 and 0.0262). The sweep beta equals the grid **pooled** beta in only 1/6, but the grid's
|
||||
own seed-0 beta equals its pooled beta in only 3/6, so a seed-0 vs pooled mismatch is expected
|
||||
and is not a reproducibility measure.
|
||||
- **Not done:** no inferential test; thread count as the cause of the difference is a hypothesis, not
|
||||
verified; the metric files were not cross-checked beyond what `analyze_sweep.py` reads.
|
||||
- **Noise checks** (`results/sweep/check_noise.py` -> `analysis/noise_checks.json`, no training):
|
||||
all 30 pairs already differ at round 0 with identical sample counts; test rare-F1 differences
|
||||
scatter around zero (14 higher / 16 lower, mean signed -0.0004, mean absolute 0.0034); 5G-NIDD is
|
||||
about 2.7x noisier than X-IIoTID on the validation metric; beta-ranking Spearman is 0.30 / 0.50
|
||||
for 5G-NIDD alpha 0.75 / 0.5 and >= 0.90 elsewhere. Thread count remains **unproven and
|
||||
confounded** with host (8 threads / 29 GiB documented earlier vs 12 / 48 GiB now), possibly a
|
||||
different PyTorch version (not recorded in metadata), and a different checkout path. A sign test
|
||||
(p = 0.86 / 1.0) is reported only descriptively because the pairs are not independent.
|
||||
- **Write-up:** `docs/research/SWEEP_1THREAD_SEED0_CHECK.md` (untracked, not yet committed)
|
||||
summarizes the sweep, the run-to-run difference, the noise checks and the comparison above.
|
||||
- **Exact next task:** decide with the user whether the 5G-NIDD alpha 0.5 / 0.75 disagreement
|
||||
warrants investigating the thread-count hypothesis (needs extra runs; not started) and whether
|
||||
to commit the handoff and the write-up. Do not change the primary pooled selection on the basis of this single-seed re-run.
|
||||
- **Housekeeping:** the tmux session `sweep` still exists (windows `control`, `pool`); the
|
||||
`heartbeat` window was stopped on 2026-10-29. Backups of edited sweep scripts are `*.bak` in
|
||||
`results/sweep/`; orphaned attempt-1 artifacts are in `results/sweep/_orphans_20261027T011213Z/`.
|
||||
rsync destination was never specified and rsync was never run.
|
||||
- **Git:** this addendum is an uncommitted edit on branch `chore/posthoc-audit-and-docs-refresh`;
|
||||
nothing was committed or pushed.
|
||||
@@ -0,0 +1,166 @@
|
||||
# 1-thread seed-0 re-run: validation selection vs. the 108-run grid
|
||||
|
||||
Status: **exploratory consistency check**, single seed. It does not replace, correct or
|
||||
re-select anything in `FINAL_EXPERIMENTAL_SYNTHESIS.md`. The primary selection of the thesis
|
||||
remains the pooled validation across clients and seeds of the 108-run grid.
|
||||
|
||||
## 1. What was run
|
||||
|
||||
- 30 runs: 2 datasets (X-IIoTID, 5G-NIDD) x 3 alpha (0.25, 0.5, 0.75) x 5 beta
|
||||
(0, 0.25, 0.5, 0.75, 1), `seed = 0`, `support_weighted` mode.
|
||||
- 1 thread per training process, 3 processes in parallel. 2026-10-27T01:13Z to
|
||||
2026-10-28T18:00Z (2026-10-27 08:13 to 2026-10-29 01:00 WIB), about 40.8 h.
|
||||
- All 30 runs ended `completed exit 0` with a `COMPLETE` marker; 0 failed, retry pass was empty.
|
||||
- Outputs live in the git-ignored `results/sweep/` (runs, `summary.csv`, `analysis/`). They are
|
||||
not committed. The orchestration record (including a gate-deadlock fix) is
|
||||
`results/sweep/relaunch_report.md`, also untracked.
|
||||
|
||||
Before the main queue continued, `reference_check.py` compared the first two runs (X-IIoTID
|
||||
alpha 0.25 seed 0, beta 0 and 1) with the audit values; the maximum difference to the manual
|
||||
verdict was 4.3e-10 and every |z| was at most 0.12.
|
||||
|
||||
## 2. Method
|
||||
|
||||
`results/sweep/analyze_sweep.py` (read-only, deterministic; two executions gave byte-identical
|
||||
CSVs):
|
||||
|
||||
1. **Stage 1, validation only.** For each (dataset, alpha) choose the beta with the highest
|
||||
`val_rare_class_macro_f1_per_client_mean`; ties go to the smaller beta. `selection.csv` is
|
||||
written before any test file is opened.
|
||||
2. **Stage 2.** Test rare-class macro-F1 at every beta, read only after stage 1.
|
||||
(`test_at_selected_beta.csv`.) Descriptive; not used to choose anything.
|
||||
3. **Stage 3.** Absolute differences to the earlier grid run of the same
|
||||
(dataset, alpha, seed 0, beta). (`repro_vs_previous.csv`.)
|
||||
|
||||
`results/sweep/compare_with_grid.py` then compares the sweep's choice with the grid's seed-0
|
||||
choice and the grid's pooled choice, using validation metrics only.
|
||||
|
||||
## 3. Results
|
||||
|
||||
### 3.1 Run-to-run difference (30 pairs, same configuration, earlier run vs. 1-thread run)
|
||||
|
||||
| metric | median abs. difference | max abs. difference |
|
||||
|---|---|---|
|
||||
| test rare-class macro-F1 | 0.0022 | 0.0127 (5G-NIDD, alpha 0.5, beta 0.75) |
|
||||
| validation selection metric | 0.0029 | 0.0328 |
|
||||
|
||||
None of the 30 pairs is exactly equal. The cause is not established. Thread count is a
|
||||
hypothesis only; no experiment isolates it.
|
||||
|
||||
### 3.2 Seed-0 selection
|
||||
|
||||
| dataset | alpha | beta (sweep) | beta (grid, seed 0) | beta (grid, pooled) | sweep margin | grid s0 margin |
|
||||
|---|---|---|---|---|---|---|
|
||||
| X-IIoTID | 0.25 | 0.75 | 0.75 | 0 | 0.0413 | 0.0377 |
|
||||
| X-IIoTID | 0.50 | 0.50 | 0.50 | 0.75 | 0.0074 | 0.0078 |
|
||||
| X-IIoTID | 0.75 | 0.50 | 0.50 | 0.50 | 0.0004 | 0.0075 |
|
||||
| 5G-NIDD | 0.25 | 0.25 | 0.25 | 0 | 0.0155 | 0.0109 |
|
||||
| 5G-NIDD | 0.50 | 0.75 | 1.00 | 1.00 | 0.0053 | 0.0044 |
|
||||
| 5G-NIDD | 0.75 | 0.25 | 0.50 | 0.50 | 0.0132 | 0.0021 |
|
||||
|
||||
"Margin" is the validation score of the best beta minus that of the runner-up.
|
||||
|
||||
- The sweep and grid seed-0 choices coincide in **4 of 6** conditions.
|
||||
- The two disagreements are both 5G-NIDD. At alpha 0.5 the sweep's beta is rank 3 in the grid's
|
||||
seed-0 ranking; at alpha 0.75 it is rank 4. These two conditions also show the largest
|
||||
per-beta validation differences between the two runs (0.0328 and 0.0262).
|
||||
- The sweep beta equals the grid **pooled** beta in 1 of 6, but the grid's own seed-0 beta
|
||||
equals its pooled beta in only 3 of 6. A seed-0 versus pooled mismatch is therefore expected
|
||||
and is not a measure of reproducibility.
|
||||
- X-IIoTID alpha 0.75 has a sweep margin of 0.0004, far below the median validation run-to-run
|
||||
difference (0.0029), yet its choice coincides with the grid's. The narrow margin did not
|
||||
produce a different choice here.
|
||||
|
||||
### 3.3 Test rare-class macro-F1 at the validation-selected beta (descriptive, seed 0)
|
||||
|
||||
| dataset | alpha | selected beta | test @ selected | test @ beta 0 |
|
||||
|---|---|---|---|---|
|
||||
| X-IIoTID | 0.25 | 0.75 | 0.6692 | 0.6420 |
|
||||
| X-IIoTID | 0.50 | 0.50 | 0.6439 | 0.6131 |
|
||||
| X-IIoTID | 0.75 | 0.50 | 0.5068 | 0.5263 |
|
||||
| 5G-NIDD | 0.25 | 0.25 | 0.8876 | 0.8612 |
|
||||
| 5G-NIDD | 0.50 | 0.75 | 0.8793 | 0.8694 |
|
||||
| 5G-NIDD | 0.75 | 0.25 | 0.9065 | 0.9024 |
|
||||
|
||||
Selected minus beta 0: +0.0272, +0.0308, -0.0195 (X-IIoTID alpha 0.25 / 0.5 / 0.75) and
|
||||
+0.0264, +0.0099, +0.0042 (5G-NIDD alpha 0.25 / 0.5 / 0.75). In 5 of 6 conditions the test value
|
||||
at the validation-selected beta is above that at beta 0; in X-IIoTID alpha 0.75 it is below.
|
||||
|
||||
Four gaps (absolute value 0.0195 to 0.0308) exceed the largest test run-to-run difference seen
|
||||
in section 3.1 (0.0127). The two 5G-NIDD gaps at alpha 0.5 and 0.75 (0.0099 and 0.0042) lie
|
||||
within the range of that difference (median 0.0022, max 0.0127) and should not be read as
|
||||
improvements. The run-to-run difference compares two runs of one configuration; it is a rough
|
||||
yardstick, not a noise bound for a difference between two betas.
|
||||
|
||||
### 3.4 Noise checks (`results/sweep/check_noise.py` -> `analysis/noise_checks.json`)
|
||||
|
||||
Read-only, no training. The 30 pairs are not independent (the five betas of a condition share
|
||||
one partition), so the sign counts below are descriptive only.
|
||||
|
||||
- **Where the runs first differ.** All 30 pairs already differ at round 0 (max absolute
|
||||
difference in rare-class macro-F1 1.8e-3; 0 of 30 exactly equal). The number of evaluated
|
||||
samples is identical in every round of every pair, so data and partitions are the same. The
|
||||
difference is small at round 0 (median 3.1e-4) and **grows over the rounds** (median 2.1e-3 and
|
||||
max 1.5e-2 at the last round; larger at the last round than at round 0 in 24 of 30 pairs).
|
||||
A tiny first-round difference that is amplified by later rounds fits the pattern; it does not
|
||||
identify what produced the first-round difference.
|
||||
- **Direction.** Test rare-F1, sweep minus grid: 14 higher, 16 lower, mean signed -0.0004 against
|
||||
mean absolute 0.0034. Validation selection metric: 15 higher, 15 lower, mean signed -0.0004
|
||||
against mean absolute 0.0057. The differences scatter around zero; there is no sign of a
|
||||
systematic offset. (A two-sided sign test gives p = 0.86 and 1.0, but the pairs are dependent,
|
||||
so this is not a significance claim.)
|
||||
- **By dataset.** Mean absolute validation difference: X-IIoTID 0.0031, 5G-NIDD 0.0082 (about
|
||||
2.7 times larger). Test: 0.0026 vs 0.0042 (about 1.6 times). 5G-NIDD is the noisier dataset
|
||||
in this comparison, consistent with both disagreements sitting there.
|
||||
- **Ranking stability** (Spearman over the five betas, sweep vs grid, validation): X-IIoTID
|
||||
1.00 / 0.90 / 1.00 and 5G-NIDD 0.90 / 0.50 / 0.30 for alpha 0.25 / 0.5 / 0.75. Betas whose
|
||||
sweep validation score lies within the condition's own maximum run-to-run difference of the
|
||||
best beta (a crude yardstick): X-IIoTID alpha 0.25 {0.75}; alpha 0.5 {0.5, 0.75}; alpha 0.75
|
||||
{0, 0.25, 0.5}; 5G-NIDD alpha 0.25 {0.25}; alpha 0.5 {0, 0.75, 1}; alpha 0.75 {0, 0.25, 0.5}.
|
||||
Selections with one beta in that set (X-IIoTID alpha 0.25, 5G-NIDD alpha 0.25) are the least
|
||||
ambiguous; the others cannot separate their top candidates from the noise.
|
||||
- **Third reference point** (X-IIoTID alpha 0.25 seed 0, test rare-F1; from the earlier
|
||||
5-thread smoke run recorded in `reference_check.json`): beta 0: smoke 0.6402, grid 0.6414,
|
||||
sweep 0.6420; beta 1: smoke 0.5721, grid 0.5841, sweep 0.5931. For beta 1 the three setups
|
||||
span 0.021, larger than the sweep-vs-grid difference alone (0.0089).
|
||||
|
||||
### 3.5 What the thread hypothesis rests on
|
||||
|
||||
- The earlier grid used several cores per process (CPU 741 to 746 % in the X-IIoTID alpha 0.25
|
||||
runtime table, about 7.4 cores); the sweep forces 1 thread (`thread1_launcher.py`,
|
||||
`NLWP = 1` observed). Multi-threaded reductions can change floating-point summation order, so
|
||||
a thread effect is plausible. `BASELINE_VALIDATION.md` (F.2) reports that two invocations of
|
||||
`run_federated` with the same seed gave exactly equal results in a regression test on a small
|
||||
fixture, and that no nondeterminism source was observed at the tested scales on this CPU
|
||||
backend. That says a same-setup repeat was deterministic there; it does not show that full-size
|
||||
runs are bit-reproducible, and it does not test the thread hypothesis.
|
||||
- Confounders that were **not** controlled: the planning document recorded 8 CPU threads and
|
||||
about 29 GiB RAM for the earlier work, whereas this host has `nproc = 12` and 48 GiB; the old
|
||||
run metadata records a different repository path (`.../projects/perdalam-protean`); the
|
||||
metadata stores no PyTorch version, so a library difference cannot be excluded (current:
|
||||
2.14.1+cpu). Whether the grid ran on a different machine is inferred, not verified.
|
||||
|
||||
## 4. Interpretation limits
|
||||
|
||||
- One seed. The grid pools three. Nothing here is a significance test.
|
||||
- Where the sweep and the grid disagree (5G-NIDD alpha 0.5 and 0.75), a different beta is the
|
||||
best of the tested candidates under a slightly different realization of training; this does
|
||||
not show that either beta is better.
|
||||
- Unverified: that thread count causes the run-to-run difference (it is confounded with hardware
|
||||
and possibly library version, section 3.5). Two cheap runs would separate them: (a) repeat one
|
||||
1-thread run and compare it bit-for-bit with the sweep run (is a repeat identical?); (b) run
|
||||
the same configuration multi-threaded on this host and compare with the grid. Neither has
|
||||
been run.
|
||||
- No claim is made that support weighting improves detection, and the selected beta is "best
|
||||
among the tested candidates under the defined validation procedure", not a global optimum.
|
||||
- `S_j` handling remains the documented reproduction assumption.
|
||||
|
||||
## 5. Reproduce
|
||||
|
||||
```bash
|
||||
.venv/bin/python results/sweep/analyze_sweep.py
|
||||
.venv/bin/python results/sweep/compare_with_grid.py
|
||||
```
|
||||
|
||||
Both only read; they write into `results/sweep/analysis/`. The scripts and CSVs are untracked
|
||||
because `results/` is git-ignored.
|
||||
Reference in new issue
Block a user