From ad203eb2ee6e41427b7bd698a3e9df0234141ef6 Mon Sep 17 00:00:00 2001 From: willtanoe Date: Thu, 29 Oct 2026 01:40:05 +0100 Subject: [PATCH] docs: record 1-thread seed-0 sweep, noise checks and grid comparison --- docs/handoff/SESSION_HANDOFF.md | 79 ++++++++++ docs/research/SWEEP_1THREAD_SEED0_CHECK.md | 166 +++++++++++++++++++++ 2 files changed, 245 insertions(+) create mode 100644 docs/research/SWEEP_1THREAD_SEED0_CHECK.md diff --git a/docs/handoff/SESSION_HANDOFF.md b/docs/handoff/SESSION_HANDOFF.md index bc1784b..0ab48f3 100644 --- a/docs/handoff/SESSION_HANDOFF.md +++ b/docs/handoff/SESSION_HANDOFF.md @@ -268,3 +268,82 @@ and artifacts/analysis/final__*. Obey CLAIMS_REGISTER.md wording, keep all limit and do not run training or add inferential tests by default. Use .venv/bin/python. Do not commit or push unless asked. ``` + +--- + +## 15. Addendum (2026-10-29, WIB / UTC+7): 1-thread seed-0 sweep in `results/sweep/` + +This addendum is **separate from the 108-run grid above**, which remains the primary evidence. It +records an additional deterministic-threading re-run of seed 0 only. Everything lives in the +git-ignored `results/sweep/`; no tracked source, model or aggregation file was changed. + +- **What ran:** 30 runs = 2 datasets x 3 alpha (0.25/0.5/0.75) x 5 beta, seed 0, 1 thread per + process, pool P=3. Relaunched 2026-10-27T01:13Z (08:13 WIB); last run finished + 2026-10-28T18:00Z (2026-10-29 01:00 WIB). 30/30 `completed exit 0`, 30 `COMPLETE` markers, + 0 failed, retry pass was a no-op, `PHASE=done`. +- **Gate deadlock (fixed):** `reference_check.py` read a wrong JSON key + (`val_rare_macro_f1_per_client_mean` instead of `val_rare_class_macro_f1_per_client_mean`), so the + gate never resolved. After the fix the reference check PASSED (max difference to the manual verdict + 4.3e-10). Details, diffs and raw numbers: `results/sweep/relaunch_report.md` (untracked). +- **One journal line to keep open:** `Oct 28 12:11:37 systemd[2118]: -.slice: A process of this unit + has been killed by the OOM killer` (host-local time, UTC+1 inferred from file mtimes, i.e. + 18:11 WIB). The killed process is unidentified; no run scope failed and all 30 runs completed, so + it is probably not a training process (inference, not verified). +- **Analysis script:** `results/sweep/analyze_sweep.py` (read-only, deterministic; re-running it + reproduced all three CSVs byte-identically). Stage 1 selects beta from validation only + (`val_rare_class_macro_f1_per_client_mean`, tie -> smaller beta) and is written before any test + file is opened. Outputs in `results/sweep/analysis/`: `selection.csv`, + `test_at_selected_beta.csv`, `repro_vs_previous.csv`. +- **Status of this selection:** it is a **single-seed, per-seed diagnostic**. The primary thesis + selection remains pooled validation across clients and seeds (section 6, item 9). + +Validation-selected beta (seed 0) and test rare-class macro-F1 (global) afterwards: + +| dataset | alpha | selected beta | val margin to runner-up | test @ selected | test @ beta 0 | +|---|---|---|---|---|---| +| X-IIoTID | 0.25 | 0.75 | 0.0413 | 0.6692 | 0.6420 | +| X-IIoTID | 0.50 | 0.50 | 0.0074 | 0.6439 | 0.6131 | +| X-IIoTID | 0.75 | 0.50 | 0.0004 | 0.5068 | 0.5263 | +| 5G-NIDD | 0.25 | 0.25 | 0.0155 | 0.8876 | 0.8612 | +| 5G-NIDD | 0.50 | 0.75 | 0.0053 | 0.8793 | 0.8694 | +| 5G-NIDD | 0.75 | 0.25 | 0.0132 | 0.9065 | 0.9024 | + +- **Reproducibility caveat (important):** the 1-thread re-run is **not bit-identical** to the earlier + grid for the same (dataset, alpha, seed 0, beta). Over 30 pairs the absolute test rare-F1 + difference has median 0.0022 and max 0.0127 (5G-NIDD alpha 0.5 beta 0.75); the validation + selection metric differs by median 0.0029, max 0.0328; 0 of 30 pairs are exactly equal. Compared + with the median validation difference (0.0029), only X-IIoTID alpha 0.75 (margin 0.0004) is + clearly inside the noise. X-IIoTID alpha 0.5 (0.0074) and 5G-NIDD alpha 0.5 (0.0053) are above + the median but well below the max, so they are also not clearly separated. The other three + margins (0.0132 to 0.0413) are larger than the median difference; the max difference (0.0328) + still exceeds all but the X-IIoTID alpha 0.25 margin. None of this is a significance test. +- **Comparison with the 108-run grid** (`results/sweep/compare_with_grid.py` -> + `results/sweep/analysis/compare_with_grid.csv`, validation only, no test file opened): + the sweep's seed-0 beta equals the grid's seed-0 beta in 4/6 conditions (all three X-IIoTID, + plus 5G-NIDD alpha 0.25). The two differences are both 5G-NIDD: alpha 0.5 (sweep 0.75, grid + seed-0 1.0; sweep pick is rank 3 in the grid ranking) and alpha 0.75 (sweep 0.25, grid 0.5; + rank 4). These two conditions also have the largest per-beta validation differences + (0.0328 and 0.0262). The sweep beta equals the grid **pooled** beta in only 1/6, but the grid's + own seed-0 beta equals its pooled beta in only 3/6, so a seed-0 vs pooled mismatch is expected + and is not a reproducibility measure. +- **Not done:** no inferential test; thread count as the cause of the difference is a hypothesis, not + verified; the metric files were not cross-checked beyond what `analyze_sweep.py` reads. +- **Noise checks** (`results/sweep/check_noise.py` -> `analysis/noise_checks.json`, no training): + all 30 pairs already differ at round 0 with identical sample counts; test rare-F1 differences + scatter around zero (14 higher / 16 lower, mean signed -0.0004, mean absolute 0.0034); 5G-NIDD is + about 2.7x noisier than X-IIoTID on the validation metric; beta-ranking Spearman is 0.30 / 0.50 + for 5G-NIDD alpha 0.75 / 0.5 and >= 0.90 elsewhere. Thread count remains **unproven and + confounded** with host (8 threads / 29 GiB documented earlier vs 12 / 48 GiB now), possibly a + different PyTorch version (not recorded in metadata), and a different checkout path. A sign test + (p = 0.86 / 1.0) is reported only descriptively because the pairs are not independent. +- **Write-up:** `docs/research/SWEEP_1THREAD_SEED0_CHECK.md` (untracked, not yet committed) + summarizes the sweep, the run-to-run difference, the noise checks and the comparison above. +- **Exact next task:** decide with the user whether the 5G-NIDD alpha 0.5 / 0.75 disagreement + warrants investigating the thread-count hypothesis (needs extra runs; not started) and whether + to commit the handoff and the write-up. Do not change the primary pooled selection on the basis of this single-seed re-run. +- **Housekeeping:** the tmux session `sweep` still exists (windows `control`, `pool`); the + `heartbeat` window was stopped on 2026-10-29. Backups of edited sweep scripts are `*.bak` in + `results/sweep/`; orphaned attempt-1 artifacts are in `results/sweep/_orphans_20261027T011213Z/`. + rsync destination was never specified and rsync was never run. +- **Git:** this addendum is an uncommitted edit on branch `chore/posthoc-audit-and-docs-refresh`; + nothing was committed or pushed. diff --git a/docs/research/SWEEP_1THREAD_SEED0_CHECK.md b/docs/research/SWEEP_1THREAD_SEED0_CHECK.md new file mode 100644 index 0000000..397915d --- /dev/null +++ b/docs/research/SWEEP_1THREAD_SEED0_CHECK.md @@ -0,0 +1,166 @@ +# 1-thread seed-0 re-run: validation selection vs. the 108-run grid + +Status: **exploratory consistency check**, single seed. It does not replace, correct or +re-select anything in `FINAL_EXPERIMENTAL_SYNTHESIS.md`. The primary selection of the thesis +remains the pooled validation across clients and seeds of the 108-run grid. + +## 1. What was run + +- 30 runs: 2 datasets (X-IIoTID, 5G-NIDD) x 3 alpha (0.25, 0.5, 0.75) x 5 beta + (0, 0.25, 0.5, 0.75, 1), `seed = 0`, `support_weighted` mode. +- 1 thread per training process, 3 processes in parallel. 2026-10-27T01:13Z to + 2026-10-28T18:00Z (2026-10-27 08:13 to 2026-10-29 01:00 WIB), about 40.8 h. +- All 30 runs ended `completed exit 0` with a `COMPLETE` marker; 0 failed, retry pass was empty. +- Outputs live in the git-ignored `results/sweep/` (runs, `summary.csv`, `analysis/`). They are + not committed. The orchestration record (including a gate-deadlock fix) is + `results/sweep/relaunch_report.md`, also untracked. + +Before the main queue continued, `reference_check.py` compared the first two runs (X-IIoTID +alpha 0.25 seed 0, beta 0 and 1) with the audit values; the maximum difference to the manual +verdict was 4.3e-10 and every |z| was at most 0.12. + +## 2. Method + +`results/sweep/analyze_sweep.py` (read-only, deterministic; two executions gave byte-identical +CSVs): + +1. **Stage 1, validation only.** For each (dataset, alpha) choose the beta with the highest + `val_rare_class_macro_f1_per_client_mean`; ties go to the smaller beta. `selection.csv` is + written before any test file is opened. +2. **Stage 2.** Test rare-class macro-F1 at every beta, read only after stage 1. + (`test_at_selected_beta.csv`.) Descriptive; not used to choose anything. +3. **Stage 3.** Absolute differences to the earlier grid run of the same + (dataset, alpha, seed 0, beta). (`repro_vs_previous.csv`.) + +`results/sweep/compare_with_grid.py` then compares the sweep's choice with the grid's seed-0 +choice and the grid's pooled choice, using validation metrics only. + +## 3. Results + +### 3.1 Run-to-run difference (30 pairs, same configuration, earlier run vs. 1-thread run) + +| metric | median abs. difference | max abs. difference | +|---|---|---| +| test rare-class macro-F1 | 0.0022 | 0.0127 (5G-NIDD, alpha 0.5, beta 0.75) | +| validation selection metric | 0.0029 | 0.0328 | + +None of the 30 pairs is exactly equal. The cause is not established. Thread count is a +hypothesis only; no experiment isolates it. + +### 3.2 Seed-0 selection + +| dataset | alpha | beta (sweep) | beta (grid, seed 0) | beta (grid, pooled) | sweep margin | grid s0 margin | +|---|---|---|---|---|---|---| +| X-IIoTID | 0.25 | 0.75 | 0.75 | 0 | 0.0413 | 0.0377 | +| X-IIoTID | 0.50 | 0.50 | 0.50 | 0.75 | 0.0074 | 0.0078 | +| X-IIoTID | 0.75 | 0.50 | 0.50 | 0.50 | 0.0004 | 0.0075 | +| 5G-NIDD | 0.25 | 0.25 | 0.25 | 0 | 0.0155 | 0.0109 | +| 5G-NIDD | 0.50 | 0.75 | 1.00 | 1.00 | 0.0053 | 0.0044 | +| 5G-NIDD | 0.75 | 0.25 | 0.50 | 0.50 | 0.0132 | 0.0021 | + +"Margin" is the validation score of the best beta minus that of the runner-up. + +- The sweep and grid seed-0 choices coincide in **4 of 6** conditions. +- The two disagreements are both 5G-NIDD. At alpha 0.5 the sweep's beta is rank 3 in the grid's + seed-0 ranking; at alpha 0.75 it is rank 4. These two conditions also show the largest + per-beta validation differences between the two runs (0.0328 and 0.0262). +- The sweep beta equals the grid **pooled** beta in 1 of 6, but the grid's own seed-0 beta + equals its pooled beta in only 3 of 6. A seed-0 versus pooled mismatch is therefore expected + and is not a measure of reproducibility. +- X-IIoTID alpha 0.75 has a sweep margin of 0.0004, far below the median validation run-to-run + difference (0.0029), yet its choice coincides with the grid's. The narrow margin did not + produce a different choice here. + +### 3.3 Test rare-class macro-F1 at the validation-selected beta (descriptive, seed 0) + +| dataset | alpha | selected beta | test @ selected | test @ beta 0 | +|---|---|---|---|---| +| X-IIoTID | 0.25 | 0.75 | 0.6692 | 0.6420 | +| X-IIoTID | 0.50 | 0.50 | 0.6439 | 0.6131 | +| X-IIoTID | 0.75 | 0.50 | 0.5068 | 0.5263 | +| 5G-NIDD | 0.25 | 0.25 | 0.8876 | 0.8612 | +| 5G-NIDD | 0.50 | 0.75 | 0.8793 | 0.8694 | +| 5G-NIDD | 0.75 | 0.25 | 0.9065 | 0.9024 | + +Selected minus beta 0: +0.0272, +0.0308, -0.0195 (X-IIoTID alpha 0.25 / 0.5 / 0.75) and ++0.0264, +0.0099, +0.0042 (5G-NIDD alpha 0.25 / 0.5 / 0.75). In 5 of 6 conditions the test value +at the validation-selected beta is above that at beta 0; in X-IIoTID alpha 0.75 it is below. + +Four gaps (absolute value 0.0195 to 0.0308) exceed the largest test run-to-run difference seen +in section 3.1 (0.0127). The two 5G-NIDD gaps at alpha 0.5 and 0.75 (0.0099 and 0.0042) lie +within the range of that difference (median 0.0022, max 0.0127) and should not be read as +improvements. The run-to-run difference compares two runs of one configuration; it is a rough +yardstick, not a noise bound for a difference between two betas. + +### 3.4 Noise checks (`results/sweep/check_noise.py` -> `analysis/noise_checks.json`) + +Read-only, no training. The 30 pairs are not independent (the five betas of a condition share +one partition), so the sign counts below are descriptive only. + +- **Where the runs first differ.** All 30 pairs already differ at round 0 (max absolute + difference in rare-class macro-F1 1.8e-3; 0 of 30 exactly equal). The number of evaluated + samples is identical in every round of every pair, so data and partitions are the same. The + difference is small at round 0 (median 3.1e-4) and **grows over the rounds** (median 2.1e-3 and + max 1.5e-2 at the last round; larger at the last round than at round 0 in 24 of 30 pairs). + A tiny first-round difference that is amplified by later rounds fits the pattern; it does not + identify what produced the first-round difference. +- **Direction.** Test rare-F1, sweep minus grid: 14 higher, 16 lower, mean signed -0.0004 against + mean absolute 0.0034. Validation selection metric: 15 higher, 15 lower, mean signed -0.0004 + against mean absolute 0.0057. The differences scatter around zero; there is no sign of a + systematic offset. (A two-sided sign test gives p = 0.86 and 1.0, but the pairs are dependent, + so this is not a significance claim.) +- **By dataset.** Mean absolute validation difference: X-IIoTID 0.0031, 5G-NIDD 0.0082 (about + 2.7 times larger). Test: 0.0026 vs 0.0042 (about 1.6 times). 5G-NIDD is the noisier dataset + in this comparison, consistent with both disagreements sitting there. +- **Ranking stability** (Spearman over the five betas, sweep vs grid, validation): X-IIoTID + 1.00 / 0.90 / 1.00 and 5G-NIDD 0.90 / 0.50 / 0.30 for alpha 0.25 / 0.5 / 0.75. Betas whose + sweep validation score lies within the condition's own maximum run-to-run difference of the + best beta (a crude yardstick): X-IIoTID alpha 0.25 {0.75}; alpha 0.5 {0.5, 0.75}; alpha 0.75 + {0, 0.25, 0.5}; 5G-NIDD alpha 0.25 {0.25}; alpha 0.5 {0, 0.75, 1}; alpha 0.75 {0, 0.25, 0.5}. + Selections with one beta in that set (X-IIoTID alpha 0.25, 5G-NIDD alpha 0.25) are the least + ambiguous; the others cannot separate their top candidates from the noise. +- **Third reference point** (X-IIoTID alpha 0.25 seed 0, test rare-F1; from the earlier + 5-thread smoke run recorded in `reference_check.json`): beta 0: smoke 0.6402, grid 0.6414, + sweep 0.6420; beta 1: smoke 0.5721, grid 0.5841, sweep 0.5931. For beta 1 the three setups + span 0.021, larger than the sweep-vs-grid difference alone (0.0089). + +### 3.5 What the thread hypothesis rests on + +- The earlier grid used several cores per process (CPU 741 to 746 % in the X-IIoTID alpha 0.25 + runtime table, about 7.4 cores); the sweep forces 1 thread (`thread1_launcher.py`, + `NLWP = 1` observed). Multi-threaded reductions can change floating-point summation order, so + a thread effect is plausible. `BASELINE_VALIDATION.md` (F.2) reports that two invocations of + `run_federated` with the same seed gave exactly equal results in a regression test on a small + fixture, and that no nondeterminism source was observed at the tested scales on this CPU + backend. That says a same-setup repeat was deterministic there; it does not show that full-size + runs are bit-reproducible, and it does not test the thread hypothesis. +- Confounders that were **not** controlled: the planning document recorded 8 CPU threads and + about 29 GiB RAM for the earlier work, whereas this host has `nproc = 12` and 48 GiB; the old + run metadata records a different repository path (`.../projects/perdalam-protean`); the + metadata stores no PyTorch version, so a library difference cannot be excluded (current: + 2.14.1+cpu). Whether the grid ran on a different machine is inferred, not verified. + +## 4. Interpretation limits + +- One seed. The grid pools three. Nothing here is a significance test. +- Where the sweep and the grid disagree (5G-NIDD alpha 0.5 and 0.75), a different beta is the + best of the tested candidates under a slightly different realization of training; this does + not show that either beta is better. +- Unverified: that thread count causes the run-to-run difference (it is confounded with hardware + and possibly library version, section 3.5). Two cheap runs would separate them: (a) repeat one + 1-thread run and compare it bit-for-bit with the sweep run (is a repeat identical?); (b) run + the same configuration multi-threaded on this host and compare with the grid. Neither has + been run. +- No claim is made that support weighting improves detection, and the selected beta is "best + among the tested candidates under the defined validation procedure", not a global optimum. +- `S_j` handling remains the documented reproduction assumption. + +## 5. Reproduce + +```bash +.venv/bin/python results/sweep/analyze_sweep.py +.venv/bin/python results/sweep/compare_with_grid.py +``` + +Both only read; they write into `results/sweep/analysis/`. The scripts and CSVs are untracked +because `results/` is git-ignored.