From ffbef1281237bbf4d97f49ee461f6d770784e7ca Mon Sep 17 00:00:00 2001 From: willtanoe Date: Fri, 30 Oct 2026 04:35:59 +0100 Subject: [PATCH] docs: add factual sweep summary report Server-side, read-only summary of the completed 30/30 seed-0 sweep: queue outcome, per-run wall time and peak RSS, OOM evidence, the reference check against the pre-fixed 2-sigma tolerance, deviations, four analysis-script outputs, and the reproducibility runs. Does not claim significance, recommend a beta, or commit data. --- kisikisithesis/SWEEP_SUMMARY_REPORT.md | 685 +++++++++++++++++++++++++ 1 file changed, 685 insertions(+) create mode 100644 kisikisithesis/SWEEP_SUMMARY_REPORT.md diff --git a/kisikisithesis/SWEEP_SUMMARY_REPORT.md b/kisikisithesis/SWEEP_SUMMARY_REPORT.md new file mode 100644 index 0000000..508ae92 --- /dev/null +++ b/kisikisithesis/SWEEP_SUMMARY_REPORT.md @@ -0,0 +1,685 @@ +# Sweep summary report (server-side, read-only) + +Generated: 2026-10-29 (UTC) by an agent session on the server that holds the +sweep outputs. Scope: the state of `results/sweep/` and `results/sweep_repro/` +as found on disk, plus the four analysis scripts now version-controlled under +`scripts/sweep/`. + +This report is descriptive. It does **not** claim statistical significance, does +**not** select or recommend a beta, and does **not** compare the sweep with the +paper. Where the analysis scripts emit a "selected" or "best" column, that +column is reproduced verbatim as script output, not adopted here. + +The raw data (`results/`) is git-ignored and is **not** committed. This report is +also not committed. + +--- + +## 1. Repository state at the start of the session + +Commands and output: + +``` +$ git status --short +?? kisikisithesis/RESTORE_VERIFICATION_REPORT.md +$ git branch --show-current +chore/posthoc-audit-and-docs-refresh +$ git log --oneline -5 +e66ecc4 docs: add 5G-NIDD repeat and multi-thread test (section 3.7) +b40c928 docs: add repeat and multi-thread test (section 3.6) for the 1-thread sweep +ad203eb docs: record 1-thread seed-0 sweep, noise checks and grid comparison +91b0e37 chore: declare matplotlib and apply three behavior-preserving nits +7831272 docs: refresh README and session handoff for current state +``` + +- `scripts/sweep/` did not exist. +- `git log --oneline --all -- scripts/sweep` returned nothing, so no commit + adding it existed. +- `git rev-parse main` and `git rev-parse origin/main` were both + `34bab458651b80c617490a75b94b27fdcbbc19fb`. +- The only pre-existing untracked file was + `kisikisithesis/RESTORE_VERIFICATION_REPORT.md` (not created by this session, + not touched, not committed). + +`results/sweep/` and `results/sweep_repro/` exist on this server, matching the +task statement that they are git-ignored (`.gitignore` line `results/`). + +--- + +## 2. Queue outcome + +Source: `results/sweep/queue.txt`, `results/sweep/summary.csv`, +`results/sweep/runs/*/COMPLETE`, `results/sweep/logs/pool.log`. + +- Queue rows: **30** (2 datasets x 3 alphas x 5 betas, seed 0). +- Final `summary.csv` status counts: **30 completed, 0 failed** (exit code 0 in + all 30 rows). +- `COMPLETE` markers on disk: **30**. +- `results/sweep/PHASE`: `done`. +- Last `HEARTBEAT` (2026-10-29T00:02:03Z): `rows: completed=30 failed=0 + pending=0 total=30`, `training_processes: 0`. +- The sweep is finished; no row is marked unfinished. + +Failed attempts that were later replaced in `summary.csv` (the upsert keeps one +row per config) are listed in section 4. + +--- + +## 3. Completed runs: wall time and peak memory + +Values are from `results/sweep/summary.csv`, which records `/usr/bin/time -v` +wall seconds and maximum resident set size for the completed attempt. Peak RSS +is converted from KiB to GiB (divide by 1,048,576). Rows are ordered by start +time. `wall` is `h:mm:ss`. + +| # | run | started (UTC) | ended (UTC) | wall | peak RSS (GiB) | status | +|---:|---|---|---|---:|---:|---| +| 1 | `5gnidd__alpha0.25__seed0__support_weighted__beta0` | 2026-10-26T07:31:29Z | 2026-10-26T12:55:25Z | 5:23:56 | 13.12 | completed | +| 2 | `xiiotid__alpha0.25__seed0__support_weighted__beta0` | 2026-10-26T07:31:29Z | 2026-10-26T11:02:12Z | 3:30:42 | 8.38 | completed | +| 3 | `xiiotid__alpha0.25__seed0__support_weighted__beta1` | 2026-10-26T07:31:29Z | 2026-10-26T11:06:03Z | 3:34:33 | 8.38 | completed | +| 4 | `5gnidd__alpha0.25__seed0__support_weighted__beta1` | 2026-10-26T11:02:12Z | 2026-10-26T16:21:43Z | 5:19:30 | 13.11 | completed | +| 5 | `xiiotid__alpha0.25__seed0__support_weighted__beta0.25` | 2026-10-27T01:13:26Z | 2026-10-27T04:25:41Z | 3:12:15 | 8.38 | completed | +| 6 | `5gnidd__alpha0.25__seed0__support_weighted__beta0.25` | 2026-10-27T01:13:26Z | 2026-10-27T06:22:48Z | 5:09:21 | 13.12 | completed | +| 7 | `xiiotid__alpha0.25__seed0__support_weighted__beta0.5` | 2026-10-27T01:13:26Z | 2026-10-27T04:25:12Z | 3:11:45 | 8.38 | completed | +| 8 | `5gnidd__alpha0.25__seed0__support_weighted__beta0.5` | 2026-10-27T04:25:12Z | 2026-10-27T09:46:00Z | 5:20:47 | 13.11 | completed | +| 9 | `xiiotid__alpha0.25__seed0__support_weighted__beta0.75` | 2026-10-27T04:25:41Z | 2026-10-27T07:55:55Z | 3:30:12 | 8.38 | completed | +| 10 | `5gnidd__alpha0.25__seed0__support_weighted__beta0.75` | 2026-10-27T06:22:48Z | 2026-10-27T11:46:25Z | 5:23:36 | 13.11 | completed | +| 11 | `xiiotid__alpha0.5__seed0__support_weighted__beta0` | 2026-10-27T07:55:55Z | 2026-10-27T11:20:41Z | 3:24:46 | 6.21 | completed | +| 12 | `5gnidd__alpha0.5__seed0__support_weighted__beta0` | 2026-10-27T09:46:01Z | 2026-10-27T15:20:57Z | 5:34:55 | 10.28 | completed | +| 13 | `xiiotid__alpha0.5__seed0__support_weighted__beta0.25` | 2026-10-27T11:20:42Z | 2026-10-27T14:55:30Z | 3:34:47 | 6.21 | completed | +| 14 | `5gnidd__alpha0.5__seed0__support_weighted__beta0.25` | 2026-10-27T11:46:25Z | 2026-10-27T17:36:49Z | 5:50:23 | 10.29 | completed | +| 15 | `xiiotid__alpha0.5__seed0__support_weighted__beta0.5` | 2026-10-27T14:55:31Z | 2026-10-27T18:44:43Z | 3:49:12 | 6.21 | completed | +| 16 | `5gnidd__alpha0.5__seed0__support_weighted__beta0.5` | 2026-10-27T15:20:58Z | 2026-10-27T21:26:19Z | 6:05:21 | 10.29 | completed | +| 17 | `xiiotid__alpha0.5__seed0__support_weighted__beta0.75` | 2026-10-27T17:36:49Z | 2026-10-27T21:35:28Z | 3:58:39 | 6.21 | completed | +| 18 | `5gnidd__alpha0.5__seed0__support_weighted__beta0.75` | 2026-10-27T18:44:44Z | 2026-10-28T00:42:07Z | 5:57:23 | 10.29 | completed | +| 19 | `xiiotid__alpha0.5__seed0__support_weighted__beta1` | 2026-10-27T21:26:20Z | 2026-10-28T01:05:49Z | 3:39:29 | 6.21 | completed | +| 20 | `5gnidd__alpha0.5__seed0__support_weighted__beta1` | 2026-10-27T21:35:29Z | 2026-10-28T03:18:55Z | 5:43:26 | 10.31 | completed | +| 21 | `xiiotid__alpha0.75__seed0__support_weighted__beta0` | 2026-10-28T00:42:07Z | 2026-10-28T04:23:48Z | 3:41:40 | 6.48 | completed | +| 22 | `5gnidd__alpha0.75__seed0__support_weighted__beta0` | 2026-10-28T01:05:49Z | 2026-10-28T06:53:35Z | 5:47:44 | 8.61 | completed | +| 23 | `xiiotid__alpha0.75__seed0__support_weighted__beta0.25` | 2026-10-28T03:18:56Z | 2026-10-28T07:05:59Z | 3:47:03 | 6.48 | completed | +| 24 | `5gnidd__alpha0.75__seed0__support_weighted__beta0.25` | 2026-10-28T04:23:48Z | 2026-10-28T10:17:38Z | 5:53:48 | 8.61 | completed | +| 25 | `xiiotid__alpha0.75__seed0__support_weighted__beta0.5` | 2026-10-28T06:53:35Z | 2026-10-28T10:40:37Z | 3:47:01 | 6.48 | completed | +| 26 | `5gnidd__alpha0.75__seed0__support_weighted__beta0.5` | 2026-10-28T07:06:00Z | 2026-10-28T12:40:40Z | 5:34:40 | 8.61 | completed | +| 27 | `xiiotid__alpha0.75__seed0__support_weighted__beta0.75` | 2026-10-28T10:17:38Z | 2026-10-28T13:32:01Z | 3:14:23 | 6.48 | completed | +| 28 | `5gnidd__alpha0.75__seed0__support_weighted__beta0.75` | 2026-10-28T10:40:37Z | 2026-10-28T15:26:24Z | 4:45:46 | 8.61 | completed | +| 29 | `xiiotid__alpha0.75__seed0__support_weighted__beta1` | 2026-10-28T12:40:40Z | 2026-10-28T15:37:02Z | 2:56:21 | 6.48 | completed | +| 30 | `5gnidd__alpha0.75__seed0__support_weighted__beta1` | 2026-10-28T13:32:01Z | 2026-10-28T18:00:12Z | 4:28:10 | 8.61 | completed | + +Totals over the 30 completed attempts: + +- Sum of wall time: **486,694 s = 135.19 h** (not pool wall time; runs overlap). +- X-IIoTID: n=15, wall 2:56:21-3:58:39, mean 3:31:31, max peak RSS 8.38 GiB. +- 5G-NIDD: n=15, wall 4:28:10-6:05:21, mean 5:29:15, max peak RSS 13.12 GiB. +- Global maximum peak RSS: **13.12 GiB** (`5gnidd alpha0.25 beta0` and + `5gnidd alpha0.25 beta0.25`). +- First pool start: 2026-10-26T02:10:14Z. Last completion: + 2026-10-28T18:00:12Z. Span: **2 days 15:49:58**. +- From the final relaunch (2026-10-27T01:13:26Z) to the last completion: + **1 day 16:46:46**. + +### 3.1 Reproducibility runs in `results/sweep_repro/` (not queue rows) + +These four runs are outside `results/sweep/summary.csv` and were launched with +`results/sweep_repro/run_variant.sh` (1-thread = `thread1_launcher.py`, as in the +sweep; 8-thread = plain `run_baseline.py`). They write `logs/start.epoch`, +`logs/end.epoch`, `logs/exit_code` and `/usr/bin/time -v` output instead of a +`COMPLETE` marker. All four exited 0 with empty `stderr.log`. + +| config | variant dir | start (UTC) | end (UTC) | wall | peak RSS (GiB) | exit | +|---|---|---:|---:|---:|---:|---:| +| xiiotid alpha0.25 seed0 beta0 | `a_1thread_repeat` | 2026-10-29T00:40:42Z | 2026-10-29T03:22:26Z | 2:41:44 | 8.38 | 0 | +| xiiotid alpha0.25 seed0 beta0 | `b_8thread` | 2026-10-29T00:40:42Z | 2026-10-29T01:56:28Z | 1:15:46 | 8.39 | 0 | +| 5gnidd alpha0.5 seed0 beta0.75 | `5gnidd_a0.5_b0.75/a_1thread_repeat` | 2026-10-29T11:57:21Z | 2026-10-29T16:36:33Z | 4:39:11 | 10.29 | 0 | +| 5gnidd alpha0.5 seed0 beta0.75 | `5gnidd_a0.5_b0.75/b_8thread` | 2026-10-29T11:57:21Z | 2026-10-29T14:39:23Z | 2:42:01 | 10.43 | 0 | + +Peak RSS converted from `/usr/bin/time -v` KiB (8,786,400 / 8,794,116 / +10,787,408 / 10,939,972). The two 1-thread repeats reproduce the corresponding +sweep runs byte-for-byte in the compared artifacts: for 5G-NIDD, +`test_global_metrics.json`, `val_selection.json`, `history.json`, +`test_per_class.csv` and `per_client.csv` are identical to +`results/sweep/runs/5gnidd__alpha0.5__seed0__support_weighted__beta0.75/...` +(`sha256sum` equal for all five); for X-IIoTID the same is recorded by +`compare_repro.json` (section 8.6). The 8-thread variants differ from the +1-thread repeats (5G-NIDD rare macro-F1 0.875941789 vs 0.879314583; X-IIoTID +0.643834668 vs 0.641979301). + +--- + +## 4. All attempts, including failed and retried rows + +Reconstructed from `results/sweep/logs/pool.log`, +`results/sweep/logs/continuation.log`, per-run `logs/run.log`, and the orphan +manifest. Filesystem timestamps quoted in UTC (`TZ=UTC stat`). + +| run | attempt | started (UTC) | ended / outcome | evidence | +|---|---:|---|---|---| +| `5gnidd alpha0.25 beta1` | 1 | 2026-10-26T02:10:14Z | 2026-10-26T02:22:50Z, exit 143 (OOM) | `runs/...beta1/logs/run.log:3`, `pool.log:11`; journal OOM-kill | +| `5gnidd alpha0.25 beta0` | 1 | 2026-10-26T02:10:14Z | 2026-10-26T02:22:53Z, exit 143 (OOM) | `runs/...beta0/logs/run.log:3`, `pool.log:16`; journal OOM-kill | +| `xiiotid alpha0.25 beta0` | 1 | 2026-10-26T02:10:14Z | no END line, no COMPLETE; second attempt started 07:31:29Z | `logs/run.log` shows two START lines and no END for the first | +| `xiiotid alpha0.25 beta1` | 1 | 2026-10-26T02:10:14Z | no END line, no COMPLETE; second attempt started 07:31:29Z | same | +| `xiiotid alpha0.25 beta0.25` | 1 | 2026-10-26T02:22:51Z | orphan; result files written 2026-10-26T05:41:53Z, no `time.txt`, no COMPLETE | `_orphans_20261027T011213Z/` | +| `5gnidd alpha0.25 beta0.25` | 1 | 2026-10-26T02:22:53Z | orphan; result files written 2026-10-26T07:29:08Z, no `time.txt`, no COMPLETE | `_orphans_20261027T011213Z/` | +| `xiiotid alpha0.25 beta0.25` | 2 (gate) | gate started 2026-10-26T11:06:04Z | 2026-10-26T23:06:09Z, exit 4, "reference gate did not resolve within 12 h" | `pool.log:220`, inspection report | +| `5gnidd alpha0.25 beta0.25` | 2 (gate) | gate started 2026-10-26T12:55:25Z | 2026-10-27T00:55:31Z, exit 4, same reason | `pool.log:50`, `pool.log:53`, `pool.log:253`; inspection report | +| `xiiotid alpha0.5 beta0` | 1 (gate only) | gate from 2026-10-26T16:21:43Z | wrapper SIGTERMed 2026-10-27T01:06-01:07Z before training | `pool.log` gate lines, relaunch report | +| `5gnidd alpha0.5 beta0` | 1 (gate only) | gate from 2026-10-26T23:06:09Z | wrapper SIGTERMed 2026-10-27T01:06-01:07Z before training | same | +| `xiiotid alpha0.75 beta0` | 1 (gate only) | gate from 2026-10-27T00:55:31Z | wrapper SIGTERMed 2026-10-27T01:06-01:07Z before training | same | + +All six configs above eventually appear as `completed` in `summary.csv` with the +start times in section 3. The automatic retry pass at +2026-10-28T18:00:12Z reported: + +``` +[2026-10-28T18:00:12Z] RETRY: no failed rows; nothing to do +[2026-10-28T18:00:12Z] SUPERVISOR retry pass finished rc=0 +[2026-10-28T18:00:12Z] SUPERVISOR done (main=0 retry=0) +``` + +Orphan artifacts were moved (not deleted) to +`results/sweep/_orphans_20261027T011213Z/` at 2026-10-27T01:12:13Z. The orphan +manifest records, for the record only, test rare macro-F1 0.645839538 +(X-IIoTID beta0.25) and 0.887639250 (5G-NIDD beta0.25); attempt-1 wall time was +never measured (`time.txt` 0 bytes). These are **not** the accepted results. + +The manifest prints filesystem times in host local time (Europe/Berlin, UTC+1 +on 2026-10-26); the UTC values quoted above come from `TZ=UTC stat` on the same +files and refer to the same inodes (identical nanosecond fractions). + +--- + +## 5. Out-of-memory events + +`journalctl -k` (kernel ring): + +``` +$ journalctl -k --no-pager | grep -iE 'oom|out of memory|killed process' +(no matching lines) +``` + +`journalctl -b --utc -o short-iso` (systemd journal) contains: + +``` +2026-10-26T02:01:37+00:00 aster systemd[2118]: run-r0a3b67e68dfa4510b7690d3eb1c285ef.scope: A process of this unit has been killed by the OOM killer. +2026-10-26T02:01:37+00:00 aster systemd[2118]: run-r0a3b67e68dfa4510b7690d3eb1c285ef.scope: Failed with result 'oom-kill'. +2026-10-26T02:01:37+00:00 aster systemd[2118]: app.slice: A process of this unit has been killed by the OOM killer. +2026-10-26T02:22:50+00:00 aster systemd[2118]: run-r351421db66ec4b9dabb5153489c169a3.scope: A process of this unit has been killed by the OOM killer. +2026-10-26T02:22:51+00:00 aster systemd[2118]: run-r351421db66ec4b9dabb5153489c169a3.scope: Failed with result 'oom-kill'. +2026-10-26T02:22:53+00:00 aster systemd[2118]: run-r349991742c5d4469859a140260c1cc20.scope: A process of this unit has been killed by the OOM killer. +2026-10-26T02:22:54+00:00 aster systemd[2118]: run-r349991742c5d4469859a140260c1cc20.scope: Failed with result 'oom-kill'. +2026-10-26T07:29:09+00:00 aster systemd[2118]: app.slice: A process of this unit has been killed by the OOM killer. +2026-10-28T11:11:37+00:00 aster systemd[2118]: -.slice: A process of this unit has been killed by the OOM killer. +``` + +Interpretation limited to what the logs show: + +- The 02:22:50 / 02:22:53 entries are the two 5G-NIDD first-wave run scopes; + both rows recorded exit 143. +- The 02:01:37 `run-r0a3b67e...` entry is before the queue started (pool start + 02:10:14). `launch_report.md` says the third kill in + `/sys/fs/cgroup/user.slice/memory.events` is "most likely the earlier + deliberate 512 MiB under a 200 MB cap test", i.e. not a queue row. That + attribution is the launch report's inference; it is not independently + confirmed here. +- The 07:29:09 `app.slice` entry coincides with the end of the 5G-NIDD beta0.25 + orphan attempt; the log does not name a run. +- The 2026-10-28T11:11:37 `-.slice` entry falls inside the completed + `5gnidd alpha0.75 beta0.75` run window, which exited 0. No sweep `run.log` + records a failure for it; its cause is not identified by these logs. +- `dmesg` is not readable by this user: + `dmesg: read kernel buffer failed: Operation not permitted`. + +Sweep logs themselves contain no literal "OOM" string. The only failure markers +are the two `END exit_code=143` lines in `pool.log`, each mirrored in the +corresponding `runs//logs/run.log` (four lines in total). + +--- + +## 6. Reference check against the pre-fixed tolerance + +`results/sweep/reference_check.md` reports **Status: PASS**, generated +`2026-10-27T01:10:36Z`. `results/sweep/reference_check.json` has +`complete: true`, `stop: false`, `stop_reasons: []`, `missing_betas: []`. + +Tolerance: the check compares each fresh value with the audit seed-0 value using +a band of **+/- 2 x between-seed standard deviation** (`statistics.stdev`, +ddof=1, over audit seeds 0,1,2). The comparison code is + +``` +within = abs(delta) <= 2 * std +``` + +and is identical in `reference_check.py.bak` (mtime +`2026-10-26T02:06:56Z`) and the current `reference_check.py`. The first reference +run started at `2026-10-26T02:10:14Z`, i.e. the tolerance predates the reference +results. `launch_report.md` (mtime `2026-10-26T02:31:53Z`) also documents the +2-sigma rule and was written before the reference runs finished (11:02Z / 11:06Z +on 2026-10-26). The passing verdict itself was produced later, after a key-name +fix at 2026-10-27T01:10:30Z. + +Values (from `reference_check.json`): + +| beta | metric | fresh | audit seed-0 | delta | z | 2-sigma band | within 2 sigma | +|---:|---|---:|---:|---:|---:|---|---| +| 0 | test rare macro-F1 | 0.641979301 | 0.641362228 | +0.000617073 | +0.013 | [0.546078671, 0.736645785] | true | +| 0 | val rare macro-F1 (per-client mean) | 0.615114398 | 0.616380161 | -0.001265762 | -0.014 | [0.437322488, 0.795437834] | true | +| 1 | test rare macro-F1 | 0.593073182 | 0.584131463 | +0.008941719 | +0.117 | [0.431612637, 0.736650288] | true | +| 1 | val rare macro-F1 (per-client mean) | 0.579765280 | 0.578384529 | +0.001380751 | +0.030 | [0.486947852, 0.669821206] | true | + +Both runs had `config_ok: true` and the expected loader content hash +`839ba18e2d87245d198be8f5f89d4a24f13cdb6fc64c004d406c6d4f8246f3da`. + +Earlier, `reference_check.py` crashed three times with +`KeyError: 'val_rare_macro_f1_per_client_mean'` +(`logs/reference_check.log`); this caused the 12-hour gate timeouts recorded in +section 4. The crash and the fix are documented in +`inspection_2026-10-27.md` and `relaunch_report.md`. These reports are inputs; +their conclusions were not re-derived here. + +--- + +## 7. Deviations from the original plan, with timestamps + +1. **Per-process memory cap changed for 5G-NIDD (11 GiB -> 15 GiB).** + The first wave used `MemoryMax=11G` for every run (`pool.log:6-9`). Two + 5G-NIDD runs were OOM-killed at 2026-10-26T02:22:50Z and 02:22:53Z. + The in-flight 5G-NIDD beta0.25 scope was raised live to 15 GiB around + 02:23Z (per launch report), and `run_one.sh` was changed to select + `5gnidd -> 15G`, others `-> 11G`; the pre-change backup + `run_one.sh.bak` has mtime **2026-10-26T02:28:30Z**. The relaunched pool + command lines show `MemoryMax=15G` for 5G-NIDD (`pool.log:26`, `33`, and + later). +2. **Pool size changed (P=4 -> P=3).** + First pool: `P=4`, started **2026-10-26T02:10:14Z** (`pool.log:1`). + Steady-state pool: `P=3`, started **2026-10-26T07:31:29Z** (`pool.log:20`). + Final relaunch: `P=3`, pool start **2026-10-27T01:13:25Z** + (`pool.log:263`; first run command at 01:13:26Z, `pool.log:274`). + `pool.size` mtime 2026-10-27T01:13:25Z. P=5 was never used. +3. **2-slot 5G-NIDD semaphore added to `run_one.sh`.** + `run_one.sh` mtime **2026-10-27T01:10:59Z**; `pool.log` shows + `SEMAPHORE acquired slot1/slot2` from the 01:13:26Z relaunch onward. +4. **Reference-check bug fixed and gate guard added.** + `reference_check.py` mtime **2026-10-27T01:10:30Z** (key names corrected). + `run_one.sh` mtime **2026-10-27T01:10:59Z** (30-minute gate guard plus + semaphore). `status.sh` mtime **2026-10-27T01:51:16Z** (cosmetic). +5. **Retried / re-run rows.** `5gnidd alpha0.25 beta0` and `beta1` (OOM, first + attempt 02:10:14Z, retried 07:31:29Z / 11:02:12Z); `xiiotid alpha0.25 beta0` + and `beta1` (first attempt 02:10:14Z has no END, re-run 07:31:29Z); + `xiiotid alpha0.25 beta0.25` and `5gnidd alpha0.25 beta0.25` (orphan attempt + at 02:22:5xZ, gate attempt failed at 23:06:09Z / 00:55:31Z, final attempt + 01:13:26Z). See section 4 for evidence. +6. **Gate-waiting processes were terminated before training.** + At 2026-10-27T01:06-01:07Z the pool was stopped; three rows were waiting in + the gate loop (`xiiotid 0.5`, `5gnidd 0.5`, `xiiotid 0.75`). They started + later under the final relaunch. `pool.log:259-260` records the SIGTERM. +7. **Orphan attempts moved aside.** `_orphans_20261027T011213Z/` created + 2026-10-27T01:12:13Z; no result file was deleted. +8. **No queue row was skipped.** The 2026-10-21 5-thread smoke outputs were not + accepted as sweep rows; all 30 configs were run under the 1-thread protocol. +9. **Host timezone.** `timedatectl` reports `Europe/Berlin (CET, +0100)`; all + timestamps in this report are UTC, converted with `journalctl --utc` or + `TZ=UTC stat`. Some pre-existing sweep reports mix local and UTC times. + +--- + +## 8. Outputs of the four analysis scripts + +The four scripts were copied to `scripts/sweep/` and run against a scratch copy +of the data (so the real `results/` was not written). Their outputs were +byte-identical to the originals (section 9). The tables below are those outputs. + +`analyze_sweep.py` writes three files. Stage 1 uses validation only; the test +files are opened after `selection.csv` is written. + +### 8.1 `selection.csv` (validation only, 30 rows) + +The `selected` column is the script's stage-1 flag (highest +`val_rare_class_macro_f1_per_client_mean`, ties to the smaller beta). It is +reproduced as script output; this report does not adopt it. + +| dataset | alpha | beta | val_rare_f1_per_client_mean | val_rare_f1_global | val_rare_recall_per_client_mean | val_macro_f1_global | selected | margin_to_runner_up | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| xiiotid | 0.25 | 0 | 0.615114398 | 0.643275275 | 0.846913348 | 0.676267875 | 0 | | +| xiiotid | 0.25 | 0.25 | 0.612295414 | 0.647422433 | 0.847371095 | 0.680035834 | 0 | | +| xiiotid | 0.25 | 0.5 | 0.590996277 | 0.632833153 | 0.831341674 | 0.667045341 | 0 | | +| xiiotid | 0.25 | 0.75 | 0.656437133 | 0.669066681 | 0.870359212 | 0.699686253 | 1 | 0.041322734 | +| xiiotid | 0.25 | 1 | 0.579765280 | 0.593760832 | 0.784814026 | 0.633116555 | 0 | | +| xiiotid | 0.5 | 0 | 0.556929048 | 0.612060535 | 0.885829017 | 0.709691170 | 0 | | +| xiiotid | 0.5 | 0.25 | 0.562538987 | 0.620452943 | 0.889189065 | 0.716471910 | 0 | | +| xiiotid | 0.5 | 0.5 | 0.588286653 | 0.644186102 | 0.896470166 | 0.732532602 | 1 | 0.007362956 | +| xiiotid | 0.5 | 0.75 | 0.580923697 | 0.636817927 | 0.889355473 | 0.726043174 | 0 | | +| xiiotid | 0.5 | 1 | 0.558192955 | 0.614928577 | 0.857825220 | 0.714886486 | 0 | | +| xiiotid | 0.75 | 0 | 0.497123071 | 0.532261516 | 0.766962632 | 0.748317495 | 0 | | +| xiiotid | 0.75 | 0.25 | 0.499599429 | 0.526397734 | 0.756669120 | 0.748892418 | 0 | | +| xiiotid | 0.75 | 0.5 | 0.499968468 | 0.513382502 | 0.765223179 | 0.741108809 | 1 | 0.000369040 | +| xiiotid | 0.75 | 0.75 | 0.490886014 | 0.510831286 | 0.761348983 | 0.741545715 | 0 | | +| xiiotid | 0.75 | 1 | 0.477724598 | 0.507546185 | 0.762276080 | 0.741271747 | 0 | | +| 5gnidd | 0.25 | 0 | 0.732063562 | 0.863288898 | 0.888413711 | 0.833210589 | 0 | | +| 5gnidd | 0.25 | 0.25 | 0.750234422 | 0.887687791 | 0.886155769 | 0.849340729 | 1 | 0.015462491 | +| 5gnidd | 0.25 | 0.5 | 0.715027757 | 0.818058015 | 0.885062004 | 0.804059192 | 0 | | +| 5gnidd | 0.25 | 0.75 | 0.711752341 | 0.795364214 | 0.877813860 | 0.802485891 | 0 | | +| 5gnidd | 0.25 | 1 | 0.734771931 | 0.801655038 | 0.874817852 | 0.818441666 | 0 | | +| 5gnidd | 0.5 | 0 | 0.739471675 | 0.867651737 | 0.964727218 | 0.835451455 | 0 | | +| 5gnidd | 0.5 | 0.25 | 0.657948994 | 0.826207029 | 0.964727218 | 0.799036116 | 0 | | +| 5gnidd | 0.5 | 0.5 | 0.731662212 | 0.862708327 | 0.964727218 | 0.836388230 | 0 | | +| 5gnidd | 0.5 | 0.75 | 0.771920820 | 0.875152730 | 0.964727218 | 0.849873996 | 1 | 0.005265685 | +| 5gnidd | 0.5 | 1 | 0.766655135 | 0.876266492 | 0.964727218 | 0.846137526 | 0 | | +| 5gnidd | 0.75 | 0 | 0.836358274 | 0.896193888 | 0.964226565 | 0.851335066 | 0 | | +| 5gnidd | 0.75 | 0.25 | 0.849565876 | 0.902762083 | 0.964226565 | 0.856336364 | 1 | 0.013207603 | +| 5gnidd | 0.75 | 0.5 | 0.830443462 | 0.901146966 | 0.963185589 | 0.855355109 | 0 | | +| 5gnidd | 0.75 | 0.75 | 0.815299394 | 0.897742377 | 0.963715390 | 0.853576206 | 0 | | +| 5gnidd | 0.75 | 1 | 0.808079811 | 0.894306659 | 0.965304794 | 0.851265425 | 0 | | + +### 8.2 `test_at_selected_beta.csv` (test, 30 rows) + +`selected_on_validation` marks the beta flagged in 8.1. `test_rare_support` is +the number of test samples in the rare classes. + +| dataset | alpha | beta | selected_on_validation | test_rare_f1_global | test_rare_recall_global | test_macro_f1 | test_accuracy | test_rare_support | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| xiiotid | 0.25 | 0 | 0 | 0.641979301 | 0.830387426 | 0.675034969 | 0.873838520 | 79180 | +| xiiotid | 0.25 | 0.25 | 0 | 0.645839538 | 0.824822512 | 0.678550791 | 0.868704362 | 79180 | +| xiiotid | 0.25 | 0.5 | 0 | 0.631488471 | 0.793292581 | 0.665776684 | 0.854517874 | 79180 | +| xiiotid | 0.25 | 0.75 | 1 | 0.669170594 | 0.794023733 | 0.699726946 | 0.855561902 | 79180 | +| xiiotid | 0.25 | 1 | 0 | 0.593073182 | 0.704711149 | 0.632415448 | 0.849138063 | 79180 | +| xiiotid | 0.5 | 0 | 0 | 0.613084485 | 0.871186482 | 0.710555367 | 0.896929946 | 25454 | +| xiiotid | 0.5 | 0.25 | 0 | 0.622176085 | 0.873858741 | 0.717774657 | 0.897568645 | 25454 | +| xiiotid | 0.5 | 0.5 | 1 | 0.643904823 | 0.874747143 | 0.732555353 | 0.898287181 | 25454 | +| xiiotid | 0.5 | 0.75 | 0 | 0.636012409 | 0.863519815 | 0.725833853 | 0.886133476 | 25454 | +| xiiotid | 0.5 | 1 | 0 | 0.615308344 | 0.838787843 | 0.715340930 | 0.881257254 | 25454 | +| xiiotid | 0.75 | 0 | 0 | 0.526279577 | 0.819166043 | 0.745053791 | 0.912854432 | 6251 | +| xiiotid | 0.75 | 0.25 | 0 | 0.520979721 | 0.806403017 | 0.745946340 | 0.916207602 | 6251 | +| xiiotid | 0.75 | 0.5 | 1 | 0.506784362 | 0.796163189 | 0.737595025 | 0.911871818 | 6251 | +| xiiotid | 0.75 | 0.75 | 0 | 0.507162154 | 0.807061731 | 0.739196779 | 0.913057096 | 6251 | +| xiiotid | 0.75 | 1 | 0 | 0.504951940 | 0.817143701 | 0.739630728 | 0.911939373 | 6251 | +| 5gnidd | 0.25 | 0 | 0 | 0.861244967 | 0.984948540 | 0.832176777 | 0.744659240 | 141677 | +| 5gnidd | 0.25 | 0.25 | 1 | 0.887639250 | 0.980086383 | 0.850377106 | 0.740711498 | 141677 | +| 5gnidd | 0.25 | 0.5 | 0 | 0.818078455 | 0.977174954 | 0.804997750 | 0.740863651 | 141677 | +| 5gnidd | 0.25 | 0.75 | 0 | 0.795982829 | 0.949513575 | 0.804428539 | 0.730101942 | 141677 | +| 5gnidd | 0.25 | 1 | 0 | 0.802243321 | 0.935212949 | 0.819876826 | 0.752912488 | 141677 | +| 5gnidd | 0.5 | 0 | 0 | 0.869410144 | 0.974261201 | 0.836168314 | 0.735143537 | 104843 | +| 5gnidd | 0.5 | 0.25 | 0 | 0.827176768 | 0.974219638 | 0.799716621 | 0.728399479 | 104843 | +| 5gnidd | 0.5 | 0.5 | 0 | 0.868956930 | 0.974261221 | 0.840088463 | 0.739103616 | 104843 | +| 5gnidd | 0.5 | 0.75 | 1 | 0.879314583 | 0.974272043 | 0.853591585 | 0.745403554 | 104843 | +| 5gnidd | 0.5 | 1 | 0 | 0.878757266 | 0.974272032 | 0.848514697 | 0.741887596 | 104843 | +| 5gnidd | 0.75 | 0 | 0 | 0.902386981 | 0.959484554 | 0.855113260 | 0.747373312 | 28000 | +| 5gnidd | 0.75 | 0.25 | 1 | 0.906545156 | 0.960038373 | 0.858596769 | 0.749429428 | 28000 | +| 5gnidd | 0.75 | 0.5 | 0 | 0.904655697 | 0.958552306 | 0.857440422 | 0.749585693 | 28000 | +| 5gnidd | 0.75 | 0.75 | 0 | 0.901701106 | 0.960284499 | 0.855855667 | 0.750461598 | 28000 | +| 5gnidd | 0.75 | 1 | 0 | 0.899195897 | 0.964090767 | 0.854143914 | 0.750432812 | 28000 | + +### 8.3 `repro_vs_previous.csv` (30 rows, absolute differences) + +Differences are `|new - previous grid|`; the script prints them with three +significant digits. "new" is the 1-thread sweep run; "previous" is the earlier +grid run at the same `(dataset, alpha, seed0, beta)`. + +| dataset | alpha | beta | test_rare_f1_new | test_rare_f1_prev | abs_diff_test_rare_f1 | abs_diff_test_macro_f1 | abs_diff_test_accuracy | abs_diff_val_sel_metric | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| xiiotid | 0.25 | 0 | 0.641979301 | 0.641362228 | 6.171e-04 | 4.767e-04 | 7.124e-04 | 1.266e-03 | +| xiiotid | 0.25 | 0.25 | 0.645839538 | 0.645064558 | 7.750e-04 | 7.555e-04 | 2.070e-03 | 2.383e-03 | +| xiiotid | 0.25 | 0.5 | 0.631488471 | 0.631742435 | 2.540e-04 | 1.888e-04 | 1.302e-03 | 4.251e-03 | +| xiiotid | 0.25 | 0.75 | 0.669170594 | 0.666176967 | 2.994e-03 | 2.684e-03 | 9.151e-04 | 2.404e-03 | +| xiiotid | 0.25 | 1 | 0.593073182 | 0.584131463 | 8.942e-03 | 7.927e-03 | 2.045e-03 | 1.381e-03 | +| xiiotid | 0.5 | 0 | 0.613084485 | 0.613575398 | 4.909e-04 | 7.455e-05 | 4.956e-03 | 4.106e-04 | +| xiiotid | 0.5 | 0.25 | 0.622176085 | 0.632154112 | 9.978e-03 | 7.525e-03 | 5.951e-03 | 9.248e-03 | +| xiiotid | 0.5 | 0.5 | 0.643904823 | 0.640829406 | 3.075e-03 | 1.813e-03 | 1.290e-04 | 2.525e-03 | +| xiiotid | 0.5 | 0.75 | 0.636012409 | 0.636188757 | 1.763e-04 | 4.646e-04 | 2.536e-03 | 2.921e-03 | +| xiiotid | 0.5 | 1 | 0.615308344 | 0.615509445 | 2.011e-04 | 1.042e-03 | 3.378e-03 | 6.750e-03 | +| xiiotid | 0.75 | 0 | 0.526279577 | 0.525867706 | 4.119e-04 | 6.807e-04 | 8.905e-04 | 3.200e-04 | +| xiiotid | 0.75 | 0.25 | 0.520979721 | 0.520851745 | 1.280e-04 | 7.959e-04 | 1.069e-03 | 2.708e-03 | +| xiiotid | 0.75 | 0.5 | 0.506784362 | 0.511888897 | 5.105e-03 | 4.382e-03 | 2.493e-03 | 4.419e-03 | +| xiiotid | 0.75 | 0.75 | 0.507162154 | 0.502310077 | 4.852e-03 | 2.765e-03 | 7.799e-04 | 4.018e-03 | +| xiiotid | 0.75 | 1 | 0.504951940 | 0.506063809 | 1.112e-03 | 2.007e-04 | 1.535e-03 | 1.421e-03 | +| 5gnidd | 0.25 | 0 | 0.861244967 | 0.859892744 | 1.352e-03 | 2.497e-03 | 1.851e-04 | 4.297e-03 | +| 5gnidd | 0.25 | 0.25 | 0.887639250 | 0.885537113 | 2.102e-03 | 5.700e-04 | 1.147e-03 | 2.936e-03 | +| 5gnidd | 0.25 | 0.5 | 0.818078455 | 0.824837056 | 6.759e-03 | 3.770e-03 | 3.742e-03 | 2.423e-03 | +| 5gnidd | 0.25 | 0.75 | 0.795982829 | 0.804422938 | 8.440e-03 | 2.071e-03 | 1.871e-02 | 1.510e-03 | +| 5gnidd | 0.25 | 1 | 0.802243321 | 0.805922621 | 3.679e-03 | 2.820e-03 | 1.398e-03 | 3.167e-04 | +| 5gnidd | 0.5 | 0 | 0.869410144 | 0.874293205 | 4.883e-03 | 2.393e-03 | 2.920e-04 | 8.742e-04 | +| 5gnidd | 0.5 | 0.25 | 0.827176768 | 0.827855971 | 6.792e-04 | 5.151e-03 | 3.763e-03 | 3.512e-03 | +| 5gnidd | 0.5 | 0.5 | 0.868956930 | 0.878225000 | 9.268e-03 | 4.119e-03 | 1.583e-03 | 3.283e-02 | +| 5gnidd | 0.5 | 0.75 | 0.879314583 | 0.866626856 | 1.269e-02 | 5.548e-03 | 1.361e-03 | 2.126e-02 | +| 5gnidd | 0.5 | 1 | 0.878757266 | 0.880962035 | 2.205e-03 | 1.955e-03 | 9.499e-04 | 2.222e-03 | +| 5gnidd | 0.75 | 0 | 0.902386981 | 0.900366644 | 2.020e-03 | 1.505e-03 | 4.811e-04 | 1.677e-03 | +| 5gnidd | 0.75 | 0.25 | 0.906545156 | 0.901681021 | 4.864e-03 | 3.485e-03 | 9.129e-04 | 2.618e-02 | +| 5gnidd | 0.75 | 0.5 | 0.904655697 | 0.904447205 | 2.085e-04 | 1.315e-04 | 7.813e-05 | 6.348e-03 | +| 5gnidd | 0.75 | 0.75 | 0.901701106 | 0.904214210 | 2.513e-03 | 1.576e-03 | 1.398e-04 | 9.367e-03 | +| 5gnidd | 0.75 | 1 | 0.899195897 | 0.900797707 | 1.602e-03 | 1.031e-03 | 2.879e-05 | 7.694e-03 | + +### 8.4 `check_noise.py` -> `noise_checks.json` (excerpt) + +The full file is 380 lines (379 newlines; no trailing newline) and was written +unchanged. Key blocks: + +- `A_summary.first_round_with_difference_counts`: `{"0": 30}` — every paired run + first differs from the earlier grid in round 0, and + `n_samples_equal_all_rounds` is `true` for all 30 pairs. +- `B_sign.test_rare_f1`: n=30, sweep higher 14, lower 16, mean signed diff + -0.0004104998, mean absolute diff 0.0034124859. The script labels its + two-sided sign-test p-value `0.8555355519` as + `NOT_independent_pairs`; the 30 pairs share partitions across betas. This + report does not interpret it. +- `B_sign.val_selection_metric`: n=30, sweep higher 15, lower 15, mean signed + diff -0.0004188014, mean absolute diff 0.0056621689; script p-value + `1.0`, again labelled as non-independent. +- `C_rank` (rank stability, sweep vs earlier seed-0 grid over 5 betas): + +| dataset | alpha | spearman | max_abs_val_diff | betas_within_max_diff_of_sweep_best | +|---|---:|---:|---:|---| +| xiiotid | 0.25 | 1.0 | 0.004251150 | `0.75` | +| xiiotid | 0.5 | 0.9 | 0.009248424 | `0.5`, `0.75` | +| xiiotid | 0.75 | 1.0 | 0.004419246 | `0`, `0.25`, `0.5` | +| 5gnidd | 0.25 | 0.9 | 0.004296561 | `0.25` | +| 5gnidd | 0.5 | 0.5 | 0.032825056 | `0`, `0.75`, `1` | +| 5gnidd | 0.75 | 0.3 | 0.026177811 | `0`, `0.25`, `0.5` | + +### 8.5 `compare_with_grid.py` -> `compare_with_grid.csv` (6 rows) + +The `beta_*` columns are script output (validation-only ranking), not a +recommendation. + +| dataset | alpha | beta_sweep_seed0 | beta_grid_seed0 | beta_grid_pooled | sweep_equals_grid_s0 | sweep_equals_pooled | grid_s0_equals_pooled | sweep_margin | grid_s0_margin | max_abs_val_diff_s0 | sweep_pick_rank_in_grid_s0 | +|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| xiiotid | 0.25 | 0.75 | 0.75 | 0.0 | 1 | 0 | 0 | 0.041323 | 0.037653 | 0.004251 | 1 | +| xiiotid | 0.5 | 0.5 | 0.5 | 0.75 | 1 | 0 | 0 | 0.007363 | 0.007759 | 0.009248 | 1 | +| xiiotid | 0.75 | 0.5 | 0.5 | 0.5 | 1 | 1 | 1 | 0.000369 | 0.007496 | 0.004419 | 1 | +| 5gnidd | 0.25 | 0.25 | 0.25 | 0.0 | 1 | 0 | 0 | 0.015462 | 0.010939 | 0.004297 | 1 | +| 5gnidd | 0.5 | 0.75 | 1.0 | 1.0 | 0 | 0 | 1 | 0.005266 | 0.004390 | 0.032825 | 3 | +| 5gnidd | 0.75 | 0.25 | 0.5 | 0.5 | 0 | 0 | 1 | 0.013208 | 0.002110 | 0.026178 | 4 | + +### 8.6 `compare_repro.py` -> `compare_repro.json` and stdout + +Printed table: + +``` +a_1thread_repeat__vs__sweep test_rare=0.000e+00 val_sel=0.000e+00 round0=0.000e+00 last=0.000e+00 all_json_equal=True snapshot_identical=True +b_8thread__vs__grid test_rare=2.472e-03 val_sel=8.021e-04 round0=4.840e-05 last=2.242e-03 all_json_equal=False snapshot_identical=False +a_1thread_repeat__vs__grid test_rare=6.171e-04 val_sel=1.266e-03 round0=4.552e-05 last=3.359e-04 all_json_equal=False snapshot_identical=False +b_8thread__vs__sweep test_rare=1.855e-03 val_sel=4.637e-04 round0=9.392e-05 last=1.906e-03 all_json_equal=False snapshot_identical=False +sweep__vs__grid test_rare=6.171e-04 val_sel=1.266e-03 round0=4.552e-05 last=3.359e-04 all_json_equal=False snapshot_identical=False +a_1thread_repeat__vs__b_8thread test_rare=1.855e-03 val_sel=4.637e-04 round0=9.392e-05 last=1.906e-03 all_json_equal=False snapshot_identical=False +``` + +Facts recorded in the JSON: + +- `a_1thread_repeat` vs `sweep`: all four JSON files byte-identical and + JSON-equal; prototype snapshot byte-identical; test rare F1 and validation + metric differences 0.0; all ten per-round rare-F1 differences 0.0. +- `b_8thread` vs `grid`: JSON files not byte-identical; test rare F1 + b_8thread 0.643834668 vs grid 0.641362228, abs diff 0.002472440; validation + metric abs diff 0.000802084; prototype snapshot differs. +- `a_1thread_repeat` vs `grid`: test rare F1 0.641979301 vs 0.641362228, + abs diff 0.000617073; validation metric abs diff 0.001265762. +- The remaining pairs are context and show the same pattern (different + runs differ; the repeated 1-thread run is identical to the sweep run). + +This script analyses only the X-IIoTID `alpha0.25 seed0 beta0` pair. A second +reproducibility pair exists under +`results/sweep_repro/5gnidd_a0.5_b0.75/` (5G-NIDD `alpha0.5 seed0 beta0.75`, +1-thread and 8-thread). `compare_repro.py` does not analyse it; its wall time, +peak RSS and the byte-identical 1-thread comparison are reported in section 3.1. + +--- + +## 9. Scripts moved into version control + +Commit on branch `chore/posthoc-audit-and-docs-refresh`: + +``` +9fe584d08e0471ed517cb6f781e522f99d002520 +chore: version-control sweep analysis scripts +``` + +Pushed without force (`e66ecc4..9fe584d`); after the push `HEAD` and +`origin/chore/posthoc-audit-and-docs-refresh` both pointed at +`9fe584d08e0471ed517cb6f781e522f99d002520`. `main` and `origin/main` both +remained at `34bab458651b80c617490a75b94b27fdcbbc19fb`, and `9fe584d` is not an +ancestor of `main`. + +Files added (5 files, 697 insertions): + +``` +scripts/sweep/README.md +scripts/sweep/analyze_sweep.py +scripts/sweep/check_noise.py +scripts/sweep/compare_repro.py +scripts/sweep/compare_with_grid.py +``` + +The originals remain in `results/sweep/` and `results/sweep_repro/`. The copies +change only path resolution: a `--repo-root` argument with a default of +`/` derived from `scripts/sweep/` (the originals derived the root from +their old location inside `results/`). No computation was changed. + +Equivalence was tested on a scratch copy of the data +(`~/restore_work/sweep_scripts_check`), not on the real `results/`. Original +scripts were run from a mirrored `results/sweep/` layout; copies were run with +`--repo-root`. Comparison: + +``` +IDENTICAL results/sweep/analysis/selection.csv 37df4e7b13fedf3436c04697fa7660125e08118e14fe4786ac67bcd546c43ac7 +IDENTICAL results/sweep/analysis/test_at_selected_beta.csv df68a30929c59c76773c92323fcdbd30dfdb8bb5de3366dd05dc7a4819b70614 +IDENTICAL results/sweep/analysis/repro_vs_previous.csv 408bfac4b324e00bc21362fb3766df6208d8787eae3612428f229f8aa7426696 +IDENTICAL results/sweep/analysis/noise_checks.json c23509183a8c69f4584b8d09083f4dd0c0ef10a1b63dba62e1677ab52375b38a +IDENTICAL results/sweep/analysis/compare_with_grid.csv b995d8541134ee3522a5f28aa4cae097b68e2cd28b2c6fd82bc3ce5c4f04c48d +IDENTICAL results/sweep_repro/compare_repro.json 9d2e73c4021aceb89925f879100f562db8e5e996bcbe043f443d369ac088317c +``` + +Stdout tables were identical after normalizing only the printed root path +(`data_orig` vs `data_new`); the only textual difference was the path in the +"written ..." lines. Running the copies with the default root also reproduced +the same six files. The same six hashes were also compared with the on-disk +`results/` outputs and matched (`MATCH` for all six). The committed copies are +byte-identical to the copies used in that scratch run (`diff` reported no +difference for all four scripts). At the end of this session all four copies +were re-run on the scratch mirror with the repository virtualenv, e.g. + +``` +$ .venv/bin/python data_new/scripts/sweep/analyze_sweep.py --repo-root data_new +``` + +Each returned `rc=0`, and the six output files again hashed exactly as in the +block above; stdout was identical to the original scripts after normalizing the +mirror directory name (`data_orig` vs `data_new`). + +The existing test suite was run with the repository virtualenv (system +`python3` has no `pytest` or `numpy`, so `python3 -m pytest` fails with +`No module named pytest`; the venv is `.venv/`). It was run after the commit +was pushed; the commit contains no test or source changes beyond the five files +listed above: + +``` +$ PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -m pytest -q -p no:cacheprovider +........................................................................ [ 96%] +... [100%] +$ echo $? +0 +``` + +75 test nodes ran, all passed (75 `.`, 0 `F`, 0 `E`, 0 `s`), exit code 0. +`python3 -m py_compile scripts/sweep/*.py` also succeeded (`rc=0`; the +resulting `scripts/sweep/__pycache__/` was removed afterwards, so the working +tree contains only the five committed files). + +Write-target check: the only write sites in the copies are +`results/sweep/analysis/*` (three files from `analyze_sweep.py`, +`noise_checks.json` from `check_noise.py`, `compare_with_grid.csv` from +`compare_with_grid.py`) and +`results/sweep_repro/compare_repro.json`. `git ls-files results/` is empty, so +none of these is tracked. The scratch runs wrote only inside +`~/restore_work/sweep_scripts_check/`. + +Credential/privacy scan: the staged files were scanned before the commit, and +the committed blobs were re-scanned during this session with +`git show HEAD:scripts/sweep/` concatenated into one 697-line stream: + +``` +$ grep -cEi '(api[_-]?key|secret|access[_-]?token|auth[_-]?token|password|passwd|credential|bearer |private[_-]?key|BEGIN [A-Z ]*PRIVATE KEY|ssh-rsa|ssh-ed25519|ghp_|github_pat_|AKIA[0-9A-Z]{16}|xox[baprs]-)' +0 # grep exit code 1 +$ grep -cE '(/home/|/Users/|/root/|~/|localhost|github-thesis|willtanoe|([0-9]{1,3}\.){3}[0-9]{1,3})' +0 # grep exit code 1 +``` + +Both scans returned **0 matches**. The generic high-entropy scan matched +only identifier-like strings such as +`val_rare_class_macro_f1_per_client_mean` and path fragments +`results/sweep_repro/a_1thread_repeat/...`; these are not credentials. A +case-insensitive scan for `aster` matched the substring in "master" in +`compare_with_grid.py` (`final__six_condition_master.csv`), not a hostname. + +--- + +## 10. Server-side hashes + +`~/restore_work/final_hashes.txt` contains the sorted SHA-256 list of every file +under `results/sweep/` and `results/sweep_repro/` on this server. + +Command used: + +``` +find results/sweep results/sweep_repro -type f -print0 \ + | xargs -0 sha256sum | LC_ALL=C sort -k2 +``` + +- Files hashed: **912** (`results/sweep`: 818, `results/sweep_repro`: 94). +- Hashing was done in two complete independent passes, both finishing + 2026-10-29T17:21:11Z. `diff pass1 pass2` reported no differences, so **no + file changed during hashing** and the file set did not change between passes + (the file header states this explicitly). +- At hashing time `results/sweep/PHASE` was `done`, the last heartbeat was + `2026-10-29T00:02:03Z`, and no `run_one.sh` / `xargs` / `launch_all` process + was running. +- The written manifest was read back with `sha256sum -c`: **912 OK, 0 failures** + (`verify rc=0`) at 2026-10-29T17:21:20Z, and again at the end of the session + at 2026-10-29T17:27:50Z with the same result (912 OK, 0 failures, file count + still 912). Nothing under `results/` was written by this session. +- The manifest itself is not committed (it lives in `~/restore_work/`, outside + the repository). + +--- + +## 11. Open issues / unverified items + +1. **Test suite required the repository virtualenv.** `.venv/bin/python -m + pytest` passes (75 tests); the system `python3 -m pytest` fails with + `No module named pytest`. Nothing was installed to change that. +2. **First attempts of `xiiotid alpha0.25 beta0/beta1` have no END line and no + measured wall time.** Their run directories were overwritten by the + 07:31:29Z re-run; the cause of the missing END is not recorded in the logs. +3. **The 2026-10-28T11:11:37Z `-.slice` OOM entry is unattributed.** It falls + inside a run that exited 0; no sweep log records a failure for it. +4. **`dmesg` is unreadable (EPERM);** OOM evidence is limited to + `journalctl -b`. +5. **The reference check's PASS verdict was produced after a bug fix,** not by + the original crash-prone invocation. The tolerance itself predates the + results (`reference_check.py.bak`, 2026-10-26T02:06:56Z; first reference run + 02:10:14Z). +6. **Single seed.** The sweep is seed 0 only. No between-seed claim is made. +7. **The 5G-NIDD beta0.25 live cap raise is not reproducible from logs alone** + (the run.log still shows the 11G launch command); this is noted in the + pre-existing launch report. +8. **Report is not committed.** `kisikisithesis/SWEEP_SUMMARY_REPORT.md` is + untracked and is not part of commit `9fe584d`. +9. `results/` was not modified by this session; the analysis scripts were run + only against the scratch copy. The pre-existing on-disk analysis outputs + matched the fresh scratch outputs.