willtanoe/thesis-prototype-agregation
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add factual sweep summary report

Browse Source 
Server-side, read-only summary of the completed 30/30 seed-0 sweep:
queue outcome, per-run wall time and peak RSS, OOM evidence, the
reference check against the pre-fixed 2-sigma tolerance, deviations,
four analysis-script outputs, and the reproducibility runs.  Does not
claim significance, recommend a beta, or commit data.
This commit is contained in:
willtanoe committed 2026-10-30 04:35:59 +01:00
1 parent 9fe584d08e
commit ffbef12812
1 file changed
+685
Download Patch File Download Diff File Expand all files Collapse all files
kisikisithesis/SWEEP_SUMMARY_REPORT.md
+685
View File
@@ -0,0 +1,685 @@
# Sweep summary report (server-side, read-only)
Generated: 2026-10-29 (UTC) by an agent session on the server that holds the
sweep outputs. Scope: the state of `results/sweep/` and `results/sweep_repro/`
as found on disk, plus the four analysis scripts now version-controlled under
`scripts/sweep/`.
This report is descriptive. It does **not** claim statistical significance, does
**not** select or recommend a beta, and does **not** compare the sweep with the
paper. Where the analysis scripts emit a "selected" or "best" column, that
column is reproduced verbatim as script output, not adopted here.
The raw data (`results/`) is git-ignored and is **not** committed. This report is
also not committed.
---
## 1. Repository state at the start of the session
Commands and output:
```
$ git status --short
?? kisikisithesis/RESTORE_VERIFICATION_REPORT.md
$ git branch --show-current
chore/posthoc-audit-and-docs-refresh
$ git log --oneline -5
e66ecc4 docs: add 5G-NIDD repeat and multi-thread test (section 3.7)
b40c928 docs: add repeat and multi-thread test (section 3.6) for the 1-thread sweep
ad203eb docs: record 1-thread seed-0 sweep, noise checks and grid comparison
91b0e37 chore: declare matplotlib and apply three behavior-preserving nits
7831272 docs: refresh README and session handoff for current state
```
- `scripts/sweep/` did not exist.
- `git log --oneline --all -- scripts/sweep` returned nothing, so no commit
adding it existed.
- `git rev-parse main` and `git rev-parse origin/main` were both
`34bab458651b80c617490a75b94b27fdcbbc19fb`.
- The only pre-existing untracked file was
`kisikisithesis/RESTORE_VERIFICATION_REPORT.md` (not created by this session,
not touched, not committed).
`results/sweep/` and `results/sweep_repro/` exist on this server, matching the
task statement that they are git-ignored (`.gitignore` line `results/`).
---
## 2. Queue outcome
Source: `results/sweep/queue.txt`, `results/sweep/summary.csv`,
`results/sweep/runs/*/COMPLETE`, `results/sweep/logs/pool.log`.
- Queue rows: **30** (2 datasets x 3 alphas x 5 betas, seed 0).
- Final `summary.csv` status counts: **30 completed, 0 failed** (exit code 0 in
all 30 rows).
- `COMPLETE` markers on disk: **30**.
- `results/sweep/PHASE`: `done`.
- Last `HEARTBEAT` (2026-10-29T00:02:03Z): `rows: completed=30 failed=0
pending=0 total=30`, `training_processes: 0`.
- The sweep is finished; no row is marked unfinished.
Failed attempts that were later replaced in `summary.csv` (the upsert keeps one
row per config) are listed in section 4.
---
## 3. Completed runs: wall time and peak memory
Values are from `results/sweep/summary.csv`, which records `/usr/bin/time -v`
wall seconds and maximum resident set size for the completed attempt. Peak RSS
is converted from KiB to GiB (divide by 1,048,576). Rows are ordered by start
time. `wall` is `h:mm:ss`.
| # | run | started (UTC) | ended (UTC) | wall | peak RSS (GiB) | status |
|---:|---|---|---|---:|---:|---|
| 1 | `5gnidd__alpha0.25__seed0__support_weighted__beta0` | 2026-10-26T07:31:29Z | 2026-10-26T12:55:25Z | 5:23:56 | 13.12 | completed |
| 2 | `xiiotid__alpha0.25__seed0__support_weighted__beta0` | 2026-10-26T07:31:29Z | 2026-10-26T11:02:12Z | 3:30:42 | 8.38 | completed |
| 3 | `xiiotid__alpha0.25__seed0__support_weighted__beta1` | 2026-10-26T07:31:29Z | 2026-10-26T11:06:03Z | 3:34:33 | 8.38 | completed |
| 4 | `5gnidd__alpha0.25__seed0__support_weighted__beta1` | 2026-10-26T11:02:12Z | 2026-10-26T16:21:43Z | 5:19:30 | 13.11 | completed |
| 5 | `xiiotid__alpha0.25__seed0__support_weighted__beta0.25` | 2026-10-27T01:13:26Z | 2026-10-27T04:25:41Z | 3:12:15 | 8.38 | completed |
| 6 | `5gnidd__alpha0.25__seed0__support_weighted__beta0.25` | 2026-10-27T01:13:26Z | 2026-10-27T06:22:48Z | 5:09:21 | 13.12 | completed |
| 7 | `xiiotid__alpha0.25__seed0__support_weighted__beta0.5` | 2026-10-27T01:13:26Z | 2026-10-27T04:25:12Z | 3:11:45 | 8.38 | completed |
| 8 | `5gnidd__alpha0.25__seed0__support_weighted__beta0.5` | 2026-10-27T04:25:12Z | 2026-10-27T09:46:00Z | 5:20:47 | 13.11 | completed |
| 9 | `xiiotid__alpha0.25__seed0__support_weighted__beta0.75` | 2026-10-27T04:25:41Z | 2026-10-27T07:55:55Z | 3:30:12 | 8.38 | completed |
| 10 | `5gnidd__alpha0.25__seed0__support_weighted__beta0.75` | 2026-10-27T06:22:48Z | 2026-10-27T11:46:25Z | 5:23:36 | 13.11 | completed |
| 11 | `xiiotid__alpha0.5__seed0__support_weighted__beta0` | 2026-10-27T07:55:55Z | 2026-10-27T11:20:41Z | 3:24:46 | 6.21 | completed |
| 12 | `5gnidd__alpha0.5__seed0__support_weighted__beta0` | 2026-10-27T09:46:01Z | 2026-10-27T15:20:57Z | 5:34:55 | 10.28 | completed |
| 13 | `xiiotid__alpha0.5__seed0__support_weighted__beta0.25` | 2026-10-27T11:20:42Z | 2026-10-27T14:55:30Z | 3:34:47 | 6.21 | completed |
| 14 | `5gnidd__alpha0.5__seed0__support_weighted__beta0.25` | 2026-10-27T11:46:25Z | 2026-10-27T17:36:49Z | 5:50:23 | 10.29 | completed |
| 15 | `xiiotid__alpha0.5__seed0__support_weighted__beta0.5` | 2026-10-27T14:55:31Z | 2026-10-27T18:44:43Z | 3:49:12 | 6.21 | completed |
| 16 | `5gnidd__alpha0.5__seed0__support_weighted__beta0.5` | 2026-10-27T15:20:58Z | 2026-10-27T21:26:19Z | 6:05:21 | 10.29 | completed |
| 17 | `xiiotid__alpha0.5__seed0__support_weighted__beta0.75` | 2026-10-27T17:36:49Z | 2026-10-27T21:35:28Z | 3:58:39 | 6.21 | completed |
| 18 | `5gnidd__alpha0.5__seed0__support_weighted__beta0.75` | 2026-10-27T18:44:44Z | 2026-10-28T00:42:07Z | 5:57:23 | 10.29 | completed |
| 19 | `xiiotid__alpha0.5__seed0__support_weighted__beta1` | 2026-10-27T21:26:20Z | 2026-10-28T01:05:49Z | 3:39:29 | 6.21 | completed |
| 20 | `5gnidd__alpha0.5__seed0__support_weighted__beta1` | 2026-10-27T21:35:29Z | 2026-10-28T03:18:55Z | 5:43:26 | 10.31 | completed |
| 21 | `xiiotid__alpha0.75__seed0__support_weighted__beta0` | 2026-10-28T00:42:07Z | 2026-10-28T04:23:48Z | 3:41:40 | 6.48 | completed |
| 22 | `5gnidd__alpha0.75__seed0__support_weighted__beta0` | 2026-10-28T01:05:49Z | 2026-10-28T06:53:35Z | 5:47:44 | 8.61 | completed |
| 23 | `xiiotid__alpha0.75__seed0__support_weighted__beta0.25` | 2026-10-28T03:18:56Z | 2026-10-28T07:05:59Z | 3:47:03 | 6.48 | completed |
| 24 | `5gnidd__alpha0.75__seed0__support_weighted__beta0.25` | 2026-10-28T04:23:48Z | 2026-10-28T10:17:38Z | 5:53:48 | 8.61 | completed |
| 25 | `xiiotid__alpha0.75__seed0__support_weighted__beta0.5` | 2026-10-28T06:53:35Z | 2026-10-28T10:40:37Z | 3:47:01 | 6.48 | completed |
| 26 | `5gnidd__alpha0.75__seed0__support_weighted__beta0.5` | 2026-10-28T07:06:00Z | 2026-10-28T12:40:40Z | 5:34:40 | 8.61 | completed |
| 27 | `xiiotid__alpha0.75__seed0__support_weighted__beta0.75` | 2026-10-28T10:17:38Z | 2026-10-28T13:32:01Z | 3:14:23 | 6.48 | completed |
| 28 | `5gnidd__alpha0.75__seed0__support_weighted__beta0.75` | 2026-10-28T10:40:37Z | 2026-10-28T15:26:24Z | 4:45:46 | 8.61 | completed |
| 29 | `xiiotid__alpha0.75__seed0__support_weighted__beta1` | 2026-10-28T12:40:40Z | 2026-10-28T15:37:02Z | 2:56:21 | 6.48 | completed |
| 30 | `5gnidd__alpha0.75__seed0__support_weighted__beta1` | 2026-10-28T13:32:01Z | 2026-10-28T18:00:12Z | 4:28:10 | 8.61 | completed |
Totals over the 30 completed attempts:
- Sum of wall time: **486,694 s = 135.19 h** (not pool wall time; runs overlap).
- X-IIoTID: n=15, wall 2:56:21-3:58:39, mean 3:31:31, max peak RSS 8.38 GiB.
- 5G-NIDD: n=15, wall 4:28:10-6:05:21, mean 5:29:15, max peak RSS 13.12 GiB.
- Global maximum peak RSS: **13.12 GiB** (`5gnidd alpha0.25 beta0` and
`5gnidd alpha0.25 beta0.25`).
- First pool start: 2026-10-26T02:10:14Z. Last completion:
2026-10-28T18:00:12Z. Span: **2 days 15:49:58**.
- From the final relaunch (2026-10-27T01:13:26Z) to the last completion:
**1 day 16:46:46**.
### 3.1 Reproducibility runs in `results/sweep_repro/` (not queue rows)
These four runs are outside `results/sweep/summary.csv` and were launched with
`results/sweep_repro/run_variant.sh` (1-thread = `thread1_launcher.py`, as in the
sweep; 8-thread = plain `run_baseline.py`). They write `logs/start.epoch`,
`logs/end.epoch`, `logs/exit_code` and `/usr/bin/time -v` output instead of a
`COMPLETE` marker. All four exited 0 with empty `stderr.log`.
| config | variant dir | start (UTC) | end (UTC) | wall | peak RSS (GiB) | exit |
|---|---|---:|---:|---:|---:|---:|
| xiiotid alpha0.25 seed0 beta0 | `a_1thread_repeat` | 2026-10-29T00:40:42Z | 2026-10-29T03:22:26Z | 2:41:44 | 8.38 | 0 |
| xiiotid alpha0.25 seed0 beta0 | `b_8thread` | 2026-10-29T00:40:42Z | 2026-10-29T01:56:28Z | 1:15:46 | 8.39 | 0 |
| 5gnidd alpha0.5 seed0 beta0.75 | `5gnidd_a0.5_b0.75/a_1thread_repeat` | 2026-10-29T11:57:21Z | 2026-10-29T16:36:33Z | 4:39:11 | 10.29 | 0 |
| 5gnidd alpha0.5 seed0 beta0.75 | `5gnidd_a0.5_b0.75/b_8thread` | 2026-10-29T11:57:21Z | 2026-10-29T14:39:23Z | 2:42:01 | 10.43 | 0 |
Peak RSS converted from `/usr/bin/time -v` KiB (8,786,400 / 8,794,116 /
10,787,408 / 10,939,972). The two 1-thread repeats reproduce the corresponding
sweep runs byte-for-byte in the compared artifacts: for 5G-NIDD,
`test_global_metrics.json`, `val_selection.json`, `history.json`,
`test_per_class.csv` and `per_client.csv` are identical to
`results/sweep/runs/5gnidd__alpha0.5__seed0__support_weighted__beta0.75/...`
(`sha256sum` equal for all five); for X-IIoTID the same is recorded by
`compare_repro.json` (section 8.6). The 8-thread variants differ from the
1-thread repeats (5G-NIDD rare macro-F1 0.875941789 vs 0.879314583; X-IIoTID
0.643834668 vs 0.641979301).
---
## 4. All attempts, including failed and retried rows
Reconstructed from `results/sweep/logs/pool.log`,
`results/sweep/logs/continuation.log`, per-run `logs/run.log`, and the orphan
manifest. Filesystem timestamps quoted in UTC (`TZ=UTC stat`).
| run | attempt | started (UTC) | ended / outcome | evidence |
|---|---:|---|---|---|
| `5gnidd alpha0.25 beta1` | 1 | 2026-10-26T02:10:14Z | 2026-10-26T02:22:50Z, exit 143 (OOM) | `runs/...beta1/logs/run.log:3`, `pool.log:11`; journal OOM-kill |
| `5gnidd alpha0.25 beta0` | 1 | 2026-10-26T02:10:14Z | 2026-10-26T02:22:53Z, exit 143 (OOM) | `runs/...beta0/logs/run.log:3`, `pool.log:16`; journal OOM-kill |
| `xiiotid alpha0.25 beta0` | 1 | 2026-10-26T02:10:14Z | no END line, no COMPLETE; second attempt started 07:31:29Z | `logs/run.log` shows two START lines and no END for the first |
| `xiiotid alpha0.25 beta1` | 1 | 2026-10-26T02:10:14Z | no END line, no COMPLETE; second attempt started 07:31:29Z | same |
| `xiiotid alpha0.25 beta0.25` | 1 | 2026-10-26T02:22:51Z | orphan; result files written 2026-10-26T05:41:53Z, no `time.txt`, no COMPLETE | `_orphans_20261027T011213Z/` |
| `5gnidd alpha0.25 beta0.25` | 1 | 2026-10-26T02:22:53Z | orphan; result files written 2026-10-26T07:29:08Z, no `time.txt`, no COMPLETE | `_orphans_20261027T011213Z/` |
| `xiiotid alpha0.25 beta0.25` | 2 (gate) | gate started 2026-10-26T11:06:04Z | 2026-10-26T23:06:09Z, exit 4, "reference gate did not resolve within 12 h" | `pool.log:220`, inspection report |
| `5gnidd alpha0.25 beta0.25` | 2 (gate) | gate started 2026-10-26T12:55:25Z | 2026-10-27T00:55:31Z, exit 4, same reason | `pool.log:50`, `pool.log:53`, `pool.log:253`; inspection report |
| `xiiotid alpha0.5 beta0` | 1 (gate only) | gate from 2026-10-26T16:21:43Z | wrapper SIGTERMed 2026-10-27T01:06-01:07Z before training | `pool.log` gate lines, relaunch report |
| `5gnidd alpha0.5 beta0` | 1 (gate only) | gate from 2026-10-26T23:06:09Z | wrapper SIGTERMed 2026-10-27T01:06-01:07Z before training | same |
| `xiiotid alpha0.75 beta0` | 1 (gate only) | gate from 2026-10-27T00:55:31Z | wrapper SIGTERMed 2026-10-27T01:06-01:07Z before training | same |
All six configs above eventually appear as `completed` in `summary.csv` with the
start times in section 3. The automatic retry pass at
2026-10-28T18:00:12Z reported:
```
[2026-10-28T18:00:12Z] RETRY: no failed rows; nothing to do
[2026-10-28T18:00:12Z] SUPERVISOR retry pass finished rc=0
[2026-10-28T18:00:12Z] SUPERVISOR done (main=0 retry=0)
```
Orphan artifacts were moved (not deleted) to
`results/sweep/_orphans_20261027T011213Z/` at 2026-10-27T01:12:13Z. The orphan
manifest records, for the record only, test rare macro-F1 0.645839538
(X-IIoTID beta0.25) and 0.887639250 (5G-NIDD beta0.25); attempt-1 wall time was
never measured (`time.txt` 0 bytes). These are **not** the accepted results.
The manifest prints filesystem times in host local time (Europe/Berlin, UTC+1
on 2026-10-26); the UTC values quoted above come from `TZ=UTC stat` on the same
files and refer to the same inodes (identical nanosecond fractions).
---
## 5. Out-of-memory events
`journalctl -k` (kernel ring):
```
$ journalctl -k --no-pager | grep -iE 'oom|out of memory|killed process'
(no matching lines)
```
`journalctl -b --utc -o short-iso` (systemd journal) contains:
```
2026-10-26T02:01:37+00:00 aster systemd[2118]: run-r0a3b67e68dfa4510b7690d3eb1c285ef.scope: A process of this unit has been killed by the OOM killer.
2026-10-26T02:01:37+00:00 aster systemd[2118]: run-r0a3b67e68dfa4510b7690d3eb1c285ef.scope: Failed with result 'oom-kill'.
2026-10-26T02:01:37+00:00 aster systemd[2118]: app.slice: A process of this unit has been killed by the OOM killer.
2026-10-26T02:22:50+00:00 aster systemd[2118]: run-r351421db66ec4b9dabb5153489c169a3.scope: A process of this unit has been killed by the OOM killer.
2026-10-26T02:22:51+00:00 aster systemd[2118]: run-r351421db66ec4b9dabb5153489c169a3.scope: Failed with result 'oom-kill'.
2026-10-26T02:22:53+00:00 aster systemd[2118]: run-r349991742c5d4469859a140260c1cc20.scope: A process of this unit has been killed by the OOM killer.
2026-10-26T02:22:54+00:00 aster systemd[2118]: run-r349991742c5d4469859a140260c1cc20.scope: Failed with result 'oom-kill'.
2026-10-26T07:29:09+00:00 aster systemd[2118]: app.slice: A process of this unit has been killed by the OOM killer.
2026-10-28T11:11:37+00:00 aster systemd[2118]: -.slice: A process of this unit has been killed by the OOM killer.
```
Interpretation limited to what the logs show:
- The 02:22:50 / 02:22:53 entries are the two 5G-NIDD first-wave run scopes;
both rows recorded exit 143.
- The 02:01:37 `run-r0a3b67e...` entry is before the queue started (pool start
02:10:14). `launch_report.md` says the third kill in
`/sys/fs/cgroup/user.slice/memory.events` is "most likely the earlier
deliberate 512 MiB under a 200 MB cap test", i.e. not a queue row. That
attribution is the launch report's inference; it is not independently
confirmed here.
- The 07:29:09 `app.slice` entry coincides with the end of the 5G-NIDD beta0.25
orphan attempt; the log does not name a run.
- The 2026-10-28T11:11:37 `-.slice` entry falls inside the completed
`5gnidd alpha0.75 beta0.75` run window, which exited 0. No sweep `run.log`
records a failure for it; its cause is not identified by these logs.
- `dmesg` is not readable by this user:
`dmesg: read kernel buffer failed: Operation not permitted`.
Sweep logs themselves contain no literal "OOM" string. The only failure markers
are the two `END exit_code=143` lines in `pool.log`, each mirrored in the
corresponding `runs/<run>/logs/run.log` (four lines in total).
---
## 6. Reference check against the pre-fixed tolerance
`results/sweep/reference_check.md` reports **Status: PASS**, generated
`2026-10-27T01:10:36Z`. `results/sweep/reference_check.json` has
`complete: true`, `stop: false`, `stop_reasons: []`, `missing_betas: []`.
Tolerance: the check compares each fresh value with the audit seed-0 value using
a band of **+/- 2 x between-seed standard deviation** (`statistics.stdev`,
ddof=1, over audit seeds 0,1,2). The comparison code is
```
within = abs(delta) <= 2 * std
```
and is identical in `reference_check.py.bak` (mtime
`2026-10-26T02:06:56Z`) and the current `reference_check.py`. The first reference
run started at `2026-10-26T02:10:14Z`, i.e. the tolerance predates the reference
results. `launch_report.md` (mtime `2026-10-26T02:31:53Z`) also documents the
2-sigma rule and was written before the reference runs finished (11:02Z / 11:06Z
on 2026-10-26). The passing verdict itself was produced later, after a key-name
fix at 2026-10-27T01:10:30Z.
Values (from `reference_check.json`):
| beta | metric | fresh | audit seed-0 | delta | z | 2-sigma band | within 2 sigma |
|---:|---|---:|---:|---:|---:|---|---|
| 0 | test rare macro-F1 | 0.641979301 | 0.641362228 | +0.000617073 | +0.013 | [0.546078671, 0.736645785] | true |
| 0 | val rare macro-F1 (per-client mean) | 0.615114398 | 0.616380161 | -0.001265762 | -0.014 | [0.437322488, 0.795437834] | true |
| 1 | test rare macro-F1 | 0.593073182 | 0.584131463 | +0.008941719 | +0.117 | [0.431612637, 0.736650288] | true |
| 1 | val rare macro-F1 (per-client mean) | 0.579765280 | 0.578384529 | +0.001380751 | +0.030 | [0.486947852, 0.669821206] | true |
Both runs had `config_ok: true` and the expected loader content hash
`839ba18e2d87245d198be8f5f89d4a24f13cdb6fc64c004d406c6d4f8246f3da`.
Earlier, `reference_check.py` crashed three times with
`KeyError: 'val_rare_macro_f1_per_client_mean'`
(`logs/reference_check.log`); this caused the 12-hour gate timeouts recorded in
section 4. The crash and the fix are documented in
`inspection_2026-10-27.md` and `relaunch_report.md`. These reports are inputs;
their conclusions were not re-derived here.
---
## 7. Deviations from the original plan, with timestamps
1. **Per-process memory cap changed for 5G-NIDD (11 GiB -> 15 GiB).**
The first wave used `MemoryMax=11G` for every run (`pool.log:6-9`). Two
5G-NIDD runs were OOM-killed at 2026-10-26T02:22:50Z and 02:22:53Z.
The in-flight 5G-NIDD beta0.25 scope was raised live to 15 GiB around
02:23Z (per launch report), and `run_one.sh` was changed to select
`5gnidd -> 15G`, others `-> 11G`; the pre-change backup
`run_one.sh.bak` has mtime **2026-10-26T02:28:30Z**. The relaunched pool
command lines show `MemoryMax=15G` for 5G-NIDD (`pool.log:26`, `33`, and
later).
2. **Pool size changed (P=4 -> P=3).**
First pool: `P=4`, started **2026-10-26T02:10:14Z** (`pool.log:1`).
Steady-state pool: `P=3`, started **2026-10-26T07:31:29Z** (`pool.log:20`).
Final relaunch: `P=3`, pool start **2026-10-27T01:13:25Z**
(`pool.log:263`; first run command at 01:13:26Z, `pool.log:274`).
`pool.size` mtime 2026-10-27T01:13:25Z. P=5 was never used.
3. **2-slot 5G-NIDD semaphore added to `run_one.sh`.**
`run_one.sh` mtime **2026-10-27T01:10:59Z**; `pool.log` shows
`SEMAPHORE acquired slot1/slot2` from the 01:13:26Z relaunch onward.
4. **Reference-check bug fixed and gate guard added.**
`reference_check.py` mtime **2026-10-27T01:10:30Z** (key names corrected).
`run_one.sh` mtime **2026-10-27T01:10:59Z** (30-minute gate guard plus
semaphore). `status.sh` mtime **2026-10-27T01:51:16Z** (cosmetic).
5. **Retried / re-run rows.** `5gnidd alpha0.25 beta0` and `beta1` (OOM, first
attempt 02:10:14Z, retried 07:31:29Z / 11:02:12Z); `xiiotid alpha0.25 beta0`
and `beta1` (first attempt 02:10:14Z has no END, re-run 07:31:29Z);
`xiiotid alpha0.25 beta0.25` and `5gnidd alpha0.25 beta0.25` (orphan attempt
at 02:22:5xZ, gate attempt failed at 23:06:09Z / 00:55:31Z, final attempt
01:13:26Z). See section 4 for evidence.
6. **Gate-waiting processes were terminated before training.**
At 2026-10-27T01:06-01:07Z the pool was stopped; three rows were waiting in
the gate loop (`xiiotid 0.5`, `5gnidd 0.5`, `xiiotid 0.75`). They started
later under the final relaunch. `pool.log:259-260` records the SIGTERM.
7. **Orphan attempts moved aside.** `_orphans_20261027T011213Z/` created
2026-10-27T01:12:13Z; no result file was deleted.
8. **No queue row was skipped.** The 2026-10-21 5-thread smoke outputs were not
accepted as sweep rows; all 30 configs were run under the 1-thread protocol.
9. **Host timezone.** `timedatectl` reports `Europe/Berlin (CET, +0100)`; all
timestamps in this report are UTC, converted with `journalctl --utc` or
`TZ=UTC stat`. Some pre-existing sweep reports mix local and UTC times.
---
## 8. Outputs of the four analysis scripts
The four scripts were copied to `scripts/sweep/` and run against a scratch copy
of the data (so the real `results/` was not written). Their outputs were
byte-identical to the originals (section 9). The tables below are those outputs.
`analyze_sweep.py` writes three files. Stage 1 uses validation only; the test
files are opened after `selection.csv` is written.
### 8.1 `selection.csv` (validation only, 30 rows)
The `selected` column is the script's stage-1 flag (highest
`val_rare_class_macro_f1_per_client_mean`, ties to the smaller beta). It is
reproduced as script output; this report does not adopt it.
| dataset | alpha | beta | val_rare_f1_per_client_mean | val_rare_f1_global | val_rare_recall_per_client_mean | val_macro_f1_global | selected | margin_to_runner_up |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| xiiotid | 0.25 | 0 | 0.615114398 | 0.643275275 | 0.846913348 | 0.676267875 | 0 | |
| xiiotid | 0.25 | 0.25 | 0.612295414 | 0.647422433 | 0.847371095 | 0.680035834 | 0 | |
| xiiotid | 0.25 | 0.5 | 0.590996277 | 0.632833153 | 0.831341674 | 0.667045341 | 0 | |
| xiiotid | 0.25 | 0.75 | 0.656437133 | 0.669066681 | 0.870359212 | 0.699686253 | 1 | 0.041322734 |
| xiiotid | 0.25 | 1 | 0.579765280 | 0.593760832 | 0.784814026 | 0.633116555 | 0 | |
| xiiotid | 0.5 | 0 | 0.556929048 | 0.612060535 | 0.885829017 | 0.709691170 | 0 | |
| xiiotid | 0.5 | 0.25 | 0.562538987 | 0.620452943 | 0.889189065 | 0.716471910 | 0 | |
| xiiotid | 0.5 | 0.5 | 0.588286653 | 0.644186102 | 0.896470166 | 0.732532602 | 1 | 0.007362956 |
| xiiotid | 0.5 | 0.75 | 0.580923697 | 0.636817927 | 0.889355473 | 0.726043174 | 0 | |
| xiiotid | 0.5 | 1 | 0.558192955 | 0.614928577 | 0.857825220 | 0.714886486 | 0 | |
| xiiotid | 0.75 | 0 | 0.497123071 | 0.532261516 | 0.766962632 | 0.748317495 | 0 | |
| xiiotid | 0.75 | 0.25 | 0.499599429 | 0.526397734 | 0.756669120 | 0.748892418 | 0 | |
| xiiotid | 0.75 | 0.5 | 0.499968468 | 0.513382502 | 0.765223179 | 0.741108809 | 1 | 0.000369040 |
| xiiotid | 0.75 | 0.75 | 0.490886014 | 0.510831286 | 0.761348983 | 0.741545715 | 0 | |
| xiiotid | 0.75 | 1 | 0.477724598 | 0.507546185 | 0.762276080 | 0.741271747 | 0 | |
| 5gnidd | 0.25 | 0 | 0.732063562 | 0.863288898 | 0.888413711 | 0.833210589 | 0 | |
| 5gnidd | 0.25 | 0.25 | 0.750234422 | 0.887687791 | 0.886155769 | 0.849340729 | 1 | 0.015462491 |
| 5gnidd | 0.25 | 0.5 | 0.715027757 | 0.818058015 | 0.885062004 | 0.804059192 | 0 | |
| 5gnidd | 0.25 | 0.75 | 0.711752341 | 0.795364214 | 0.877813860 | 0.802485891 | 0 | |
| 5gnidd | 0.25 | 1 | 0.734771931 | 0.801655038 | 0.874817852 | 0.818441666 | 0 | |
| 5gnidd | 0.5 | 0 | 0.739471675 | 0.867651737 | 0.964727218 | 0.835451455 | 0 | |
| 5gnidd | 0.5 | 0.25 | 0.657948994 | 0.826207029 | 0.964727218 | 0.799036116 | 0 | |
| 5gnidd | 0.5 | 0.5 | 0.731662212 | 0.862708327 | 0.964727218 | 0.836388230 | 0 | |
| 5gnidd | 0.5 | 0.75 | 0.771920820 | 0.875152730 | 0.964727218 | 0.849873996 | 1 | 0.005265685 |
| 5gnidd | 0.5 | 1 | 0.766655135 | 0.876266492 | 0.964727218 | 0.846137526 | 0 | |
| 5gnidd | 0.75 | 0 | 0.836358274 | 0.896193888 | 0.964226565 | 0.851335066 | 0 | |
| 5gnidd | 0.75 | 0.25 | 0.849565876 | 0.902762083 | 0.964226565 | 0.856336364 | 1 | 0.013207603 |
| 5gnidd | 0.75 | 0.5 | 0.830443462 | 0.901146966 | 0.963185589 | 0.855355109 | 0 | |
| 5gnidd | 0.75 | 0.75 | 0.815299394 | 0.897742377 | 0.963715390 | 0.853576206 | 0 | |
| 5gnidd | 0.75 | 1 | 0.808079811 | 0.894306659 | 0.965304794 | 0.851265425 | 0 | |
### 8.2 `test_at_selected_beta.csv` (test, 30 rows)
`selected_on_validation` marks the beta flagged in 8.1. `test_rare_support` is
the number of test samples in the rare classes.
| dataset | alpha | beta | selected_on_validation | test_rare_f1_global | test_rare_recall_global | test_macro_f1 | test_accuracy | test_rare_support |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| xiiotid | 0.25 | 0 | 0 | 0.641979301 | 0.830387426 | 0.675034969 | 0.873838520 | 79180 |
| xiiotid | 0.25 | 0.25 | 0 | 0.645839538 | 0.824822512 | 0.678550791 | 0.868704362 | 79180 |
| xiiotid | 0.25 | 0.5 | 0 | 0.631488471 | 0.793292581 | 0.665776684 | 0.854517874 | 79180 |
| xiiotid | 0.25 | 0.75 | 1 | 0.669170594 | 0.794023733 | 0.699726946 | 0.855561902 | 79180 |
| xiiotid | 0.25 | 1 | 0 | 0.593073182 | 0.704711149 | 0.632415448 | 0.849138063 | 79180 |
| xiiotid | 0.5 | 0 | 0 | 0.613084485 | 0.871186482 | 0.710555367 | 0.896929946 | 25454 |
| xiiotid | 0.5 | 0.25 | 0 | 0.622176085 | 0.873858741 | 0.717774657 | 0.897568645 | 25454 |
| xiiotid | 0.5 | 0.5 | 1 | 0.643904823 | 0.874747143 | 0.732555353 | 0.898287181 | 25454 |
| xiiotid | 0.5 | 0.75 | 0 | 0.636012409 | 0.863519815 | 0.725833853 | 0.886133476 | 25454 |
| xiiotid | 0.5 | 1 | 0 | 0.615308344 | 0.838787843 | 0.715340930 | 0.881257254 | 25454 |
| xiiotid | 0.75 | 0 | 0 | 0.526279577 | 0.819166043 | 0.745053791 | 0.912854432 | 6251 |
| xiiotid | 0.75 | 0.25 | 0 | 0.520979721 | 0.806403017 | 0.745946340 | 0.916207602 | 6251 |
| xiiotid | 0.75 | 0.5 | 1 | 0.506784362 | 0.796163189 | 0.737595025 | 0.911871818 | 6251 |
| xiiotid | 0.75 | 0.75 | 0 | 0.507162154 | 0.807061731 | 0.739196779 | 0.913057096 | 6251 |
| xiiotid | 0.75 | 1 | 0 | 0.504951940 | 0.817143701 | 0.739630728 | 0.911939373 | 6251 |
| 5gnidd | 0.25 | 0 | 0 | 0.861244967 | 0.984948540 | 0.832176777 | 0.744659240 | 141677 |
| 5gnidd | 0.25 | 0.25 | 1 | 0.887639250 | 0.980086383 | 0.850377106 | 0.740711498 | 141677 |
| 5gnidd | 0.25 | 0.5 | 0 | 0.818078455 | 0.977174954 | 0.804997750 | 0.740863651 | 141677 |
| 5gnidd | 0.25 | 0.75 | 0 | 0.795982829 | 0.949513575 | 0.804428539 | 0.730101942 | 141677 |
| 5gnidd | 0.25 | 1 | 0 | 0.802243321 | 0.935212949 | 0.819876826 | 0.752912488 | 141677 |
| 5gnidd | 0.5 | 0 | 0 | 0.869410144 | 0.974261201 | 0.836168314 | 0.735143537 | 104843 |
| 5gnidd | 0.5 | 0.25 | 0 | 0.827176768 | 0.974219638 | 0.799716621 | 0.728399479 | 104843 |
| 5gnidd | 0.5 | 0.5 | 0 | 0.868956930 | 0.974261221 | 0.840088463 | 0.739103616 | 104843 |
| 5gnidd | 0.5 | 0.75 | 1 | 0.879314583 | 0.974272043 | 0.853591585 | 0.745403554 | 104843 |
| 5gnidd | 0.5 | 1 | 0 | 0.878757266 | 0.974272032 | 0.848514697 | 0.741887596 | 104843 |
| 5gnidd | 0.75 | 0 | 0 | 0.902386981 | 0.959484554 | 0.855113260 | 0.747373312 | 28000 |
| 5gnidd | 0.75 | 0.25 | 1 | 0.906545156 | 0.960038373 | 0.858596769 | 0.749429428 | 28000 |
| 5gnidd | 0.75 | 0.5 | 0 | 0.904655697 | 0.958552306 | 0.857440422 | 0.749585693 | 28000 |
| 5gnidd | 0.75 | 0.75 | 0 | 0.901701106 | 0.960284499 | 0.855855667 | 0.750461598 | 28000 |
| 5gnidd | 0.75 | 1 | 0 | 0.899195897 | 0.964090767 | 0.854143914 | 0.750432812 | 28000 |
### 8.3 `repro_vs_previous.csv` (30 rows, absolute differences)
Differences are `|new - previous grid|`; the script prints them with three
significant digits. "new" is the 1-thread sweep run; "previous" is the earlier
grid run at the same `(dataset, alpha, seed0, beta)`.
| dataset | alpha | beta | test_rare_f1_new | test_rare_f1_prev | abs_diff_test_rare_f1 | abs_diff_test_macro_f1 | abs_diff_test_accuracy | abs_diff_val_sel_metric |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| xiiotid | 0.25 | 0 | 0.641979301 | 0.641362228 | 6.171e-04 | 4.767e-04 | 7.124e-04 | 1.266e-03 |
| xiiotid | 0.25 | 0.25 | 0.645839538 | 0.645064558 | 7.750e-04 | 7.555e-04 | 2.070e-03 | 2.383e-03 |
| xiiotid | 0.25 | 0.5 | 0.631488471 | 0.631742435 | 2.540e-04 | 1.888e-04 | 1.302e-03 | 4.251e-03 |
| xiiotid | 0.25 | 0.75 | 0.669170594 | 0.666176967 | 2.994e-03 | 2.684e-03 | 9.151e-04 | 2.404e-03 |
| xiiotid | 0.25 | 1 | 0.593073182 | 0.584131463 | 8.942e-03 | 7.927e-03 | 2.045e-03 | 1.381e-03 |
| xiiotid | 0.5 | 0 | 0.613084485 | 0.613575398 | 4.909e-04 | 7.455e-05 | 4.956e-03 | 4.106e-04 |
| xiiotid | 0.5 | 0.25 | 0.622176085 | 0.632154112 | 9.978e-03 | 7.525e-03 | 5.951e-03 | 9.248e-03 |
| xiiotid | 0.5 | 0.5 | 0.643904823 | 0.640829406 | 3.075e-03 | 1.813e-03 | 1.290e-04 | 2.525e-03 |
| xiiotid | 0.5 | 0.75 | 0.636012409 | 0.636188757 | 1.763e-04 | 4.646e-04 | 2.536e-03 | 2.921e-03 |
| xiiotid | 0.5 | 1 | 0.615308344 | 0.615509445 | 2.011e-04 | 1.042e-03 | 3.378e-03 | 6.750e-03 |
| xiiotid | 0.75 | 0 | 0.526279577 | 0.525867706 | 4.119e-04 | 6.807e-04 | 8.905e-04 | 3.200e-04 |
| xiiotid | 0.75 | 0.25 | 0.520979721 | 0.520851745 | 1.280e-04 | 7.959e-04 | 1.069e-03 | 2.708e-03 |
| xiiotid | 0.75 | 0.5 | 0.506784362 | 0.511888897 | 5.105e-03 | 4.382e-03 | 2.493e-03 | 4.419e-03 |
| xiiotid | 0.75 | 0.75 | 0.507162154 | 0.502310077 | 4.852e-03 | 2.765e-03 | 7.799e-04 | 4.018e-03 |
| xiiotid | 0.75 | 1 | 0.504951940 | 0.506063809 | 1.112e-03 | 2.007e-04 | 1.535e-03 | 1.421e-03 |
| 5gnidd | 0.25 | 0 | 0.861244967 | 0.859892744 | 1.352e-03 | 2.497e-03 | 1.851e-04 | 4.297e-03 |
| 5gnidd | 0.25 | 0.25 | 0.887639250 | 0.885537113 | 2.102e-03 | 5.700e-04 | 1.147e-03 | 2.936e-03 |
| 5gnidd | 0.25 | 0.5 | 0.818078455 | 0.824837056 | 6.759e-03 | 3.770e-03 | 3.742e-03 | 2.423e-03 |
| 5gnidd | 0.25 | 0.75 | 0.795982829 | 0.804422938 | 8.440e-03 | 2.071e-03 | 1.871e-02 | 1.510e-03 |
| 5gnidd | 0.25 | 1 | 0.802243321 | 0.805922621 | 3.679e-03 | 2.820e-03 | 1.398e-03 | 3.167e-04 |
| 5gnidd | 0.5 | 0 | 0.869410144 | 0.874293205 | 4.883e-03 | 2.393e-03 | 2.920e-04 | 8.742e-04 |
| 5gnidd | 0.5 | 0.25 | 0.827176768 | 0.827855971 | 6.792e-04 | 5.151e-03 | 3.763e-03 | 3.512e-03 |
| 5gnidd | 0.5 | 0.5 | 0.868956930 | 0.878225000 | 9.268e-03 | 4.119e-03 | 1.583e-03 | 3.283e-02 |
| 5gnidd | 0.5 | 0.75 | 0.879314583 | 0.866626856 | 1.269e-02 | 5.548e-03 | 1.361e-03 | 2.126e-02 |
| 5gnidd | 0.5 | 1 | 0.878757266 | 0.880962035 | 2.205e-03 | 1.955e-03 | 9.499e-04 | 2.222e-03 |
| 5gnidd | 0.75 | 0 | 0.902386981 | 0.900366644 | 2.020e-03 | 1.505e-03 | 4.811e-04 | 1.677e-03 |
| 5gnidd | 0.75 | 0.25 | 0.906545156 | 0.901681021 | 4.864e-03 | 3.485e-03 | 9.129e-04 | 2.618e-02 |
| 5gnidd | 0.75 | 0.5 | 0.904655697 | 0.904447205 | 2.085e-04 | 1.315e-04 | 7.813e-05 | 6.348e-03 |
| 5gnidd | 0.75 | 0.75 | 0.901701106 | 0.904214210 | 2.513e-03 | 1.576e-03 | 1.398e-04 | 9.367e-03 |
| 5gnidd | 0.75 | 1 | 0.899195897 | 0.900797707 | 1.602e-03 | 1.031e-03 | 2.879e-05 | 7.694e-03 |
### 8.4 `check_noise.py` -> `noise_checks.json` (excerpt)
The full file is 380 lines (379 newlines; no trailing newline) and was written
unchanged. Key blocks:
- `A_summary.first_round_with_difference_counts`: `{"0": 30}` — every paired run
first differs from the earlier grid in round 0, and
`n_samples_equal_all_rounds` is `true` for all 30 pairs.
- `B_sign.test_rare_f1`: n=30, sweep higher 14, lower 16, mean signed diff
-0.0004104998, mean absolute diff 0.0034124859. The script labels its
two-sided sign-test p-value `0.8555355519` as
`NOT_independent_pairs`; the 30 pairs share partitions across betas. This
report does not interpret it.
- `B_sign.val_selection_metric`: n=30, sweep higher 15, lower 15, mean signed
diff -0.0004188014, mean absolute diff 0.0056621689; script p-value
`1.0`, again labelled as non-independent.
- `C_rank` (rank stability, sweep vs earlier seed-0 grid over 5 betas):
| dataset | alpha | spearman | max_abs_val_diff | betas_within_max_diff_of_sweep_best |
|---|---:|---:|---:|---|
| xiiotid | 0.25 | 1.0 | 0.004251150 | `0.75` |
| xiiotid | 0.5 | 0.9 | 0.009248424 | `0.5`, `0.75` |
| xiiotid | 0.75 | 1.0 | 0.004419246 | `0`, `0.25`, `0.5` |
| 5gnidd | 0.25 | 0.9 | 0.004296561 | `0.25` |
| 5gnidd | 0.5 | 0.5 | 0.032825056 | `0`, `0.75`, `1` |
| 5gnidd | 0.75 | 0.3 | 0.026177811 | `0`, `0.25`, `0.5` |
### 8.5 `compare_with_grid.py` -> `compare_with_grid.csv` (6 rows)
The `beta_*` columns are script output (validation-only ranking), not a
recommendation.
| dataset | alpha | beta_sweep_seed0 | beta_grid_seed0 | beta_grid_pooled | sweep_equals_grid_s0 | sweep_equals_pooled | grid_s0_equals_pooled | sweep_margin | grid_s0_margin | max_abs_val_diff_s0 | sweep_pick_rank_in_grid_s0 |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| xiiotid | 0.25 | 0.75 | 0.75 | 0.0 | 1 | 0 | 0 | 0.041323 | 0.037653 | 0.004251 | 1 |
| xiiotid | 0.5 | 0.5 | 0.5 | 0.75 | 1 | 0 | 0 | 0.007363 | 0.007759 | 0.009248 | 1 |
| xiiotid | 0.75 | 0.5 | 0.5 | 0.5 | 1 | 1 | 1 | 0.000369 | 0.007496 | 0.004419 | 1 |
| 5gnidd | 0.25 | 0.25 | 0.25 | 0.0 | 1 | 0 | 0 | 0.015462 | 0.010939 | 0.004297 | 1 |
| 5gnidd | 0.5 | 0.75 | 1.0 | 1.0 | 0 | 0 | 1 | 0.005266 | 0.004390 | 0.032825 | 3 |
| 5gnidd | 0.75 | 0.25 | 0.5 | 0.5 | 0 | 0 | 1 | 0.013208 | 0.002110 | 0.026178 | 4 |
### 8.6 `compare_repro.py` -> `compare_repro.json` and stdout
Printed table:
```
a_1thread_repeat__vs__sweep test_rare=0.000e+00 val_sel=0.000e+00 round0=0.000e+00 last=0.000e+00 all_json_equal=True snapshot_identical=True
b_8thread__vs__grid test_rare=2.472e-03 val_sel=8.021e-04 round0=4.840e-05 last=2.242e-03 all_json_equal=False snapshot_identical=False
a_1thread_repeat__vs__grid test_rare=6.171e-04 val_sel=1.266e-03 round0=4.552e-05 last=3.359e-04 all_json_equal=False snapshot_identical=False
b_8thread__vs__sweep test_rare=1.855e-03 val_sel=4.637e-04 round0=9.392e-05 last=1.906e-03 all_json_equal=False snapshot_identical=False
sweep__vs__grid test_rare=6.171e-04 val_sel=1.266e-03 round0=4.552e-05 last=3.359e-04 all_json_equal=False snapshot_identical=False
a_1thread_repeat__vs__b_8thread test_rare=1.855e-03 val_sel=4.637e-04 round0=9.392e-05 last=1.906e-03 all_json_equal=False snapshot_identical=False
```
Facts recorded in the JSON:
- `a_1thread_repeat` vs `sweep`: all four JSON files byte-identical and
JSON-equal; prototype snapshot byte-identical; test rare F1 and validation
metric differences 0.0; all ten per-round rare-F1 differences 0.0.
- `b_8thread` vs `grid`: JSON files not byte-identical; test rare F1
b_8thread 0.643834668 vs grid 0.641362228, abs diff 0.002472440; validation
metric abs diff 0.000802084; prototype snapshot differs.
- `a_1thread_repeat` vs `grid`: test rare F1 0.641979301 vs 0.641362228,
abs diff 0.000617073; validation metric abs diff 0.001265762.
- The remaining pairs are context and show the same pattern (different
runs differ; the repeated 1-thread run is identical to the sweep run).
This script analyses only the X-IIoTID `alpha0.25 seed0 beta0` pair. A second
reproducibility pair exists under
`results/sweep_repro/5gnidd_a0.5_b0.75/` (5G-NIDD `alpha0.5 seed0 beta0.75`,
1-thread and 8-thread). `compare_repro.py` does not analyse it; its wall time,
peak RSS and the byte-identical 1-thread comparison are reported in section 3.1.
---
## 9. Scripts moved into version control
Commit on branch `chore/posthoc-audit-and-docs-refresh`:
```
9fe584d08e0471ed517cb6f781e522f99d002520
chore: version-control sweep analysis scripts
```
Pushed without force (`e66ecc4..9fe584d`); after the push `HEAD` and
`origin/chore/posthoc-audit-and-docs-refresh` both pointed at
`9fe584d08e0471ed517cb6f781e522f99d002520`. `main` and `origin/main` both
remained at `34bab458651b80c617490a75b94b27fdcbbc19fb`, and `9fe584d` is not an
ancestor of `main`.
Files added (5 files, 697 insertions):
```
scripts/sweep/README.md
scripts/sweep/analyze_sweep.py
scripts/sweep/check_noise.py
scripts/sweep/compare_repro.py
scripts/sweep/compare_with_grid.py
```
The originals remain in `results/sweep/` and `results/sweep_repro/`. The copies
change only path resolution: a `--repo-root` argument with a default of
`<repo>/` derived from `scripts/sweep/` (the originals derived the root from
their old location inside `results/`). No computation was changed.
Equivalence was tested on a scratch copy of the data
(`~/restore_work/sweep_scripts_check`), not on the real `results/`. Original
scripts were run from a mirrored `results/sweep/` layout; copies were run with
`--repo-root`. Comparison:
```
IDENTICAL results/sweep/analysis/selection.csv 37df4e7b13fedf3436c04697fa7660125e08118e14fe4786ac67bcd546c43ac7
IDENTICAL results/sweep/analysis/test_at_selected_beta.csv df68a30929c59c76773c92323fcdbd30dfdb8bb5de3366dd05dc7a4819b70614
IDENTICAL results/sweep/analysis/repro_vs_previous.csv 408bfac4b324e00bc21362fb3766df6208d8787eae3612428f229f8aa7426696
IDENTICAL results/sweep/analysis/noise_checks.json c23509183a8c69f4584b8d09083f4dd0c0ef10a1b63dba62e1677ab52375b38a
IDENTICAL results/sweep/analysis/compare_with_grid.csv b995d8541134ee3522a5f28aa4cae097b68e2cd28b2c6fd82bc3ce5c4f04c48d
IDENTICAL results/sweep_repro/compare_repro.json 9d2e73c4021aceb89925f879100f562db8e5e996bcbe043f443d369ac088317c
```
Stdout tables were identical after normalizing only the printed root path
(`data_orig` vs `data_new`); the only textual difference was the path in the
"written ..." lines. Running the copies with the default root also reproduced
the same six files. The same six hashes were also compared with the on-disk
`results/` outputs and matched (`MATCH` for all six). The committed copies are
byte-identical to the copies used in that scratch run (`diff` reported no
difference for all four scripts). At the end of this session all four copies
were re-run on the scratch mirror with the repository virtualenv, e.g.
```
$ .venv/bin/python data_new/scripts/sweep/analyze_sweep.py --repo-root data_new
```
Each returned `rc=0`, and the six output files again hashed exactly as in the
block above; stdout was identical to the original scripts after normalizing the
mirror directory name (`data_orig` vs `data_new`).
The existing test suite was run with the repository virtualenv (system
`python3` has no `pytest` or `numpy`, so `python3 -m pytest` fails with
`No module named pytest`; the venv is `.venv/`). It was run after the commit
was pushed; the commit contains no test or source changes beyond the five files
listed above:
```
$ PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -m pytest -q -p no:cacheprovider
........................................................................ [ 96%]
... [100%]
$ echo $?
0
```
75 test nodes ran, all passed (75 `.`, 0 `F`, 0 `E`, 0 `s`), exit code 0.
`python3 -m py_compile scripts/sweep/*.py` also succeeded (`rc=0`; the
resulting `scripts/sweep/__pycache__/` was removed afterwards, so the working
tree contains only the five committed files).
Write-target check: the only write sites in the copies are
`results/sweep/analysis/*` (three files from `analyze_sweep.py`,
`noise_checks.json` from `check_noise.py`, `compare_with_grid.csv` from
`compare_with_grid.py`) and
`results/sweep_repro/compare_repro.json`. `git ls-files results/` is empty, so
none of these is tracked. The scratch runs wrote only inside
`~/restore_work/sweep_scripts_check/`.
Credential/privacy scan: the staged files were scanned before the commit, and
the committed blobs were re-scanned during this session with
`git show HEAD:scripts/sweep/<file>` concatenated into one 697-line stream:
```
$ grep -cEi '(api[_-]?key|secret|access[_-]?token|auth[_-]?token|password|passwd|credential|bearer |private[_-]?key|BEGIN [A-Z ]*PRIVATE KEY|ssh-rsa|ssh-ed25519|ghp_|github_pat_|AKIA[0-9A-Z]{16}|xox[baprs]-)' <stream>
0 # grep exit code 1
$ grep -cE '(/home/|/Users/|/root/|~/|localhost|github-thesis|willtanoe|([0-9]{1,3}\.){3}[0-9]{1,3})' <stream>
0 # grep exit code 1
```
Both scans returned **0 matches**. The generic high-entropy scan matched
only identifier-like strings such as
`val_rare_class_macro_f1_per_client_mean` and path fragments
`results/sweep_repro/a_1thread_repeat/...`; these are not credentials. A
case-insensitive scan for `aster` matched the substring in "master" in
`compare_with_grid.py` (`final__six_condition_master.csv`), not a hostname.
---
## 10. Server-side hashes
`~/restore_work/final_hashes.txt` contains the sorted SHA-256 list of every file
under `results/sweep/` and `results/sweep_repro/` on this server.
Command used:
```
find results/sweep results/sweep_repro -type f -print0 \
| xargs -0 sha256sum | LC_ALL=C sort -k2
```
- Files hashed: **912** (`results/sweep`: 818, `results/sweep_repro`: 94).
- Hashing was done in two complete independent passes, both finishing
2026-10-29T17:21:11Z. `diff pass1 pass2` reported no differences, so **no
file changed during hashing** and the file set did not change between passes
(the file header states this explicitly).
- At hashing time `results/sweep/PHASE` was `done`, the last heartbeat was
`2026-10-29T00:02:03Z`, and no `run_one.sh` / `xargs` / `launch_all` process
was running.
- The written manifest was read back with `sha256sum -c`: **912 OK, 0 failures**
(`verify rc=0`) at 2026-10-29T17:21:20Z, and again at the end of the session
at 2026-10-29T17:27:50Z with the same result (912 OK, 0 failures, file count
still 912). Nothing under `results/` was written by this session.
- The manifest itself is not committed (it lives in `~/restore_work/`, outside
the repository).
---
## 11. Open issues / unverified items
1. **Test suite required the repository virtualenv.** `.venv/bin/python -m
pytest` passes (75 tests); the system `python3 -m pytest` fails with
`No module named pytest`. Nothing was installed to change that.
2. **First attempts of `xiiotid alpha0.25 beta0/beta1` have no END line and no
measured wall time.** Their run directories were overwritten by the
07:31:29Z re-run; the cause of the missing END is not recorded in the logs.
3. **The 2026-10-28T11:11:37Z `-.slice` OOM entry is unattributed.** It falls
inside a run that exited 0; no sweep log records a failure for it.
4. **`dmesg` is unreadable (EPERM);** OOM evidence is limited to
`journalctl -b`.
5. **The reference check's PASS verdict was produced after a bug fix,** not by
the original crash-prone invocation. The tolerance itself predates the
results (`reference_check.py.bak`, 2026-10-26T02:06:56Z; first reference run
02:10:14Z).
6. **Single seed.** The sweep is seed 0 only. No between-seed claim is made.
7. **The 5G-NIDD beta0.25 live cap raise is not reproducible from logs alone**
(the run.log still shows the 11G launch command); this is noted in the
pre-existing launch report.
8. **Report is not committed.** `kisikisithesis/SWEEP_SUMMARY_REPORT.md` is
untracked and is not part of commit `9fe584d`.
9. `results/` was not modified by this session; the analysis scripts were run
only against the scratch copy. The pre-existing on-disk analysis outputs
matched the fresh scratch outputs.