chore: version-control sweep analysis scripts

Copy analyze_sweep.py, check_noise.py, compare_with_grid.py and
compare_repro.py out of the git-ignored results/ tree into scripts/sweep/.
Replace the implicit script-location repository root with a --repo-root
argument (relative default) without changing any computation.  Outputs were
verified byte-identical to the originals on a scratch copy of the data.
This commit is contained in:
willtanoe committed 2026-10-29 18:17:43 +01:00
1 parent e66ecc4b59
commit 9fe584d08e
5 files changed
+697

No files matched your search

+99
View File
@@ -0,0 +1,99 @@
# Sweep analysis scripts
Read-only analysis helpers for the completed seed-0 sweeps. They were originally
written inside `results/sweep/` and `results/sweep_repro/` (both git-ignored);
these copies are the version-controlled source.
All four scripts accept `--repo-root` (default: the repository that contains
`scripts/sweep/`, i.e. two levels up from the script). Data is read from the
sweep run directories and, where noted, from the earlier grid runs and
`artifacts/analysis/`. Nothing is retrained.
## `analyze_sweep.py`
Inputs
- `results/sweep/runs/<dataset>__alpha<alpha>__seed0__support_weighted__beta<beta>/COMPLETE`
- `.../results/<run>/val_selection.json`
- `.../results/<run>/test_global_metrics.json`
- `results/<run>/val_selection.json` and `test_global_metrics.json` for the earlier
grid runs (stage 3)
Outputs
- `results/sweep/analysis/selection.csv` (validation only)
- `results/sweep/analysis/test_at_selected_beta.csv`
- `results/sweep/analysis/repro_vs_previous.csv`
Command
```bash
python3 scripts/sweep/analyze_sweep.py
# or, against a different tree:
python3 scripts/sweep/analyze_sweep.py --repo-root /path/to/repo
```
## `check_noise.py`
Inputs
- `results/sweep/runs/<run>/results/<run>/{history.json,val_selection.json,test_global_metrics.json}`
- `results/<run>/{history.json,val_selection.json,test_global_metrics.json}` (earlier grid)
Outputs
- `results/sweep/analysis/noise_checks.json`
Command
```bash
python3 scripts/sweep/check_noise.py
# or:
python3 scripts/sweep/check_noise.py --repo-root /path/to/repo
```
## `compare_with_grid.py`
Inputs
- `results/sweep/analysis/selection.csv` (run `analyze_sweep.py` first)
- `artifacts/analysis/final__six_condition_master.csv`
- `artifacts/analysis/<dataset>__alpha<alpha>__multiseed__validation_per_seed.csv`
Outputs
- `results/sweep/analysis/compare_with_grid.csv`
Command
```bash
python3 scripts/sweep/compare_with_grid.py
# or:
python3 scripts/sweep/compare_with_grid.py --repo-root /path/to/repo
```
## `compare_repro.py`
Inputs
- `results/sweep_repro/a_1thread_repeat/results/<run>/`
- `results/sweep_repro/b_8thread/results/<run>/`
- `results/sweep/runs/<run>/results/<run>/`
- `results/<run>/` (earlier grid)
- prototype snapshots under `results/sweep_repro/*/analysis/`,
`results/sweep/runs/<run>/analysis/` and `artifacts/analysis/`
where `<run>` is `xiiotid__alpha0.25__seed0__support_weighted__beta0`.
Outputs
- `results/sweep_repro/compare_repro.json`
- a compact comparison table on stdout
Command
```bash
python3 scripts/sweep/compare_repro.py
# or:
python3 scripts/sweep/compare_repro.py --repo-root /path/to/repo
```
## Notes
- `results/` is git-ignored. These scripts write only into
`results/sweep/analysis/` and `results/sweep_repro/`; they never write into a
tracked file.
- `analyze_sweep.py` opens validation data in stage 1 and only opens test data
afterwards, by construction. The scripts do not select or recommend a beta for
the thesis; they print descriptive tables.
- The computation is unchanged from the originals. The only change is that the
repository root is a `--repo-root` argument with a relative default instead of
being derived from the script's old location inside `results/`.