chore: version-control sweep analysis scripts
Copy analyze_sweep.py, check_noise.py, compare_with_grid.py and compare_repro.py out of the git-ignored results/ tree into scripts/sweep/. Replace the implicit script-location repository root with a --repo-root argument (relative default) without changing any computation. Outputs were verified byte-identical to the originals on a scratch copy of the data.
This commit is contained in:
1 parent
e66ecc4b59
commit
9fe584d08e
5 files changed
+697
No files matched your search
@@ -0,0 +1,99 @@
|
||||
# Sweep analysis scripts
|
||||
|
||||
Read-only analysis helpers for the completed seed-0 sweeps. They were originally
|
||||
written inside `results/sweep/` and `results/sweep_repro/` (both git-ignored);
|
||||
these copies are the version-controlled source.
|
||||
|
||||
All four scripts accept `--repo-root` (default: the repository that contains
|
||||
`scripts/sweep/`, i.e. two levels up from the script). Data is read from the
|
||||
sweep run directories and, where noted, from the earlier grid runs and
|
||||
`artifacts/analysis/`. Nothing is retrained.
|
||||
|
||||
## `analyze_sweep.py`
|
||||
|
||||
Inputs
|
||||
- `results/sweep/runs/<dataset>__alpha<alpha>__seed0__support_weighted__beta<beta>/COMPLETE`
|
||||
- `.../results/<run>/val_selection.json`
|
||||
- `.../results/<run>/test_global_metrics.json`
|
||||
- `results/<run>/val_selection.json` and `test_global_metrics.json` for the earlier
|
||||
grid runs (stage 3)
|
||||
|
||||
Outputs
|
||||
- `results/sweep/analysis/selection.csv` (validation only)
|
||||
- `results/sweep/analysis/test_at_selected_beta.csv`
|
||||
- `results/sweep/analysis/repro_vs_previous.csv`
|
||||
|
||||
Command
|
||||
```bash
|
||||
python3 scripts/sweep/analyze_sweep.py
|
||||
# or, against a different tree:
|
||||
python3 scripts/sweep/analyze_sweep.py --repo-root /path/to/repo
|
||||
```
|
||||
|
||||
## `check_noise.py`
|
||||
|
||||
Inputs
|
||||
- `results/sweep/runs/<run>/results/<run>/{history.json,val_selection.json,test_global_metrics.json}`
|
||||
- `results/<run>/{history.json,val_selection.json,test_global_metrics.json}` (earlier grid)
|
||||
|
||||
Outputs
|
||||
- `results/sweep/analysis/noise_checks.json`
|
||||
|
||||
Command
|
||||
```bash
|
||||
python3 scripts/sweep/check_noise.py
|
||||
# or:
|
||||
python3 scripts/sweep/check_noise.py --repo-root /path/to/repo
|
||||
```
|
||||
|
||||
## `compare_with_grid.py`
|
||||
|
||||
Inputs
|
||||
- `results/sweep/analysis/selection.csv` (run `analyze_sweep.py` first)
|
||||
- `artifacts/analysis/final__six_condition_master.csv`
|
||||
- `artifacts/analysis/<dataset>__alpha<alpha>__multiseed__validation_per_seed.csv`
|
||||
|
||||
Outputs
|
||||
- `results/sweep/analysis/compare_with_grid.csv`
|
||||
|
||||
Command
|
||||
```bash
|
||||
python3 scripts/sweep/compare_with_grid.py
|
||||
# or:
|
||||
python3 scripts/sweep/compare_with_grid.py --repo-root /path/to/repo
|
||||
```
|
||||
|
||||
## `compare_repro.py`
|
||||
|
||||
Inputs
|
||||
- `results/sweep_repro/a_1thread_repeat/results/<run>/`
|
||||
- `results/sweep_repro/b_8thread/results/<run>/`
|
||||
- `results/sweep/runs/<run>/results/<run>/`
|
||||
- `results/<run>/` (earlier grid)
|
||||
- prototype snapshots under `results/sweep_repro/*/analysis/`,
|
||||
`results/sweep/runs/<run>/analysis/` and `artifacts/analysis/`
|
||||
|
||||
where `<run>` is `xiiotid__alpha0.25__seed0__support_weighted__beta0`.
|
||||
|
||||
Outputs
|
||||
- `results/sweep_repro/compare_repro.json`
|
||||
- a compact comparison table on stdout
|
||||
|
||||
Command
|
||||
```bash
|
||||
python3 scripts/sweep/compare_repro.py
|
||||
# or:
|
||||
python3 scripts/sweep/compare_repro.py --repo-root /path/to/repo
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- `results/` is git-ignored. These scripts write only into
|
||||
`results/sweep/analysis/` and `results/sweep_repro/`; they never write into a
|
||||
tracked file.
|
||||
- `analyze_sweep.py` opens validation data in stage 1 and only opens test data
|
||||
afterwards, by construction. The scripts do not select or recommend a beta for
|
||||
the thesis; they print descriptive tables.
|
||||
- The computation is unchanged from the originals. The only change is that the
|
||||
repository root is a `--repo-root` argument with a relative default instead of
|
||||
being derived from the script's old location inside `results/`.
|
||||
Reference in new issue
Block a user