research: finalize experimental synthesis
This commit is contained in:
willtanoe committed
1 parent
d30e9831e6
commit
8a571a2387
21 files changed
+1325
-69
No files matched your search
@@ -3,30 +3,36 @@
|
||||
This is the canonical continuation point for the next agent session. It is written to be
|
||||
readable without prior chat history, terminal scrollback, or git history.
|
||||
|
||||
Last updated: 2026-09-30 (Session 11). Session 11 type: **final 5G-NIDD beta-validation condition**
|
||||
(`alpha=0.25`), completing the planned experiment grid. Sessions 6–8 completed the X-IIoTID
|
||||
three-alpha study (`alpha=0.25/0.50/0.75`, seeds `{0,1,2}`); Session 9 ran 5G-NIDD `alpha=0.50`;
|
||||
Session 10 ran 5G-NIDD `alpha=0.75`; Session 11 ran **only** 5G-NIDD `alpha=0.25` for seeds
|
||||
`{0,1,2}` across the five frozen beta candidates plus `original_protean`, using the same
|
||||
pooled-across-seed validation-only selection rule. **No condition was run after alpha=0.25.** No
|
||||
commit or push was made.
|
||||
Last updated: 2026-09-30 (Session 12). Session 12 type: **thesis synthesis of the complete audited
|
||||
grid** — no training, no partition regeneration, no source/model/aggregation change. Session 11 ran
|
||||
the final 5G-NIDD condition (`alpha=0.25`), completing the planned grid (108 full runs + 1 smoke).
|
||||
|
||||
> **Audit status (Session 11).** Independent audit verdict: **PASS**
|
||||
> (`docs/research/SESSION11_AUDIT.md`). `beta_star(5gnidd, alpha=0.25) = 0.00` confirmed from
|
||||
> validation only; run inventory 18/18; pairing hash-identical and reproduced from raw data;
|
||||
> rare/unseen 0 mismatches over 1,620 cells; leakage audit PASS; `beta=0` class **B**; source/data
|
||||
> integrity PASS; pytest 75 passed. **Session 11 is safe to checkpoint.**
|
||||
> **Session 12 scope and status.** Consolidated the 108-run grid into authoritative six-condition
|
||||
> tables, publication-ready figures, a final RQ1/RQ2/RQ3 synthesis, a Chapter-4 blueprint and a
|
||||
> claims register, then independently re-verified every number against the audited artifacts.
|
||||
> **No material scientific gap remains for the defined thesis scope (classification A).** No
|
||||
> training was executed, no tracked file changed, `planningawal/` and both raw datasets are
|
||||
> unchanged, `results/` is unchanged (108 non-smoke run dirs), and pytest is **75 passed**.
|
||||
> Deliverables: `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md`,
|
||||
> `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md`, `docs/thesis/CLAIMS_REGISTER.md`,
|
||||
> `artifacts/analysis/final__*.{csv,json}`, `artifacts/figures/final/figA–G*.png`.
|
||||
|
||||
> **Headline (Session 11).** PRIMARY pooled-validation selected beta for 5G-NIDD `alpha=0.25` =
|
||||
> **0.00** (validation only; per-seed diagnostics 0.25 / 0.00 / 0.00). The pooled validation curve
|
||||
> declines from 0.761 (beta=0) to 0.722 (beta=0.75) and rises slightly to 0.738 (beta=1.0), with
|
||||
> the between-beta spread (0.039) exceeding the across-seed std (0.003–0.022). This is the first
|
||||
> condition where the selected beta is 0, i.e. the equal-`S_j` mean; the selected mode is therefore
|
||||
> numerically the baseline and shows no support-weighting benefit. `beta=0` is class **B** for all
|
||||
> three seeds. Realized heterogeneity is the highest of the three 5G-NIDD conditions
|
||||
> (16/27 `|S_j|<M` cells, 30 missing cells, client ratio mean 17.60, mean CV 1.302), and the
|
||||
> rare-class union differs across seeds (6/6/8), so this condition's test rare support is not
|
||||
> constant.
|
||||
> **Final RQ findings (Session 12).** (1) Changing beta changes contribution weights and prototype
|
||||
> positions; for a fixed snapshot displacement is monotone increasing in beta in all six
|
||||
> conditions, and realized support imbalance and displacement co-vary within each dataset. (2) The
|
||||
> pooled validation rare-class response is **not monotone** in beta and is weak relative to seed
|
||||
> variance; the direct aggregation effect and the end-to-end training effect remain distinct.
|
||||
> (3) Selected betas are condition-dependent and non-monotone in alpha:
|
||||
> X-IIoTID `{0.00, 0.75, 0.50}` and 5G-NIDD `{0.00, 1.00, 0.50}` for alpha `{0.25, 0.50, 0.75}`;
|
||||
> both datasets select beta=0 at alpha=0.25 (no weighting benefit) and beta=0.50 at alpha=0.75;
|
||||
> no universal/global optimum. All six top-two selection gaps are smaller than the selected beta's
|
||||
> across-seed std. No inferential test was added (3 seeds).
|
||||
|
||||
> **Session 11 audit (preserved).** Verdict **PASS** (`docs/research/SESSION11_AUDIT.md`);
|
||||
> `beta_star(5gnidd, alpha=0.25) = 0.00` validation-only (per-seed 0.25/0.00/0.00); run inventory
|
||||
> 18/18; pairing hash-identical; rare/unseen 0 mismatches; leakage PASS; `beta=0` class **B**;
|
||||
> source/data integrity PASS; pytest 75 passed. Realized heterogeneity highest of the three 5G-NIDD
|
||||
> conditions (16/27 cells, 30 missing, client ratio 17.60, mean CV 1.302).
|
||||
|
||||
---
|
||||
|
||||
@@ -304,11 +310,13 @@ Six-condition, validation-only pooled selections:
|
||||
|
||||
## 15. Git status
|
||||
|
||||
No commit or push was made. `HEAD = bbd35085063194db58e4472e70b204d4be4a3607`, branch `main`, in
|
||||
sync with `origin/main`. Only untracked docs/analysis artifacts exist (no tracked modification);
|
||||
`results/` is gitignored. `planningawal/` and both raw datasets are unchanged. `equal_available`
|
||||
absent. No test-based selection. The 18 runs are the only `results/5gnidd__alpha0.25__*`
|
||||
directories.
|
||||
No commit or push was made in Session 12. `HEAD = d30e9831e6c17fc6b28383ae4592070ccd83657e`,
|
||||
branch `main`, in sync with `origin/main`. `git diff` / `git diff --cached` are empty (no tracked
|
||||
modification); only untracked docs/analysis/figures artifacts exist. `planningawal/` and both raw
|
||||
datasets are unchanged (SHA-256 match). `src/`, `tests/`, `configs/`, `scripts/`, `pyproject.toml`
|
||||
are unchanged. `results/` is gitignored and holds exactly 108 non-smoke run directories plus 1
|
||||
`smoke`. `equal_available` absent. No test-based selection. `matplotlib 3.11.2` was installed into
|
||||
the git-ignored `.venv` only; no dependency file was modified.
|
||||
|
||||
## 16. Session 11 audit result
|
||||
|
||||
@@ -321,45 +329,51 @@ directories.
|
||||
- Direct-effect, client-level, cross-alpha, cross-dataset labels: PASS.
|
||||
- Source/data integrity PASS. pytest: **75 passed, 0 failed, 0 skipped**.
|
||||
|
||||
## 17. Files created / modified this session
|
||||
## 17. Files created / modified (Session 12)
|
||||
|
||||
Created:
|
||||
|
||||
- `docs/research/5GNIDD_ALPHA025_BETA_VALIDATION.md`
|
||||
- `docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md`
|
||||
- `docs/research/CROSS_DATASET_BETA_ANALYSIS.md`
|
||||
- `docs/research/SESSION11_AUDIT.md`
|
||||
- `artifacts/analysis/5gnidd__alpha0.25__*` — 18 prototype snapshots + 3 per-seed
|
||||
effect/weights CSVs + `5gnidd__alpha0.25__multiseed__*` (validation per-seed/pooled,
|
||||
client detail/summary/deltas, test descriptive, prototype effect pooled, beta0 equivalence,
|
||||
runtime, selection summary, audit recomputation) + `5gnidd__alpha0.25__partition_characterization.csv`
|
||||
- `artifacts/analysis/5gnidd__cross_alpha__*` (regenerated to include alpha=0.25)
|
||||
- `artifacts/analysis/cross_dataset__matched_alpha_summary.{csv,json}`,
|
||||
`artifacts/analysis/cross_dataset__beta_curve.csv`
|
||||
- `results/5gnidd__alpha0.25__seed{0,1,2}__*` (18 run dirs, gitignored)
|
||||
- `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md` — final six-condition synthesis, RQ1/RQ2/RQ3
|
||||
answers, threats to validity, contribution, unsupported claims, gap assessment.
|
||||
- `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md` — precise Chapter-4 blueprint (purpose / artifact /
|
||||
key numbers / allowed / avoided per subsection).
|
||||
- `docs/thesis/CLAIMS_REGISTER.md` — claim / evidence / status / limitation / thesis-safe wording.
|
||||
- `artifacts/analysis/final__six_condition_master.{csv,json}`
|
||||
- `artifacts/analysis/final__pooled_validation_curves.csv` (30 rows)
|
||||
- `artifacts/analysis/final__realized_heterogeneity.csv`
|
||||
- `artifacts/analysis/final__prototype_displacement_by_beta.csv`
|
||||
- `artifacts/analysis/final__prototype_displacement_by_class.csv`
|
||||
- `artifacts/analysis/final__client_effect_summary.csv` (90 rows)
|
||||
- `artifacts/analysis/final__selected_vs_original.csv`
|
||||
- `artifacts/analysis/final__selection_strength.csv`
|
||||
- `artifacts/figures/final/figA–G*.png` — 8 thesis-friendly figures (matplotlib only).
|
||||
|
||||
Modified:
|
||||
|
||||
- `docs/handoff/SESSION_HANDOFF.md` (this file)
|
||||
- `artifacts/analysis/5gnidd__cross_alpha__*` (extended from 2 to 3 alphas; previous 2-alpha
|
||||
content was superseded, not silently lost — it is reproducible from the frozen manifests)
|
||||
- `docs/handoff/SESSION_HANDOFF.md` (this file).
|
||||
|
||||
No source or test code was changed.
|
||||
No source, test, config, model or aggregation file was changed. The synthesis generator is kept
|
||||
outside the repository at `/tmp/opencode/s12/synthesize_final_analysis.py` (Session-11 convention
|
||||
for ad-hoc analysis helpers). `matplotlib 3.11.2` was installed into the git-ignored `.venv`.
|
||||
|
||||
## 18. Exact recommended next phase
|
||||
|
||||
1. **Session 11 is audited and safe to checkpoint** (verdict PASS). Before checkpointing, stage
|
||||
only the intended docs/analysis artifacts; do not commit `results/` or the raw datasets.
|
||||
2. **The planned experiment grid is COMPLETE** (two datasets × three alphas × three seeds × five
|
||||
betas + baseline = 108 runs). Do **not** run any new condition.
|
||||
3. Remaining work is analysis/writing, not new training:
|
||||
- optionally add an inferential comparison (e.g. paired sign/permutation test over seeds) to
|
||||
formalise the weak selection effects — clearly labelled as post-hoc if added;
|
||||
- freeze the final thesis tables/report from the six-condition artifacts;
|
||||
- decide and document the `beta=0` vs `original_protean` reporting convention.
|
||||
4. Do not infer a monotone alpha→beta law; do not claim dataset causality or superiority.
|
||||
1. **The experiment grid is COMPLETE** (2 datasets × 3 alphas × 3 seeds × (5 betas + baseline) =
|
||||
108 runs). Do **not** run any new condition; no new training is required (gap classification A).
|
||||
2. **Synthesis is complete and verified.** All six selections, selected/original test summaries,
|
||||
heterogeneity values, displacement summaries and client counts were independently recomputed
|
||||
from raw artifacts with zero mismatches.
|
||||
3. Remaining work is **writing and formatting**, not experiments:
|
||||
- draft Chapter 4 from `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md` using the `final__*` tables
|
||||
and `figA–G` figures, respecting `docs/thesis/CLAIMS_REGISTER.md`;
|
||||
- optionally add appendices with the full per-seed and per-class tables;
|
||||
- if the examiners require inferential support, add a clearly post-hoc, explicitly
|
||||
underpowered analysis and label it as such (not done by default).
|
||||
4. Do not infer a monotone alpha→beta law; do not claim dataset causality, superiority,
|
||||
generalization, or a global optimum.
|
||||
5. Update `EXPERIMENT_PROTOCOL.md` only if a real ambiguity appears; update this handoff.
|
||||
6. Do not commit or push unless asked.
|
||||
6. Do not commit or push unless asked. When checkpointing, stage only intended docs/analysis/
|
||||
figure artifacts; never commit `results/` or raw datasets.
|
||||
|
||||
## 19. Important commands
|
||||
|
||||
@@ -394,16 +408,15 @@ No source or test code was changed.
|
||||
## 20. Recommended next-session prompt
|
||||
|
||||
```
|
||||
You are continuing the perdalam-protean research repository (Session 12).
|
||||
You are continuing the perdalam-protean research repository (Session 13).
|
||||
|
||||
FIRST, read completely:
|
||||
1. docs/handoff/SESSION_HANDOFF.md
|
||||
2. docs/research/SESSION11_AUDIT.md
|
||||
3. docs/research/5GNIDD_ALPHA025_BETA_VALIDATION.md
|
||||
4. docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md
|
||||
2. docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md
|
||||
3. docs/thesis/CHAPTER4_RESULTS_OUTLINE.md
|
||||
4. docs/thesis/CLAIMS_REGISTER.md
|
||||
5. docs/research/CROSS_DATASET_BETA_ANALYSIS.md
|
||||
6. docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md
|
||||
7. docs/research/EXPERIMENT_PROTOCOL.md
|
||||
6. docs/research/EXPERIMENT_PROTOCOL.md
|
||||
|
||||
Critical rules:
|
||||
- planningawal/ is immutable.
|
||||
@@ -412,14 +425,16 @@ Critical rules:
|
||||
- Do NOT add an executable equal_available mode.
|
||||
- Beta selection uses validation only; PRIMARY is pooled across clients and seeds.
|
||||
- The planned 2-dataset x 3-alpha experiment grid is COMPLETE. Do NOT run new conditions.
|
||||
- Scientific gap classification is A (NO MATERIAL GAP); no new experiment is required.
|
||||
|
||||
Context: Session 11 ran the final condition 5G-NIDD alpha=0.25 (18 runs, ~19.11 h), audited PASS,
|
||||
and selected beta_star=0.00 (validation only; per-seed 0.25/0.00/0.00). The six-condition pooled
|
||||
selections are 5G-NIDD 0.25/0.50/0.75 = 0.00/1.00/0.50 and X-IIoTID = 0.00/0.75/0.50; no universal
|
||||
or monotone beta. beta=0 equivalence class B. pytest 75 passed.
|
||||
Context: Session 12 completed the thesis synthesis with no training. Six pooled selections:
|
||||
X-IIoTID 0.25/0.50/0.75 = 0.00/0.75/0.50 and 5G-NIDD = 0.00/1.00/0.50; no universal/monotone beta;
|
||||
all top-two gaps below selected-beta across-seed std; beta=0 equivalence class B everywhere.
|
||||
Authoritative artifacts: artifacts/analysis/final__*.{csv,json}; figures artifacts/figures/final/.
|
||||
All synthesis numbers independently verified; pytest 75 passed; no tracked file changed.
|
||||
|
||||
Task: no new training. Consolidate the final thesis-ready six-condition tables/report from the
|
||||
existing audited artifacts, optionally add a clearly post-hoc inferential comparison of the weak
|
||||
pooled selections, and keep all limitations (S_j assumption, three seeds, no global optimum)
|
||||
explicit. Use .venv/bin/python. Do not commit or push unless asked.
|
||||
Task: write/format thesis chapters from the existing synthesis. Use the final__* tables and
|
||||
figA-G figures, obey CLAIMS_REGISTER.md wording, keep all limitations explicit, and do not run
|
||||
training or add inferential tests by default. Use .venv/bin/python. Do not commit or push unless
|
||||
asked.
|
||||
```
|
||||
Reference in new issue
Block a user