From 783127296a3200254115c76bdfa52ac0e93e6b2e Mon Sep 17 00:00:00 2001 From: willtanoe Date: Sat, 17 Oct 2026 04:08:38 +0700 Subject: [PATCH] docs: refresh README and session handoff for current state - README: add deep-comparison, post-hoc audit, and defense-preparation phases; correct the not-computed list (exploratory post-hoc tests now exist, with explicit small-n limits); add kisikisithesis/ and POSTHOC_STATISTICAL_AUDIT.md to the structure, artifact index, key documents, and reproduction commands. - SESSION_HANDOFF.md: reflect HEAD 34bab45 and the post-16 sessions, document the prototype-snapshot tracking rule, refresh the next-session prompt. --- README.md | 27 +++++++++++-- docs/handoff/SESSION_HANDOFF.md | 69 ++++++++++++++++++++++++++------- 2 files changed, 77 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index 239e84e..01c069a 100644 --- a/README.md +++ b/README.md @@ -170,7 +170,10 @@ Documented phases, in order. Session numbers are used only where the repository | Session 10 — 5G-NIDD a0.75 | second 5G-NIDD condition | `docs/research/5GNIDD_ALPHA075_BETA_VALIDATION.md`, `SESSION10_AUDIT.md` | audit PASS | | Session 11 — 5G-NIDD a0.25 | final condition; completes the grid | `docs/research/5GNIDD_ALPHA025_BETA_VALIDATION.md`, `SESSION11_AUDIT.md` | audit PASS | | Session 12 — synthesis | consolidate all six conditions | `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md`, `CROSS_DATASET_BETA_ANALYSIS.md`, `docs/thesis/*`, `artifacts/analysis/final__*`, `artifacts/figures/final/*` | independent recomputation, zero mismatches | +| Deep paired comparison | original vs proposed per condition | `docs/research/ORIGINAL_VS_PROPOSED_DEEP_COMPARISON.md`, `artifacts/analysis/deepcmp__*.csv`, `scripts/deep_comparison.py`, `scripts/make_comparison_figures.py` | manifest-hash pairing checks in code | +| Post-hoc statistical audit | exploratory paired tests, selection stability | `docs/research/POSTHOC_STATISTICAL_AUDIT.md`, `artifacts/analysis/posthoc__*.csv`, `scripts/posthoc_statistical_audit.py` | manifest-hash pairing re-verified in code | | Proposal phase | write the thesis proposal and audit aids | `docs/proposal/*` | committed `e27c5a8` | +| Defense preparation | robustness audit and examiner Q&A aids | `kisikisithesis/ROBUSTNESS_AUDIT_DAN_PERSIAPAN_PENGUJI.md`, `kisikisithesis/robustness_checks.py` | read-only checks over the actual implementation and persisted snapshots | | Handoff / README | keep repository state current | `docs/handoff/SESSION_HANDOFF.md`, `README.md` | this document | ## 7. Experiment Coverage and Reproducibility @@ -234,9 +237,16 @@ run. No scikit-learn `classification_report` is used; metrics are pure NumPy. - **Per-seed** values and **pooled** values are different quantities; the pooled score averages per-seed per-client-mean scores. -**Not computed by this repository:** micro-F1, ROC/AUC, statistical significance tests, and -persisted per-class unseen-class recall (unseen-class evaluation is defined in code but only the -unseen flags and union counts are saved). +**Not computed by this repository:** micro-F1, ROC/AUC, and persisted per-class unseen-class +recall (unseen-class evaluation is defined in code but only the unseen flags and union counts +are saved). + +**Exploratory statistical audit (no inferential claims).** `scripts/posthoc_statistical_audit.py` +(read-only) computes paired per-seed deltas for the selected beta and exploratory exact tests +(two-sided sign-flip permutation, exact Wilcoxon signed-rank) plus paired-t 95% CIs — all with +explicit small-n limitations: three seeds give a minimum attainable sign-flip p of 0.25, so no +statistical significance is claimed anywhere. Outputs: `artifacts/analysis/posthoc__*.csv`; +full discussion in `docs/research/POSTHOC_STATISTICAL_AUDIT.md`. ## 9. Quantitative Experimental Results @@ -759,10 +769,11 @@ scripts/ training runner, partition generation, and read-only ana configs/ dataset, experiment protocol, and model configuration (YAML) docs/proposal/ thesis proposal and internal audit aids docs/research/ frozen protocol, per-condition reports, cross-alpha/cross-dataset synthesis, - session audits + post-hoc statistical audit, session audits docs/baseline/ PROTEAN baseline investigation, implementation, and validation docs/thesis/ Chapter 4 blueprint and claims register docs/handoff/ session handoff (start here for a new session) +kisikisithesis/ robustness audit and examiner-preparation aids (read-only checks) planningawal/ immutable research specification (do not modify) artifacts/partitions/ deterministic frozen Dirichlet partition manifests (18) artifacts/analysis/ committed machine-readable analysis outputs @@ -787,6 +798,12 @@ results/ local run outputs (git-ignored; 108 non-smoke run dirs + # beta=0 vs original equivalence audit (read-only) .venv/bin/python scripts/verify_beta0_equivalence.py --dataset 5gnidd --alpha 0.25 --seeds 0 1 2 + +# post-hoc statistical audit (read-only; writes only posthoc__*.csv) +.venv/bin/python scripts/posthoc_statistical_audit.py + +# deterministic robustness checks over the actual implementation (prints only) +.venv/bin/python kisikisithesis/robustness_checks.py ``` A full training run (not needed; the grid is complete) is: @@ -842,6 +859,7 @@ Analysis outputs (`artifacts/analysis/`), by function: | Per-condition test | `*__multiseed__test_descriptive.csv` | | Cross-alpha | `{xiiotid,5gnidd}__cross_alpha__*.csv/.json` | | Cross-dataset | `cross_dataset__matched_alpha_summary.csv/.json`, `cross_dataset__beta_curve.csv` | +| Post-hoc audit | `posthoc__paired_global_per_seed.csv`, `posthoc__paired_global_by_condition.csv`, `posthoc__paired_test_results.csv`, `posthoc__selection_stability.csv`, `posthoc__per_class_summary.csv`, `posthoc__per_client_summary.csv` | | beta=0 equivalence | `*__multiseed__beta0_equivalence.csv` | | Runtime | `*__multiseed__runtime.csv` | | Prototype snapshots & weights | `*__prototype_snapshot.json`, `*__prototype_weights.csv`, `*__prototype_effect*.csv` | @@ -859,6 +877,7 @@ Key documents: | `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md` | authoritative six-condition synthesis | | `docs/research/ORIGINAL_VS_PROPOSED_DEEP_COMPARISON.md` | paired original-vs-proposed deep comparison | | `docs/research/CROSS_DATASET_BETA_ANALYSIS.md` | cross-dataset comparison | +| `docs/research/POSTHOC_STATISTICAL_AUDIT.md` | exploratory paired statistical audit of the completed grid | | `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md` | X-IIoTID cross-alpha analysis | | `docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md` | 5G-NIDD cross-alpha analysis | | `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md` | Chapter 4 blueprint | diff --git a/docs/handoff/SESSION_HANDOFF.md b/docs/handoff/SESSION_HANDOFF.md index 1b20595..bc1784b 100644 --- a/docs/handoff/SESSION_HANDOFF.md +++ b/docs/handoff/SESSION_HANDOFF.md @@ -3,12 +3,17 @@ This is the canonical continuation point for the next agent session. It is written to be readable without prior chat history, terminal scrollback, or git history. -Last updated: 2026-10-02 (Session 16). Session 16 type: **handoff refresh** — no training, no -experiments, no source/model/aggregation change, no proposal content change. +Last updated: 2026-10-17 (Session 18). Session 18 type: **repository hygiene and documentation +refresh** — no training, no experiments, no source/model/aggregation change, no proposal content +change. It committed the previously untracked post-hoc statistical audit and defense-preparation +work, codified the prototype-snapshot tracking rule in `.gitignore`, refreshed this handoff and +`README.md`, declared `matplotlib` in `pyproject.toml`, and applied three behavior-preserving code +nits (docstring fix, device-safe embedding extraction, empty-table guard). > **Current state in one line.** The 108-run experimental grid is complete and synthesized; the -> thesis proposal Markdown has been finalized and committed. Remaining work is academic writing -> and, later, formatting into the university template — not new experiments. +> thesis proposal Markdown has been finalized and committed; the post-hoc statistical audit and +> robustness/defense-preparation aids are committed. Remaining work is academic writing and, later, +> formatting into the university template — not new experiments. --- @@ -80,6 +85,15 @@ It is **frozen** unless academic/supervisor feedback requires revision. It is wr - `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md` — authoritative six-condition synthesis (RQ1/RQ2/RQ3 answers, threats to validity, contribution, unsupported claims, gap assessment). +- `docs/research/ORIGINAL_VS_PROPOSED_DEEP_COMPARISON.md` — paired original-vs-proposed deep + comparison (`artifacts/analysis/deepcmp__*.csv`). +- `docs/research/POSTHOC_STATISTICAL_AUDIT.md` — exploratory paired statistical audit of the + completed grid (`scripts/posthoc_statistical_audit.py`, `artifacts/analysis/posthoc__*.csv`); + explicit small-n limits (minimum attainable sign-flip p = 0.25 with three seeds); no + significance claims. +- `kisikisithesis/ROBUSTNESS_AUDIT_DAN_PERSIAPAN_PENGUJI.md` — robustness audit and examiner + Q&A preparation; `kisikisithesis/robustness_checks.py` runs deterministic read-only checks + over the actual implementation and persisted snapshots (prints only, writes nothing). - `docs/research/CROSS_DATASET_BETA_ANALYSIS.md` — cross-dataset comparison. - `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md` — X-IIoTID cross-alpha analysis. - `docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md` — 5G-NIDD cross-alpha analysis. @@ -143,7 +157,10 @@ These constraints must be respected in every future text (proposal, chapters, sl 3. **Later, draft thesis chapters** (starting with Chapter 4) from the final synthesis documents and `artifacts/analysis/final__*` + `artifacts/figures/final/`, obeying `CLAIMS_REGISTER.md` and `CHAPTER4_RESULTS_OUTLINE.md`. -4. **No more experiments** unless required by new academic feedback. +4. **For defense preparation**, use `kisikisithesis/ROBUSTNESS_AUDIT_DAN_PERSIAPAN_PENGUJI.md` + (robustness audit, examiner Q&A, oral cheat sheet) and + `docs/research/POSTHOC_STATISTICAL_AUDIT.md`; keep the exploratory-test caveats explicit. +5. **No more experiments** unless required by new academic feedback. --- @@ -174,11 +191,20 @@ These constraints must be respected in every future text (proposal, chapters, sl ## 12. Git state -- Branch `main`, `HEAD = e27c5a83abcb84ab31c667b4d39d580012fa6f26`, in sync with `origin/main`. -- Session 15 commit: `docs: finalize thesis proposal markdown` (four proposal files). -- No tracked working-tree changes expected between sessions; only pre-existing untracked - `artifacts/analysis/*prototype_snapshot.json` files (94) remain untracked and must not be staged - accidentally. +- Branch `main`, `HEAD = 34bab45` (`docs: improve readability of PROTEAN comparison figures`); + sessions after 16 added the deep paired comparison, the redesigned final figures, and the + experiments documentation before this branch. +- Session 18 work lives on branch **`chore/posthoc-audit-and-docs-refresh`** (not yet merged or + pushed): + - `111adb2` `research: add post-hoc statistical audit and defense preparation aids` + (posthoc script + doc + 6 CSVs, `kisikisithesis/`, 4 remaining beta0-snapshot effect/weight + CSVs, `.gitignore` snapshot rule) + - documentation refresh (README, this handoff, `pyproject.toml`, three code nits) — commits + after `111adb2` on the same branch. +- Prototype-snapshot tracking rule (now explicit in `.gitignore`): the **18 beta0-round-0 + snapshots and their effect/weight CSVs are tracked**; the remaining ~76 non-beta0 snapshot + JSONs (~28 MB, inputs to already-committed displacement tables) stay **untracked** — cited by + path in the audit documents, like `results/`. Do not stage them. - `results/` is git-ignored. Do not commit raw datasets, large artifacts, or `results/`. ## 13. Integrity / important commands @@ -196,12 +222,18 @@ git diff --stat .venv/bin/python scripts/analyze_partition_heterogeneity.py --dataset 5gnidd --alphas 0.25 0.5 0.75 .venv/bin/python scripts/analyze_cross_alpha.py --dataset 5gnidd --alphas 0.25 0.5 0.75 .venv/bin/python scripts/verify_beta0_equivalence.py --dataset 5gnidd --alpha 0.25 --seeds 0 1 2 + +# Post-hoc statistical audit (read-only; writes only posthoc__*.csv) +.venv/bin/python scripts/posthoc_statistical_audit.py + +# Deterministic robustness checks (prints only, writes nothing) +.venv/bin/python kisikisithesis/robustness_checks.py ``` ## 14. Recommended next-session prompt ``` -You are continuing the perdalam-protean research repository (Session 17). +You are continuing the perdalam-protean research repository (Session 19). FIRST, read completely: 1. docs/handoff/SESSION_HANDOFF.md @@ -218,11 +250,18 @@ Critical rules: - Beta selection uses validation only; PRIMARY is pooled across clients and seeds. - The 108-run 2-dataset x 3-alpha x 3-seed grid is COMPLETE. Do NOT run new training. - The proposal Markdown is finalized and frozen unless academic feedback requires revision. + - Non-beta0 prototype snapshots stay untracked (see .gitignore); do not stage them. -Context: Session 15 finalized and committed the proposal Markdown -(docs/proposal/THESIS_PROPOSAL_DRAFT.md) at HEAD e27c5a8. Session 16 refreshed this handoff. -"Optimasi" in the title means empirical beta selection among predefined candidates, not a global -optimum. All six conditions are synthesized; no material scientific gap remains (classification A). +Context: Session 18 (branch chore/posthoc-audit-and-docs-refresh) committed the previously +untracked post-hoc statistical audit (scripts/posthoc_statistical_audit.py, +docs/research/POSTHOC_STATISTICAL_AUDIT.md, artifacts/analysis/posthoc__*.csv) and the +defense-preparation aids (kisikisithesis/), refreshed the handoff/README, and added matplotlib +to pyproject.toml. Sessions after 16 added the deep paired comparison +(docs/research/ORIGINAL_VS_PROPOSED_DEEP_COMPARISON.md, deepcmp__*.csv) and the redesigned +final figures. "Optimasi" in the title means empirical beta selection among predefined +candidates, not a global optimum. All six conditions are synthesized; no material scientific +gap remains (classification A). The post-hoc tests are EXPLORATORY ONLY (n=3 seeds; minimum +attainable sign-flip p = 0.25); never present them as significance. Task: perform the requested writing/formatting or review work using the final synthesis documents and artifacts/analysis/final__*. Obey CLAIMS_REGISTER.md wording, keep all limitations explicit,