docs: refresh README and session handoff for current state
- README: add deep-comparison, post-hoc audit, and defense-preparation phases;
correct the not-computed list (exploratory post-hoc tests now exist, with
explicit small-n limits); add kisikisithesis/ and POSTHOC_STATISTICAL_AUDIT.md
to the structure, artifact index, key documents, and reproduction commands.
- SESSION_HANDOFF.md: reflect HEAD 34bab45 and the post-16 sessions, document
the prototype-snapshot tracking rule, refresh the next-session prompt.
This commit is contained in:
1 parent
111adb272e
commit
783127296a
2 files changed
+77
-19
No files matched your search
@@ -170,7 +170,10 @@ Documented phases, in order. Session numbers are used only where the repository
|
||||
| Session 10 — 5G-NIDD a0.75 | second 5G-NIDD condition | `docs/research/5GNIDD_ALPHA075_BETA_VALIDATION.md`, `SESSION10_AUDIT.md` | audit PASS |
|
||||
| Session 11 — 5G-NIDD a0.25 | final condition; completes the grid | `docs/research/5GNIDD_ALPHA025_BETA_VALIDATION.md`, `SESSION11_AUDIT.md` | audit PASS |
|
||||
| Session 12 — synthesis | consolidate all six conditions | `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md`, `CROSS_DATASET_BETA_ANALYSIS.md`, `docs/thesis/*`, `artifacts/analysis/final__*`, `artifacts/figures/final/*` | independent recomputation, zero mismatches |
|
||||
| Deep paired comparison | original vs proposed per condition | `docs/research/ORIGINAL_VS_PROPOSED_DEEP_COMPARISON.md`, `artifacts/analysis/deepcmp__*.csv`, `scripts/deep_comparison.py`, `scripts/make_comparison_figures.py` | manifest-hash pairing checks in code |
|
||||
| Post-hoc statistical audit | exploratory paired tests, selection stability | `docs/research/POSTHOC_STATISTICAL_AUDIT.md`, `artifacts/analysis/posthoc__*.csv`, `scripts/posthoc_statistical_audit.py` | manifest-hash pairing re-verified in code |
|
||||
| Proposal phase | write the thesis proposal and audit aids | `docs/proposal/*` | committed `e27c5a8` |
|
||||
| Defense preparation | robustness audit and examiner Q&A aids | `kisikisithesis/ROBUSTNESS_AUDIT_DAN_PERSIAPAN_PENGUJI.md`, `kisikisithesis/robustness_checks.py` | read-only checks over the actual implementation and persisted snapshots |
|
||||
| Handoff / README | keep repository state current | `docs/handoff/SESSION_HANDOFF.md`, `README.md` | this document |
|
||||
|
||||
## 7. Experiment Coverage and Reproducibility
|
||||
@@ -234,9 +237,16 @@ run. No scikit-learn `classification_report` is used; metrics are pure NumPy.
|
||||
- **Per-seed** values and **pooled** values are different quantities; the pooled score averages
|
||||
per-seed per-client-mean scores.
|
||||
|
||||
**Not computed by this repository:** micro-F1, ROC/AUC, statistical significance tests, and
|
||||
persisted per-class unseen-class recall (unseen-class evaluation is defined in code but only the
|
||||
unseen flags and union counts are saved).
|
||||
**Not computed by this repository:** micro-F1, ROC/AUC, and persisted per-class unseen-class
|
||||
recall (unseen-class evaluation is defined in code but only the unseen flags and union counts
|
||||
are saved).
|
||||
|
||||
**Exploratory statistical audit (no inferential claims).** `scripts/posthoc_statistical_audit.py`
|
||||
(read-only) computes paired per-seed deltas for the selected beta and exploratory exact tests
|
||||
(two-sided sign-flip permutation, exact Wilcoxon signed-rank) plus paired-t 95% CIs — all with
|
||||
explicit small-n limitations: three seeds give a minimum attainable sign-flip p of 0.25, so no
|
||||
statistical significance is claimed anywhere. Outputs: `artifacts/analysis/posthoc__*.csv`;
|
||||
full discussion in `docs/research/POSTHOC_STATISTICAL_AUDIT.md`.
|
||||
|
||||
## 9. Quantitative Experimental Results
|
||||
|
||||
@@ -759,10 +769,11 @@ scripts/ training runner, partition generation, and read-only ana
|
||||
configs/ dataset, experiment protocol, and model configuration (YAML)
|
||||
docs/proposal/ thesis proposal and internal audit aids
|
||||
docs/research/ frozen protocol, per-condition reports, cross-alpha/cross-dataset synthesis,
|
||||
session audits
|
||||
post-hoc statistical audit, session audits
|
||||
docs/baseline/ PROTEAN baseline investigation, implementation, and validation
|
||||
docs/thesis/ Chapter 4 blueprint and claims register
|
||||
docs/handoff/ session handoff (start here for a new session)
|
||||
kisikisithesis/ robustness audit and examiner-preparation aids (read-only checks)
|
||||
planningawal/ immutable research specification (do not modify)
|
||||
artifacts/partitions/ deterministic frozen Dirichlet partition manifests (18)
|
||||
artifacts/analysis/ committed machine-readable analysis outputs
|
||||
@@ -787,6 +798,12 @@ results/ local run outputs (git-ignored; 108 non-smoke run dirs +
|
||||
|
||||
# beta=0 vs original equivalence audit (read-only)
|
||||
.venv/bin/python scripts/verify_beta0_equivalence.py --dataset 5gnidd --alpha 0.25 --seeds 0 1 2
|
||||
|
||||
# post-hoc statistical audit (read-only; writes only posthoc__*.csv)
|
||||
.venv/bin/python scripts/posthoc_statistical_audit.py
|
||||
|
||||
# deterministic robustness checks over the actual implementation (prints only)
|
||||
.venv/bin/python kisikisithesis/robustness_checks.py
|
||||
```
|
||||
|
||||
A full training run (not needed; the grid is complete) is:
|
||||
@@ -842,6 +859,7 @@ Analysis outputs (`artifacts/analysis/`), by function:
|
||||
| Per-condition test | `*__multiseed__test_descriptive.csv` |
|
||||
| Cross-alpha | `{xiiotid,5gnidd}__cross_alpha__*.csv/.json` |
|
||||
| Cross-dataset | `cross_dataset__matched_alpha_summary.csv/.json`, `cross_dataset__beta_curve.csv` |
|
||||
| Post-hoc audit | `posthoc__paired_global_per_seed.csv`, `posthoc__paired_global_by_condition.csv`, `posthoc__paired_test_results.csv`, `posthoc__selection_stability.csv`, `posthoc__per_class_summary.csv`, `posthoc__per_client_summary.csv` |
|
||||
| beta=0 equivalence | `*__multiseed__beta0_equivalence.csv` |
|
||||
| Runtime | `*__multiseed__runtime.csv` |
|
||||
| Prototype snapshots & weights | `*__prototype_snapshot.json`, `*__prototype_weights.csv`, `*__prototype_effect*.csv` |
|
||||
@@ -859,6 +877,7 @@ Key documents:
|
||||
| `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md` | authoritative six-condition synthesis |
|
||||
| `docs/research/ORIGINAL_VS_PROPOSED_DEEP_COMPARISON.md` | paired original-vs-proposed deep comparison |
|
||||
| `docs/research/CROSS_DATASET_BETA_ANALYSIS.md` | cross-dataset comparison |
|
||||
| `docs/research/POSTHOC_STATISTICAL_AUDIT.md` | exploratory paired statistical audit of the completed grid |
|
||||
| `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md` | X-IIoTID cross-alpha analysis |
|
||||
| `docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md` | 5G-NIDD cross-alpha analysis |
|
||||
| `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md` | Chapter 4 blueprint |
|
||||
|
||||
Reference in new issue
Block a user