diff --git a/docs/handoff/SESSION_HANDOFF.md b/docs/handoff/SESSION_HANDOFF.md index fc480b3..1b20595 100644 --- a/docs/handoff/SESSION_HANDOFF.md +++ b/docs/handoff/SESSION_HANDOFF.md @@ -3,438 +3,229 @@ This is the canonical continuation point for the next agent session. It is written to be readable without prior chat history, terminal scrollback, or git history. -Last updated: 2026-09-30 (Session 12). Session 12 type: **thesis synthesis of the complete audited -grid** — no training, no partition regeneration, no source/model/aggregation change. Session 11 ran -the final 5G-NIDD condition (`alpha=0.25`), completing the planned grid (108 full runs + 1 smoke). +Last updated: 2026-10-02 (Session 16). Session 16 type: **handoff refresh** — no training, no +experiments, no source/model/aggregation change, no proposal content change. -> **Session 12 scope and status.** Consolidated the 108-run grid into authoritative six-condition -> tables, publication-ready figures, a final RQ1/RQ2/RQ3 synthesis, a Chapter-4 blueprint and a -> claims register, then independently re-verified every number against the audited artifacts. -> **No material scientific gap remains for the defined thesis scope (classification A).** No -> training was executed, no tracked file changed, `planningawal/` and both raw datasets are -> unchanged, `results/` is unchanged (108 non-smoke run dirs), and pytest is **75 passed**. -> Deliverables: `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md`, -> `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md`, `docs/thesis/CLAIMS_REGISTER.md`, -> `artifacts/analysis/final__*.{csv,json}`, `artifacts/figures/final/figA–G*.png`. - -> **Final RQ findings (Session 12).** (1) Changing beta changes contribution weights and prototype -> positions; for a fixed snapshot displacement is monotone increasing in beta in all six -> conditions, and realized support imbalance and displacement co-vary within each dataset. (2) The -> pooled validation rare-class response is **not monotone** in beta and is weak relative to seed -> variance; the direct aggregation effect and the end-to-end training effect remain distinct. -> (3) Selected betas are condition-dependent and non-monotone in alpha: -> X-IIoTID `{0.00, 0.75, 0.50}` and 5G-NIDD `{0.00, 1.00, 0.50}` for alpha `{0.25, 0.50, 0.75}`; -> both datasets select beta=0 at alpha=0.25 (no weighting benefit) and beta=0.50 at alpha=0.75; -> no universal/global optimum. All six top-two selection gaps are smaller than the selected beta's -> across-seed std. No inferential test was added (3 seeds). - -> **Session 11 audit (preserved).** Verdict **PASS** (`docs/research/SESSION11_AUDIT.md`); -> `beta_star(5gnidd, alpha=0.25) = 0.00` validation-only (per-seed 0.25/0.00/0.00); run inventory -> 18/18; pairing hash-identical; rare/unseen 0 mismatches; leakage PASS; `beta=0` class **B**; -> source/data integrity PASS; pytest 75 passed. Realized heterogeneity highest of the three 5G-NIDD -> conditions (16/27 cells, 30 missing, client ratio 17.60, mean CV 1.302). +> **Current state in one line.** The 108-run experimental grid is complete and synthesized; the +> thesis proposal Markdown has been finalized and committed. Remaining work is academic writing +> and, later, formatting into the university template — not new experiments. --- -## 1. Project objective +## 1. Final research scope -Build the code and experiment infrastructure for a thesis study that modifies **only** the -server-side prototype aggregation of PROTEAN (Chennoufi et al., 2025), a federated -intrusion-detection framework. The mechanism weights each contributing client's class prototype -by its class sample support: +The thesis studies exactly one principal method, one component and one tunable mechanism: + +- principal method: **PROTEAN** (Chennoufi et al., 2025), a federated intrusion-detection + framework; +- component studied: **server-side prototype aggregation** (only this component is modified); +- tunable mechanism: **beta-controlled class-sample-support weighting** of each client's local + class prototype; +- analysis: **rare-class** and **per-client** behavior; +- datasets: **X-IIoTID** and **5G-NIDD**; +- non-IID conditions: symmetric Dirichlet **alpha = 0.75 / 0.50 / 0.25**; +- beta candidates: **{0; 0.25; 0.50; 0.75; 1.00}**, one beta for all classes and clients. + +The aggregation is: ``` Cbar_j(beta) = ( sum_{i in S_j} N_ij^beta * C_ij ) / ( sum_{i in S_j} N_ij^beta ) S_j = { i | N_ij > 0 and a valid local prototype C_ij exists } -beta in {0.0, 0.25, 0.5, 0.75, 1.0} ``` -Both the frozen `original_protean` baseline and the proposed `support_weighted` mode are -implemented. `beta = 0` is mathematically the equal mean over the shared `S_j` but is not -bitwise identical to the baseline (see §7). +where `C_ij` is client `i`'s local class-`j` prototype and `N_ij` is client `i`'s local training +sample count for class `j`. -## 2. Authoritative source and immutability +## 2. Final title + +**Optimasi Agregasi Prototype pada PROTEAN melalui Pembobotan Berbasis Dukungan Sampel Kelas +untuk Deteksi Serangan Langka pada Data Non-IID** + +English subtitle: *Optimizing Prototype Aggregation in PROTEAN through Class-Sample-Support +Weighting for Rare Attack Detection under Non-IID Data Distributions*. + +**Meaning of "Optimasi" (explicit):** empirical tuning/selection of beta among the predefined +discrete candidates using validation data. It does **not** mean a global mathematical optimum and +does **not** imply a universal beta. This clarification appears in the proposal header note, in +§1.7, and in §3.8. The title matches the immutable research specification +`planningawal/PROTEAN_Revisi_Lurus.md`. + +## 3. Proposal status (authoritative proposal files) + +- `docs/proposal/THESIS_PROPOSAL_DRAFT.md` — **the final Markdown proposal source for the current + scope.** Written in Indonesian; BAB I–III plus a short preliminary-study section and a + bibliography. +- `docs/proposal/PROPOSAL_SOURCE_MAP.md` — internal audit aid mapping every substantive proposal + claim to PAPER / SPEC / PROJECT / PRELIM / FINAL evidence. +- `docs/proposal/LITERATURE_MATRIX.md` — internal literature matrix (categories A–H, per-reference + role, limitations, metadata confidence). +- `docs/proposal/REFERENCES_VERIFIED.md` — internal verification record for every bibliography + entry. + +The proposal was finalized and committed in Session 15 (`docs: finalize thesis proposal markdown`). +It is **frozen** unless academic/supervisor feedback requires revision. It is written prospectively +(no final experimental results exposed; only feasibility-level preliminary statements). + +## 4. Experimental completion + +- **108 full runs complete**: 2 datasets × 3 alpha values × 3 seeds × (1 `original_protean` + baseline + 5 beta settings). One unrelated `smoke` directory is excluded. +- All six `(dataset, alpha)` conditions are trained, audited and **synthesized**. +- No further training is planned **unless new supervisor/academic feedback explicitly requires + it**. The scientific-gap classification for the defined scope is **A — NO MATERIAL GAP**. +- Full result tables are deliberately **not** reproduced here; load them from the synthesis + documents and `artifacts/analysis/final__*` instead. + +## 5. Important final research documents + +- `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md` — authoritative six-condition synthesis + (RQ1/RQ2/RQ3 answers, threats to validity, contribution, unsupported claims, gap assessment). +- `docs/research/CROSS_DATASET_BETA_ANALYSIS.md` — cross-dataset comparison. +- `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md` — X-IIoTID cross-alpha analysis. +- `docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md` — 5G-NIDD cross-alpha analysis. +- `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md` — precise Chapter-4 blueprint (allowed/avoided + interpretation per subsection). +- `docs/thesis/CLAIMS_REGISTER.md` — claim / evidence / status / limitation / thesis-safe wording. +- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen reproduction protocol. + +## 6. Claims / interpretation constraints + +These constraints must be respected in every future text (proposal, chapters, slides): + +1. `N_ij` is **class-specific sample support, not prototype quality**. +2. Support weighting is **not inherently novel**; sample-count prototype weighting is already + discussed in related work (e.g. FedProto). +3. **No claim that support weighting always improves detection.** Response is condition-dependent + and non-monotone. +4. **No universal/global-optimum beta claim.** The selected beta is "best among the tested + candidates under the defined pooled-validation procedure". +5. **No causal alpha -> beta claim.** Selection is not monotone in alpha; only three alpha levels, + two datasets, three seeds. +6. **No claim that larger prototype displacement means better detection.** Displacement is a + representation-aggregation metric, not detection quality. +7. `beta = 0` is **semantically equivalent** to the equal mean over the shared `S_j`, but is + **not guaranteed bitwise identical** to `original_protean` (floating-point divergence; + equivalence class **B** in every condition). +8. **Test data is never used for beta selection.** +9. **Primary selection = pooled validation across clients and seeds; per-seed selection is + diagnostic only.** +10. **`S_j` participant/denominator handling remains a documented reproduction assumption** (no + verified official PROTEAN code); all results are conditional on it. + +## 7. Known open academic items (all non-blocking) + +1. **Zhao et al. (2018)** remains an arXiv/preprint reference; it is flagged in the bibliography as + **[metadata perlu diverifikasi]** (no peer-reviewed venue asserted). +2. **1–2 additional peer-reviewed IDS-specific rare/class-imbalance references** could be added + later if needed; this is an identified literature gap, not an error. +3. **FedProto sample-count weighting attribution** should be rechecked against the primary paper + before defense if necessary (the claim is inherited from the immutable spec and is conservative + for novelty). +4. **`S_j` reproduction assumption** remains documented and unresolved; carry the caveat in all + downstream reporting. + +## 8. Research evolution (brief) + +- The earlier scope was broad: Differential Privacy plus multiple rare-class preservation + mechanisms. +- Supervisor feedback pushed the study toward a single, interpretable mechanism. +- The final scope isolates PROTEAN's server-side prototype aggregation and tunes `beta`. +- This avoids a whole-method A/B/C comparison and enables mechanism-level analysis of the chain: + support imbalance -> contribution weight -> global-prototype displacement -> rare-class + detection -> per-client impact. + +## 9. Next recommended work + +1. **Keep the proposal Markdown frozen** unless academic/supervisor feedback explicitly requires + revision. +2. **Later, convert the finalized Markdown to DOCX** using the university template (out of scope + for the current sessions). +3. **Later, draft thesis chapters** (starting with Chapter 4) from the final synthesis documents and + `artifacts/analysis/final__*` + `artifacts/figures/final/`, obeying `CLAIMS_REGISTER.md` and + `CHAPTER4_RESULTS_OUTLINE.md`. +4. **No more experiments** unless required by new academic feedback. + +--- + +## 10. Authoritative source and immutability - `planningawal/PROTEAN_Revisi_Lurus.md` — authoritative research specification. -- `planningawal/` is immutable. Re-verified unchanged this session: `git status --porcelain - planningawal/` empty. -- Raw datasets re-verified unchanged (SHA-256): - `~/datasets/5G-NIDD/5G-NIDD.csv` - `fa36f80859585f474504ca69eb951a079b35145537976c86b00aae3aab46ee59`; - `~/datasets/X-IIoTID/X-IIoTID-dataset.csv` - `7b9290057ee42e784da3c0d84b781815502c9205c74175c96374e71a5ffd98a0`. -- Frozen baseline (`original_protean`) source unchanged; no `equal_available` mode added; beta - candidate set unchanged (`{0, .25, .5, .75, 1}`). No tracked source/test/config file differs - from HEAD. Only new docs/analysis artifacts are untracked. +- `planningawal/` is **immutable**. Re-verified unchanged: `git status --porcelain planningawal/` + is empty. +- Raw datasets are external and read-only; do not modify, rename, move, copy or commit them: + - `~/datasets/5G-NIDD/5G-NIDD.csv` (SHA-256 + `fa36f80859585f474504ca69eb951a079b35145537976c86b00aae3aab46ee59`) + - `~/datasets/X-IIoTID/X-IIoTID-dataset.csv` (SHA-256 + `7b9290057ee42e784da3c0d84b781815502c9205c74175c96374e71a5ffd98a0`) -## 3. Session 11 objective and outcome +## 11. Methodological decisions / assumptions -Objective: run exactly the **final** condition — 5G-NIDD `alpha=0.25`, seeds `{0,1,2}`, -`original_protean` + five betas — after a Stage-A verification gate; then build the 5G-NIDD -three-alpha comparison and the cross-dataset comparison. Do not run anything after it, do not -change methodology. +1. `S_j = { i | N_ij > 0 and a valid C_ij exists }` with equal `1/|S_j|` weights in the baseline + is a **REPRODUCTION ASSUMPTION** (materially exercised in up to 19/27 class-condition cells). +2. `N_ij` is sample support, not prototype quality. +3. One beta is used for all classes/clients; candidates are frozen at + `{0, 0.25, 0.50, 0.75, 1.00}`. +4. Primary selection = pooled validation across clients and seeds; per-seed selection is secondary + (stability diagnostics); test is read only after the pooled beta is fixed. +5. No `equal_available` executable mode was added (it is arithmetically `support_weighted` at + `beta = 0` under the shared `S_j`). +6. `lambda = 1.0`, Adam, `lr = 1e-3`, batch 256, 10 clients, 10 rounds, 3 local epochs remain + project decisions; see `docs/research/EXPERIMENT_PROTOCOL.md` for the full frozen protocol. -Stage A result: raw hash valid; all three 5G-NIDD `alpha=0.25` manifests reproduce exactly from raw -data (support match, partition-hash match, split-hash match); label semantics and rare/unseen logic -valid; pairing framework valid. Gate = **GO**. +## 12. Git state -Outcome: 18/18 runs rc=0. **PRIMARY pooled-validation selected beta = 0.00** (validation only); -`test_used_for_selection = false` for all 15 runs. Pooled means: 0.00 → 0.761453, 0.25 → 0.749286, -0.50 → 0.725715, 0.75 → 0.722235, 1.00 → 0.737636. Reports: -`docs/research/5GNIDD_ALPHA025_BETA_VALIDATION.md`, `docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md`, -`docs/research/CROSS_DATASET_BETA_ANALYSIS.md`, audit `docs/research/SESSION11_AUDIT.md`. +- Branch `main`, `HEAD = e27c5a83abcb84ab31c667b4d39d580012fa6f26`, in sync with `origin/main`. +- Session 15 commit: `docs: finalize thesis proposal markdown` (four proposal files). +- No tracked working-tree changes expected between sessions; only pre-existing untracked + `artifacts/analysis/*prototype_snapshot.json` files (94) remain untracked and must not be staged + accidentally. +- `results/` is git-ignored. Do not commit raw datasets, large artifacts, or `results/`. -## 4. Condition and pairing - -| Field | Value | -|---|---| -| dataset / alpha / seeds | `5gnidd` / 0.25 / `{0,1,2}` | -| runs | 15 `support_weighted` + 3 `original_protean`, all new this session | -| training | 10 clients, 10 rounds, 3 local epochs, batch 256, lr 1e-3, mu 0.1, lambda 1.0, Adam | -| code revision | only docs/analysis added; HEAD `bbd35085063194db58e4472e70b204d4be4a3607` | - -Manifests (reused, never regenerated; one content hash per seed across all six runs; each -reproduced from the raw dataset with split/`N_ij`/partition hash verified): - -| seed | manifest content hash | partition hash | split hash | rng seed | round-0 snapshot SHA-256 | -|---|---|---|---|---|---| -| 0 | `ab6d8a8b63d478c4…` | `e628c17369f25e35…` | `18f89f4807de5493…` | 708356900 | `e2243ac4ed4391ff…` | -| 1 | `74df102e4edb4509…` | `05aec12ceb4c94bd…` | `3d05b7fe48d5a06c…` | 902143161 | `26a162991b7cbde9…` | -| 2 | `061c094d1913f942…` | `d9631ea6b2d3907e…` | `fc758d060dc51dec…` | 218373793 | `e287bff5ca71f8d2…` | - -Splits: train/validation/test = 851,123 / 121,589 / 243,177 for every seed (10 clients, 9 classes, -`Benign`). `train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv` identical across each -seed's runs. `manifest_reload_ok = true`, `smoke = false` everywhere. Pairing proven by content -hashes. - -## 5. Validation results (USED FOR SELECTION) and pooled selection - -Primary metric: validation per-client-mean rare-class macro-F1. All 10 clients eligible in every -seed. Full tables: `artifacts/analysis/5gnidd__alpha0.25__multiseed__validation_per_seed.csv`. - -Per-seed validation per-client-mean rare-class macro-F1: - -| beta | seed 0 | seed 1 | seed 2 | -|---|---|---|---| -| 0.00 | 0.736360 | **0.771268** | **0.776732** | -| 0.25 | **0.747299** | 0.767300 | 0.733259 | -| 0.50 | 0.717450 | 0.738223 | 0.721471 | -| 0.75 | 0.713262 | 0.728478 | 0.724966 | -| 1.00 | 0.734455 | 0.739297 | 0.739157 | - -Pooled across seeds (`...__validation_pooled.csv`): - -| beta | pooled mean | std (sample) | min | max | selected | -|---|---|---|---|---|---| -| **0.00** | **0.761453** | 0.021903 | 0.736360 | 0.776732 | **yes** | -| 0.25 | 0.749286 | 0.017107 | 0.733259 | 0.767300 | | -| 0.50 | 0.725715 | 0.011017 | 0.717450 | 0.738223 | | -| 0.75 | 0.722235 | 0.007967 | 0.713262 | 0.728478 | | -| 1.00 | 0.737636 | 0.002756 | 0.734455 | 0.739297 | | - -- **PRIMARY selected beta (`beta_star_5gnidd_alpha025`) = 0.00**, max pooled validation - per-client-mean rare-class macro-F1; **no tie-break used**; validation-only. Top-two gap ≈ 1.2e-2. -- **SECONDARY per-seed diagnostics only:** seed 0 → 0.25, seed 1 → 0.00, seed 2 → 0.00. -- Between-beta spread ≈ 0.039; between-seed std 0.003–0.022. At this condition the spread exceeds - the seed noise, but the sample is only 3 project-defined seeds. - -## 6. Final test evaluation of the pooled-selected beta - -`beta_star = 0.00` was fixed from pooled validation before test was read. - -| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) | -|---|---|---|---|---| -| rare-class macro-F1 | 0.859893 | 0.813889 | 0.849925 | 0.841236 ± 0.024202 | -| rare-class recall | 0.985234 | 0.972659 | 0.925756 | 0.961216 ± 0.031347 | -| macro-F1 | 0.829680 | 0.802987 | 0.817254 | 0.816640 ± 0.013357 | -| accuracy | 0.744844 | 0.737779 | 0.708052 | 0.730225 ± 0.019525 | -| weighted-F1 | 0.717404 | 0.712302 | 0.683684 | 0.704463 ± 0.018175 | - -`original_protean` for the same seeds/condition: - -| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) | -|---|---|---|---|---| -| rare-class macro-F1 | 0.864903 | 0.814848 | 0.847072 | 0.842274 ± 0.025370 | -| rare-class recall | 0.985444 | 0.972333 | 0.923890 | 0.960556 ± 0.032423 | -| macro-F1 | 0.835043 | 0.808373 | 0.814800 | 0.819405 ± 0.013919 | -| accuracy | 0.745280 | 0.741295 | 0.706329 | 0.730968 ± 0.021431 | -| weighted-F1 | 0.718041 | 0.716327 | 0.681939 | 0.705435 ± 0.020367 | - -Descriptive difference for test rare-class macro-F1 (`beta_star` − `original_protean`) is -−0.005010 / −0.000959 / +0.002853 (mean −0.001039). Because beta_star = 0, this difference is just -floating-point divergence (see §7). The global rare evaluation support differs by seed -(**141,677 / 129,824 / 147,630**), so cross-seed test levels are **not** directly comparable at -this condition. Test weighted-F1 (`≈0.704`) again sits **below** accuracy (`≈0.730`), as at -alpha=0.50/0.75. - -## 7. `beta = 0` equivalence result (all three seeds) - -For seeds 0, 1, 2, `support_weighted beta=0` vs frozen `original_protean`: - -- Same manifest/split/hashes, round-0 snapshot byte-identical, support/char/rare_unseen - identical: **yes** for all three. -- Independent aggregation (`ΣC/n` vs `(1/n)ΣC`): 4.4e-16 / 4.4e-16 / 6.7e-16. -- History rare-macro-F1 max diff (rounds ≥ 1): 2.17e-2 / 6.85e-2 / 2.74e-2; test metric max diff - 5.36e-3 / 5.39e-3 / 2.85e-3. -- Classification: **B** in all seeds — semantic equivalence + floating-point divergence; no - semantic discrepancy. Machine-readable: - `artifacts/analysis/5gnidd__alpha0.25__multiseed__beta0_equivalence.csv`. - -Conditional on the shared `S_j` reproduction assumption; not proof `beta=0` reproduces original -PROTEAN. Since the selected beta is 0, the selected-run test metrics differ from original only by -this divergence. - -## 8. Partition characterization (realized) and comparison with alpha=0.50/0.75 - -5G-NIDD `alpha=0.25` (training split), `M = 10`: - -| seed | `|S_j| < M` cells | missing client-class cells | client-size ratio | union rare | union unseen | max unseen/client | -|---|---|---|---|---|---|---|---| -| 0 | 6 / 9 | 10 | 9.93 | 6 | 6 | 3 | -| 1 | 5 / 9 | 10 | 27.83 | 6 | 5 | 3 | -| 2 | 5 / 9 | 10 | 15.05 | 8 | 5 | 2 | -| **sum/mean** | **16 / 27** | **30** | mean **17.60** | 6/6/8 | 6/5/5 | 3 | - -Mean class-support CV = **1.302**; max per-class `max/min` = 185,504. All 10 clients have exactly -two rare classes and are eligible. The rare-class union **differs across seeds** (6/6/8), so test -rare support is not constant (141,677 / 129,824 / 147,630). This is the most heterogeneous of the -three 5G-NIDD conditions. Full tables: -`artifacts/analysis/5gnidd__cross_alpha__partition_heterogeneity_*` and -`...__partition_characterization.csv`. - -### 5G-NIDD three-alpha comparison - -| statistic | a0.25 | a0.50 | a0.75 | -|---|---|---|---| -| pooled-selected beta | **0.00** | **1.00** | **0.50** | -| `|S_j| < M` cells (sum/27) | **16/27** | 4/27 | 4/27 | -| missing client-class cells | **30** | 4 | 5 | -| client-size ratio mean | **17.60** | 9.59 | 6.31 | -| mean class-support CV | **1.302** | 1.225 | 0.916 | -| union rare per seed | 6/6/8 | 6/5/6 | 6/6/6 | -| union unseen per seed | 6/5/5 | 2/0/2 | 2/2/0 | -| test rare support per seed | 141,677 / 129,824 / 147,630 | 104,843 / 13,375 / 128,995 | 28,000 / 28,000 / 28,000 | -| mean displacement `‖Cbar(1)−Cbar(0)‖` | **2.919** | 2.340 | 1.385 | -| `beta=0` class | B/B/B | B/B/B | B/B/B | - -Realized imbalance and mean displacement decrease as alpha rises; the selected beta is **not -monotone** in alpha (0.00 → 1.00 → 0.50). No causal/monotone claim (three alphas, three seeds, one -dataset). - -## 9. Direct prototype-effect findings (fixed round-0 snapshots, no retraining) - -`||Cbar_j(1) − Cbar_j(0)||`, cross-seed mean (std), 5G-NIDD alpha=0.25: - -| class | mean disp | std | -|---|---|---| -| SYNScan | 3.815 | 1.402 | -| HTTPFlood | 3.760 | 1.539 | -| TCPConnectScan | 3.258 | 2.574 | -| SlowrateDoS | 3.179 | 1.157 | -| SYNFlood | 3.176 | 1.007 | -| UDPScan | 2.816 | 1.604 | -| Benign | 2.648 | 0.519 | -| UDPFlood | 2.408 | 0.578 | -| ICMPFlood | 1.215 | 0.151 | - -Mean across classes **2.919** at beta=1.0 (monotone: 1.295 / 2.055 / 2.559 / 2.919 for beta -0.25/0.50/0.75/1.00), the largest of the three 5G-NIDD alphas (a0.50 = 2.340, a0.75 = 1.385). -Displacement is a direct-aggregation measure only, not a detection result. Outputs: -`artifacts/analysis/5gnidd__alpha0.25__multiseed__prototype_effect_pooled.csv` and per-seed -`...__prototype_effect.csv` / `...__prototype_weights.csv`. - -## 10. Client-level findings (validation, vs beta = 0) - -Detail: `...__client_validation_detail.csv`; summary: `...__client_validation_summary.csv`. - -At the selected `beta=0.00` the delta is trivially 0/0/10 in every seed. At a common -`beta=0.75`: **3/4/3** (seed 0, mean Δ −0.023), **2/5/3** (seed 1, mean −0.043), **1/4/5** (seed 2, -mean −0.052): at this low-alpha condition support weighting at beta=0.75 **degrades** the -rare-class validation metric for most clients, consistent with the pooled selection of beta=0. -`N_ij` vs ΔF1 is non-monotonic. Client ids are per-seed realisations and are not pooled. - -## 11. Runtime / resources - -CPU only, no GPU, no swap, exit 0 on all 18 runs. Per-run wall-clock 62:00–64:48; CPU 739–743 %; -peak RSS ~13.2 GiB (seed 0), ~11.9 GiB (seed 1), ~16.1 GiB (seed 2). New compute: 18 runs, -**68,798 s ≈ 19.11 h** wall-clock, sequential (8 cores). Machine-readable: -`artifacts/analysis/5gnidd__alpha0.25__multiseed__runtime.csv`. - -## 12. Cross-dataset comparison (descriptive; completed this session) - -Six-condition, validation-only pooled selections: - -| dataset | alpha=0.25 | alpha=0.50 | alpha=0.75 | -|---|---|---|---| -| 5G-NIDD | **0.00** (0.7615) | **1.00** (0.6924) | **0.50** (0.8091) | -| X-IIoTID | **0.00** (0.5917) | **0.75** (0.5794) | **0.50** (0.4767) | - -- Matched-alpha selections coincide at alpha=0.25 (both 0.00) and alpha=0.75 (both 0.50) and - differ at alpha=0.50 (1.00 vs 0.75). The selected beta takes values `{0.00, 0.50, 1.00}` across - the six conditions — **not universal, not a function of alpha alone, not monotone**. -- Realized heterogeneity is close but not identical at matched alpha (X-IIoTID has more missing - cells at alpha=0.25/0.50; 5G-NIDD has a higher client-size ratio). Dataset structural - differences and **rare-set composition differences are kept explicit**: 5G-NIDD rare classes are - low-volume flood/scan types; X-IIoTID rare classes are low-volume behavioural categories. A - "rare class" is not a comparable object across datasets. -- Mean displacement at beta=1 falls with alpha on both datasets (5G-NIDD 2.919/2.340/1.385; - X-IIoTID 3.422/2.099/1.305). -- No causality, no universal beta, no cross-dataset superiority, no monotone law. Machine-readable: - `artifacts/analysis/cross_dataset__matched_alpha_summary.csv` / `.json` and - `cross_dataset__beta_curve.csv`. Report: `docs/research/CROSS_DATASET_BETA_ANALYSIS.md`. - -## 13. Methodological decisions / assumptions - -1. **`S_j` participant/denominator rule** (`S_j = {i | N_ij > 0 and valid C_ij}`, equal weights in - the baseline) is a **REPRODUCTION ASSUMPTION**; material in **16/27** class-condition cells at - 5G-NIDD alpha=0.25 (and 19/27 at X-IIoTID alpha=0.25). Carry this caveat in all reporting. -2. `N_ij` is class-specific sample support, not prototype quality. -3. Beta is one value for all classes/clients; candidates frozen at `{0,.25,.5,.75,1}`. -4. **PRIMARY selection = pooled validation across clients and seeds**; per-seed selection is - secondary stability diagnostics; test used only after the pooled beta is fixed. -5. No new source scripts this session; analysis used the existing read-only - `analyze_partition_heterogeneity.py`, `analyze_prototype_effect.py`, `analyze_multiseed_beta.py`, - `verify_beta0_equivalence.py`, `analyze_cross_alpha.py`, plus the ad-hoc Session-11 helpers only - under `/tmp/opencode/s11/` (outside the repo). -6. `lambda=1.0`, Adam, lr 1e-3, batch 256 remain PROJECT DECISIONS. No `equal_available` mode. - -## 14. Unresolved questions / blockers - -1. `S_j` reproduction ambiguity — unresolved; no official PROTEAN code exists. -2. Pooled beta selection remains statistically weak at all six conditions; only three seeds. -3. `beta = 0` is not bitwise identical to `original_protean`; documentation decision still open. -4. Session-2 manifests still store `preprocessing_hash = null` / `config_hash = null`. -5. At both alpha=0.25 conditions the selected beta is 0, so those conditions cannot demonstrate a - support-weighting benefit. -6. No inferential trend test; selected beta differs across conditions but no monotone/causal claim. - -## 15. Git status - -No commit or push was made in Session 12. `HEAD = d30e9831e6c17fc6b28383ae4592070ccd83657e`, -branch `main`, in sync with `origin/main`. `git diff` / `git diff --cached` are empty (no tracked -modification); only untracked docs/analysis/figures artifacts exist. `planningawal/` and both raw -datasets are unchanged (SHA-256 match). `src/`, `tests/`, `configs/`, `scripts/`, `pyproject.toml` -are unchanged. `results/` is gitignored and holds exactly 108 non-smoke run directories plus 1 -`smoke`. `equal_available` absent. No test-based selection. `matplotlib 3.11.2` was installed into -the git-ignored `.venv` only; no dependency file was modified. - -## 16. Session 11 audit result - -- Verdict: **PASS** (`docs/research/SESSION11_AUDIT.md`). -- Run inventory 18/18, pairing hash-identical, manifests reproduced from raw data. -- Validation recomputed exactly; **pooled `beta_star(5gnidd, alpha=0.25) = 0.00`** (validation - only). Selected-beta test and `original_protean` test means/stds exact. -- Leakage audit PASS; `test_used_for_selection=false` upheld. -- Rare/unseen integrity PASS (0 mismatches / 1,620 cells). `beta=0` equivalence class **B**. -- Direct-effect, client-level, cross-alpha, cross-dataset labels: PASS. -- Source/data integrity PASS. pytest: **75 passed, 0 failed, 0 skipped**. - -## 17. Files created / modified (Session 12) - -Created: - -- `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md` — final six-condition synthesis, RQ1/RQ2/RQ3 - answers, threats to validity, contribution, unsupported claims, gap assessment. -- `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md` — precise Chapter-4 blueprint (purpose / artifact / - key numbers / allowed / avoided per subsection). -- `docs/thesis/CLAIMS_REGISTER.md` — claim / evidence / status / limitation / thesis-safe wording. -- `artifacts/analysis/final__six_condition_master.{csv,json}` -- `artifacts/analysis/final__pooled_validation_curves.csv` (30 rows) -- `artifacts/analysis/final__realized_heterogeneity.csv` -- `artifacts/analysis/final__prototype_displacement_by_beta.csv` -- `artifacts/analysis/final__prototype_displacement_by_class.csv` -- `artifacts/analysis/final__client_effect_summary.csv` (90 rows) -- `artifacts/analysis/final__selected_vs_original.csv` -- `artifacts/analysis/final__selection_strength.csv` -- `artifacts/figures/final/figA–G*.png` — 8 thesis-friendly figures (matplotlib only). - -Modified: - -- `docs/handoff/SESSION_HANDOFF.md` (this file). - -No source, test, config, model or aggregation file was changed. The synthesis generator is kept -outside the repository at `/tmp/opencode/s12/synthesize_final_analysis.py` (Session-11 convention -for ad-hoc analysis helpers). `matplotlib 3.11.2` was installed into the git-ignored `.venv`. - -## 18. Exact recommended next phase - -1. **The experiment grid is COMPLETE** (2 datasets × 3 alphas × 3 seeds × (5 betas + baseline) = - 108 runs). Do **not** run any new condition; no new training is required (gap classification A). -2. **Synthesis is complete and verified.** All six selections, selected/original test summaries, - heterogeneity values, displacement summaries and client counts were independently recomputed - from raw artifacts with zero mismatches. -3. Remaining work is **writing and formatting**, not experiments: - - draft Chapter 4 from `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md` using the `final__*` tables - and `figA–G` figures, respecting `docs/thesis/CLAIMS_REGISTER.md`; - - optionally add appendices with the full per-seed and per-class tables; - - if the examiners require inferential support, add a clearly post-hoc, explicitly - underpowered analysis and label it as such (not done by default). -4. Do not infer a monotone alpha→beta law; do not claim dataset causality, superiority, - generalization, or a global optimum. -5. Update `EXPERIMENT_PROTOCOL.md` only if a real ambiguity appears; update this handoff. -6. Do not commit or push unless asked. When checkpointing, stage only intended docs/analysis/ - figure artifacts; never commit `results/` or raw datasets. - -## 19. Important commands +## 13. Integrity / important commands ```bash -# Tests (75 expected) +# Tests (expected: 75 passed) .venv/bin/python -m pytest -# One full paired beta run (CPU, ~62-65 min, ~12-17 GiB for 5G-NIDD) -.venv/bin/python scripts/run_baseline.py --dataset 5gnidd --alpha 0.25 --seed 1 \ - --clients 10 --rounds 10 --local-epochs 3 \ - --aggregation-mode support_weighted --beta 1.0 +# Repository status +git status --short +git diff --stat -# Pooled multi-seed validation analysis (read-only, no retraining) +# Read-only analyses (no retraining); see docs/research/EXPERIMENT_PROTOCOL.md .venv/bin/python scripts/analyze_multiseed_beta.py --dataset 5gnidd --alpha 0.25 --seeds 0 1 2 - -# Direct prototype-effect analysis (no retraining) -.venv/bin/python scripts/analyze_prototype_effect.py \ - --snapshot artifacts/analysis/5gnidd__alpha0.25__seed0__support_weighted__beta0__prototype_snapshot.json \ - --class-names "Benign,HTTPFlood,ICMPFlood,SYNFlood,SYNScan,SlowrateDoS,TCPConnectScan,UDPFlood,UDPScan" \ - --output-dir artifacts/analysis - -# Realized partition characterization (cross-alpha) .venv/bin/python scripts/analyze_partition_heterogeneity.py --dataset 5gnidd --alphas 0.25 0.5 0.75 - -# beta=0 equivalence audit -.venv/bin/python scripts/verify_beta0_equivalence.py --dataset 5gnidd --alpha 0.25 --seeds 0 1 2 - -# Cross-alpha comparison table (5G-NIDD) .venv/bin/python scripts/analyze_cross_alpha.py --dataset 5gnidd --alphas 0.25 0.5 0.75 +.venv/bin/python scripts/verify_beta0_equivalence.py --dataset 5gnidd --alpha 0.25 --seeds 0 1 2 ``` -## 20. Recommended next-session prompt +## 14. Recommended next-session prompt ``` -You are continuing the perdalam-protean research repository (Session 13). +You are continuing the perdalam-protean research repository (Session 17). FIRST, read completely: 1. docs/handoff/SESSION_HANDOFF.md - 2. docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md - 3. docs/thesis/CHAPTER4_RESULTS_OUTLINE.md + 2. docs/proposal/THESIS_PROPOSAL_DRAFT.md + 3. docs/proposal/PROPOSAL_SOURCE_MAP.md 4. docs/thesis/CLAIMS_REGISTER.md - 5. docs/research/CROSS_DATASET_BETA_ANALYSIS.md - 6. docs/research/EXPERIMENT_PROTOCOL.md + 5. docs/thesis/CHAPTER4_RESULTS_OUTLINE.md + 6. docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md Critical rules: - - planningawal/ is immutable. - - ~/datasets/5G-NIDD/5G-NIDD.csv and ~/datasets/X-IIoTID/X-IIoTID-dataset.csv are read-only. + - planningawal/ is immutable; raw datasets under ~/datasets are read-only. - The S_j participant rule is a REPRODUCTION ASSUMPTION. - Do NOT add an executable equal_available mode. - Beta selection uses validation only; PRIMARY is pooled across clients and seeds. - - The planned 2-dataset x 3-alpha experiment grid is COMPLETE. Do NOT run new conditions. - - Scientific gap classification is A (NO MATERIAL GAP); no new experiment is required. + - The 108-run 2-dataset x 3-alpha x 3-seed grid is COMPLETE. Do NOT run new training. + - The proposal Markdown is finalized and frozen unless academic feedback requires revision. -Context: Session 12 completed the thesis synthesis with no training. Six pooled selections: -X-IIoTID 0.25/0.50/0.75 = 0.00/0.75/0.50 and 5G-NIDD = 0.00/1.00/0.50; no universal/monotone beta; -all top-two gaps below selected-beta across-seed std; beta=0 equivalence class B everywhere. -Authoritative artifacts: artifacts/analysis/final__*.{csv,json}; figures artifacts/figures/final/. -All synthesis numbers independently verified; pytest 75 passed; no tracked file changed. +Context: Session 15 finalized and committed the proposal Markdown +(docs/proposal/THESIS_PROPOSAL_DRAFT.md) at HEAD e27c5a8. Session 16 refreshed this handoff. +"Optimasi" in the title means empirical beta selection among predefined candidates, not a global +optimum. All six conditions are synthesized; no material scientific gap remains (classification A). -Task: write/format thesis chapters from the existing synthesis. Use the final__* tables and -figA-G figures, obey CLAIMS_REGISTER.md wording, keep all limitations explicit, and do not run -training or add inferential tests by default. Use .venv/bin/python. Do not commit or push unless -asked. +Task: perform the requested writing/formatting or review work using the final synthesis documents +and artifacts/analysis/final__*. Obey CLAIMS_REGISTER.md wording, keep all limitations explicit, +and do not run training or add inferential tests by default. Use .venv/bin/python. Do not commit +or push unless asked. ```