docs: refresh project handoff after proposal finalization
This commit is contained in:
1 parent
e27c5a83ab
commit
eaa9217552
1 file changed
+183
-392
+183
-392
@@ -3,438 +3,229 @@
|
||||
This is the canonical continuation point for the next agent session. It is written to be
|
||||
readable without prior chat history, terminal scrollback, or git history.
|
||||
|
||||
Last updated: 2026-09-30 (Session 12). Session 12 type: **thesis synthesis of the complete audited
|
||||
grid** — no training, no partition regeneration, no source/model/aggregation change. Session 11 ran
|
||||
the final 5G-NIDD condition (`alpha=0.25`), completing the planned grid (108 full runs + 1 smoke).
|
||||
Last updated: 2026-10-02 (Session 16). Session 16 type: **handoff refresh** — no training, no
|
||||
experiments, no source/model/aggregation change, no proposal content change.
|
||||
|
||||
> **Session 12 scope and status.** Consolidated the 108-run grid into authoritative six-condition
|
||||
> tables, publication-ready figures, a final RQ1/RQ2/RQ3 synthesis, a Chapter-4 blueprint and a
|
||||
> claims register, then independently re-verified every number against the audited artifacts.
|
||||
> **No material scientific gap remains for the defined thesis scope (classification A).** No
|
||||
> training was executed, no tracked file changed, `planningawal/` and both raw datasets are
|
||||
> unchanged, `results/` is unchanged (108 non-smoke run dirs), and pytest is **75 passed**.
|
||||
> Deliverables: `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md`,
|
||||
> `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md`, `docs/thesis/CLAIMS_REGISTER.md`,
|
||||
> `artifacts/analysis/final__*.{csv,json}`, `artifacts/figures/final/figA–G*.png`.
|
||||
|
||||
> **Final RQ findings (Session 12).** (1) Changing beta changes contribution weights and prototype
|
||||
> positions; for a fixed snapshot displacement is monotone increasing in beta in all six
|
||||
> conditions, and realized support imbalance and displacement co-vary within each dataset. (2) The
|
||||
> pooled validation rare-class response is **not monotone** in beta and is weak relative to seed
|
||||
> variance; the direct aggregation effect and the end-to-end training effect remain distinct.
|
||||
> (3) Selected betas are condition-dependent and non-monotone in alpha:
|
||||
> X-IIoTID `{0.00, 0.75, 0.50}` and 5G-NIDD `{0.00, 1.00, 0.50}` for alpha `{0.25, 0.50, 0.75}`;
|
||||
> both datasets select beta=0 at alpha=0.25 (no weighting benefit) and beta=0.50 at alpha=0.75;
|
||||
> no universal/global optimum. All six top-two selection gaps are smaller than the selected beta's
|
||||
> across-seed std. No inferential test was added (3 seeds).
|
||||
|
||||
> **Session 11 audit (preserved).** Verdict **PASS** (`docs/research/SESSION11_AUDIT.md`);
|
||||
> `beta_star(5gnidd, alpha=0.25) = 0.00` validation-only (per-seed 0.25/0.00/0.00); run inventory
|
||||
> 18/18; pairing hash-identical; rare/unseen 0 mismatches; leakage PASS; `beta=0` class **B**;
|
||||
> source/data integrity PASS; pytest 75 passed. Realized heterogeneity highest of the three 5G-NIDD
|
||||
> conditions (16/27 cells, 30 missing, client ratio 17.60, mean CV 1.302).
|
||||
> **Current state in one line.** The 108-run experimental grid is complete and synthesized; the
|
||||
> thesis proposal Markdown has been finalized and committed. Remaining work is academic writing
|
||||
> and, later, formatting into the university template — not new experiments.
|
||||
|
||||
---
|
||||
|
||||
## 1. Project objective
|
||||
## 1. Final research scope
|
||||
|
||||
Build the code and experiment infrastructure for a thesis study that modifies **only** the
|
||||
server-side prototype aggregation of PROTEAN (Chennoufi et al., 2025), a federated
|
||||
intrusion-detection framework. The mechanism weights each contributing client's class prototype
|
||||
by its class sample support:
|
||||
The thesis studies exactly one principal method, one component and one tunable mechanism:
|
||||
|
||||
- principal method: **PROTEAN** (Chennoufi et al., 2025), a federated intrusion-detection
|
||||
framework;
|
||||
- component studied: **server-side prototype aggregation** (only this component is modified);
|
||||
- tunable mechanism: **beta-controlled class-sample-support weighting** of each client's local
|
||||
class prototype;
|
||||
- analysis: **rare-class** and **per-client** behavior;
|
||||
- datasets: **X-IIoTID** and **5G-NIDD**;
|
||||
- non-IID conditions: symmetric Dirichlet **alpha = 0.75 / 0.50 / 0.25**;
|
||||
- beta candidates: **{0; 0.25; 0.50; 0.75; 1.00}**, one beta for all classes and clients.
|
||||
|
||||
The aggregation is:
|
||||
|
||||
```
|
||||
Cbar_j(beta) = ( sum_{i in S_j} N_ij^beta * C_ij ) / ( sum_{i in S_j} N_ij^beta )
|
||||
S_j = { i | N_ij > 0 and a valid local prototype C_ij exists }
|
||||
beta in {0.0, 0.25, 0.5, 0.75, 1.0}
|
||||
```
|
||||
|
||||
Both the frozen `original_protean` baseline and the proposed `support_weighted` mode are
|
||||
implemented. `beta = 0` is mathematically the equal mean over the shared `S_j` but is not
|
||||
bitwise identical to the baseline (see §7).
|
||||
where `C_ij` is client `i`'s local class-`j` prototype and `N_ij` is client `i`'s local training
|
||||
sample count for class `j`.
|
||||
|
||||
## 2. Authoritative source and immutability
|
||||
## 2. Final title
|
||||
|
||||
**Optimasi Agregasi Prototype pada PROTEAN melalui Pembobotan Berbasis Dukungan Sampel Kelas
|
||||
untuk Deteksi Serangan Langka pada Data Non-IID**
|
||||
|
||||
English subtitle: *Optimizing Prototype Aggregation in PROTEAN through Class-Sample-Support
|
||||
Weighting for Rare Attack Detection under Non-IID Data Distributions*.
|
||||
|
||||
**Meaning of "Optimasi" (explicit):** empirical tuning/selection of beta among the predefined
|
||||
discrete candidates using validation data. It does **not** mean a global mathematical optimum and
|
||||
does **not** imply a universal beta. This clarification appears in the proposal header note, in
|
||||
§1.7, and in §3.8. The title matches the immutable research specification
|
||||
`planningawal/PROTEAN_Revisi_Lurus.md`.
|
||||
|
||||
## 3. Proposal status (authoritative proposal files)
|
||||
|
||||
- `docs/proposal/THESIS_PROPOSAL_DRAFT.md` — **the final Markdown proposal source for the current
|
||||
scope.** Written in Indonesian; BAB I–III plus a short preliminary-study section and a
|
||||
bibliography.
|
||||
- `docs/proposal/PROPOSAL_SOURCE_MAP.md` — internal audit aid mapping every substantive proposal
|
||||
claim to PAPER / SPEC / PROJECT / PRELIM / FINAL evidence.
|
||||
- `docs/proposal/LITERATURE_MATRIX.md` — internal literature matrix (categories A–H, per-reference
|
||||
role, limitations, metadata confidence).
|
||||
- `docs/proposal/REFERENCES_VERIFIED.md` — internal verification record for every bibliography
|
||||
entry.
|
||||
|
||||
The proposal was finalized and committed in Session 15 (`docs: finalize thesis proposal markdown`).
|
||||
It is **frozen** unless academic/supervisor feedback requires revision. It is written prospectively
|
||||
(no final experimental results exposed; only feasibility-level preliminary statements).
|
||||
|
||||
## 4. Experimental completion
|
||||
|
||||
- **108 full runs complete**: 2 datasets × 3 alpha values × 3 seeds × (1 `original_protean`
|
||||
baseline + 5 beta settings). One unrelated `smoke` directory is excluded.
|
||||
- All six `(dataset, alpha)` conditions are trained, audited and **synthesized**.
|
||||
- No further training is planned **unless new supervisor/academic feedback explicitly requires
|
||||
it**. The scientific-gap classification for the defined scope is **A — NO MATERIAL GAP**.
|
||||
- Full result tables are deliberately **not** reproduced here; load them from the synthesis
|
||||
documents and `artifacts/analysis/final__*` instead.
|
||||
|
||||
## 5. Important final research documents
|
||||
|
||||
- `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md` — authoritative six-condition synthesis
|
||||
(RQ1/RQ2/RQ3 answers, threats to validity, contribution, unsupported claims, gap assessment).
|
||||
- `docs/research/CROSS_DATASET_BETA_ANALYSIS.md` — cross-dataset comparison.
|
||||
- `docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md` — X-IIoTID cross-alpha analysis.
|
||||
- `docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md` — 5G-NIDD cross-alpha analysis.
|
||||
- `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md` — precise Chapter-4 blueprint (allowed/avoided
|
||||
interpretation per subsection).
|
||||
- `docs/thesis/CLAIMS_REGISTER.md` — claim / evidence / status / limitation / thesis-safe wording.
|
||||
- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen reproduction protocol.
|
||||
|
||||
## 6. Claims / interpretation constraints
|
||||
|
||||
These constraints must be respected in every future text (proposal, chapters, slides):
|
||||
|
||||
1. `N_ij` is **class-specific sample support, not prototype quality**.
|
||||
2. Support weighting is **not inherently novel**; sample-count prototype weighting is already
|
||||
discussed in related work (e.g. FedProto).
|
||||
3. **No claim that support weighting always improves detection.** Response is condition-dependent
|
||||
and non-monotone.
|
||||
4. **No universal/global-optimum beta claim.** The selected beta is "best among the tested
|
||||
candidates under the defined pooled-validation procedure".
|
||||
5. **No causal alpha -> beta claim.** Selection is not monotone in alpha; only three alpha levels,
|
||||
two datasets, three seeds.
|
||||
6. **No claim that larger prototype displacement means better detection.** Displacement is a
|
||||
representation-aggregation metric, not detection quality.
|
||||
7. `beta = 0` is **semantically equivalent** to the equal mean over the shared `S_j`, but is
|
||||
**not guaranteed bitwise identical** to `original_protean` (floating-point divergence;
|
||||
equivalence class **B** in every condition).
|
||||
8. **Test data is never used for beta selection.**
|
||||
9. **Primary selection = pooled validation across clients and seeds; per-seed selection is
|
||||
diagnostic only.**
|
||||
10. **`S_j` participant/denominator handling remains a documented reproduction assumption** (no
|
||||
verified official PROTEAN code); all results are conditional on it.
|
||||
|
||||
## 7. Known open academic items (all non-blocking)
|
||||
|
||||
1. **Zhao et al. (2018)** remains an arXiv/preprint reference; it is flagged in the bibliography as
|
||||
**[metadata perlu diverifikasi]** (no peer-reviewed venue asserted).
|
||||
2. **1–2 additional peer-reviewed IDS-specific rare/class-imbalance references** could be added
|
||||
later if needed; this is an identified literature gap, not an error.
|
||||
3. **FedProto sample-count weighting attribution** should be rechecked against the primary paper
|
||||
before defense if necessary (the claim is inherited from the immutable spec and is conservative
|
||||
for novelty).
|
||||
4. **`S_j` reproduction assumption** remains documented and unresolved; carry the caveat in all
|
||||
downstream reporting.
|
||||
|
||||
## 8. Research evolution (brief)
|
||||
|
||||
- The earlier scope was broad: Differential Privacy plus multiple rare-class preservation
|
||||
mechanisms.
|
||||
- Supervisor feedback pushed the study toward a single, interpretable mechanism.
|
||||
- The final scope isolates PROTEAN's server-side prototype aggregation and tunes `beta`.
|
||||
- This avoids a whole-method A/B/C comparison and enables mechanism-level analysis of the chain:
|
||||
support imbalance -> contribution weight -> global-prototype displacement -> rare-class
|
||||
detection -> per-client impact.
|
||||
|
||||
## 9. Next recommended work
|
||||
|
||||
1. **Keep the proposal Markdown frozen** unless academic/supervisor feedback explicitly requires
|
||||
revision.
|
||||
2. **Later, convert the finalized Markdown to DOCX** using the university template (out of scope
|
||||
for the current sessions).
|
||||
3. **Later, draft thesis chapters** (starting with Chapter 4) from the final synthesis documents and
|
||||
`artifacts/analysis/final__*` + `artifacts/figures/final/`, obeying `CLAIMS_REGISTER.md` and
|
||||
`CHAPTER4_RESULTS_OUTLINE.md`.
|
||||
4. **No more experiments** unless required by new academic feedback.
|
||||
|
||||
---
|
||||
|
||||
## 10. Authoritative source and immutability
|
||||
|
||||
- `planningawal/PROTEAN_Revisi_Lurus.md` — authoritative research specification.
|
||||
- `planningawal/` is immutable. Re-verified unchanged this session: `git status --porcelain
|
||||
planningawal/` empty.
|
||||
- Raw datasets re-verified unchanged (SHA-256):
|
||||
`~/datasets/5G-NIDD/5G-NIDD.csv`
|
||||
`fa36f80859585f474504ca69eb951a079b35145537976c86b00aae3aab46ee59`;
|
||||
`~/datasets/X-IIoTID/X-IIoTID-dataset.csv`
|
||||
`7b9290057ee42e784da3c0d84b781815502c9205c74175c96374e71a5ffd98a0`.
|
||||
- Frozen baseline (`original_protean`) source unchanged; no `equal_available` mode added; beta
|
||||
candidate set unchanged (`{0, .25, .5, .75, 1}`). No tracked source/test/config file differs
|
||||
from HEAD. Only new docs/analysis artifacts are untracked.
|
||||
- `planningawal/` is **immutable**. Re-verified unchanged: `git status --porcelain planningawal/`
|
||||
is empty.
|
||||
- Raw datasets are external and read-only; do not modify, rename, move, copy or commit them:
|
||||
- `~/datasets/5G-NIDD/5G-NIDD.csv` (SHA-256
|
||||
`fa36f80859585f474504ca69eb951a079b35145537976c86b00aae3aab46ee59`)
|
||||
- `~/datasets/X-IIoTID/X-IIoTID-dataset.csv` (SHA-256
|
||||
`7b9290057ee42e784da3c0d84b781815502c9205c74175c96374e71a5ffd98a0`)
|
||||
|
||||
## 3. Session 11 objective and outcome
|
||||
## 11. Methodological decisions / assumptions
|
||||
|
||||
Objective: run exactly the **final** condition — 5G-NIDD `alpha=0.25`, seeds `{0,1,2}`,
|
||||
`original_protean` + five betas — after a Stage-A verification gate; then build the 5G-NIDD
|
||||
three-alpha comparison and the cross-dataset comparison. Do not run anything after it, do not
|
||||
change methodology.
|
||||
1. `S_j = { i | N_ij > 0 and a valid C_ij exists }` with equal `1/|S_j|` weights in the baseline
|
||||
is a **REPRODUCTION ASSUMPTION** (materially exercised in up to 19/27 class-condition cells).
|
||||
2. `N_ij` is sample support, not prototype quality.
|
||||
3. One beta is used for all classes/clients; candidates are frozen at
|
||||
`{0, 0.25, 0.50, 0.75, 1.00}`.
|
||||
4. Primary selection = pooled validation across clients and seeds; per-seed selection is secondary
|
||||
(stability diagnostics); test is read only after the pooled beta is fixed.
|
||||
5. No `equal_available` executable mode was added (it is arithmetically `support_weighted` at
|
||||
`beta = 0` under the shared `S_j`).
|
||||
6. `lambda = 1.0`, Adam, `lr = 1e-3`, batch 256, 10 clients, 10 rounds, 3 local epochs remain
|
||||
project decisions; see `docs/research/EXPERIMENT_PROTOCOL.md` for the full frozen protocol.
|
||||
|
||||
Stage A result: raw hash valid; all three 5G-NIDD `alpha=0.25` manifests reproduce exactly from raw
|
||||
data (support match, partition-hash match, split-hash match); label semantics and rare/unseen logic
|
||||
valid; pairing framework valid. Gate = **GO**.
|
||||
## 12. Git state
|
||||
|
||||
Outcome: 18/18 runs rc=0. **PRIMARY pooled-validation selected beta = 0.00** (validation only);
|
||||
`test_used_for_selection = false` for all 15 runs. Pooled means: 0.00 → 0.761453, 0.25 → 0.749286,
|
||||
0.50 → 0.725715, 0.75 → 0.722235, 1.00 → 0.737636. Reports:
|
||||
`docs/research/5GNIDD_ALPHA025_BETA_VALIDATION.md`, `docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md`,
|
||||
`docs/research/CROSS_DATASET_BETA_ANALYSIS.md`, audit `docs/research/SESSION11_AUDIT.md`.
|
||||
- Branch `main`, `HEAD = e27c5a83abcb84ab31c667b4d39d580012fa6f26`, in sync with `origin/main`.
|
||||
- Session 15 commit: `docs: finalize thesis proposal markdown` (four proposal files).
|
||||
- No tracked working-tree changes expected between sessions; only pre-existing untracked
|
||||
`artifacts/analysis/*prototype_snapshot.json` files (94) remain untracked and must not be staged
|
||||
accidentally.
|
||||
- `results/` is git-ignored. Do not commit raw datasets, large artifacts, or `results/`.
|
||||
|
||||
## 4. Condition and pairing
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| dataset / alpha / seeds | `5gnidd` / 0.25 / `{0,1,2}` |
|
||||
| runs | 15 `support_weighted` + 3 `original_protean`, all new this session |
|
||||
| training | 10 clients, 10 rounds, 3 local epochs, batch 256, lr 1e-3, mu 0.1, lambda 1.0, Adam |
|
||||
| code revision | only docs/analysis added; HEAD `bbd35085063194db58e4472e70b204d4be4a3607` |
|
||||
|
||||
Manifests (reused, never regenerated; one content hash per seed across all six runs; each
|
||||
reproduced from the raw dataset with split/`N_ij`/partition hash verified):
|
||||
|
||||
| seed | manifest content hash | partition hash | split hash | rng seed | round-0 snapshot SHA-256 |
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | `ab6d8a8b63d478c4…` | `e628c17369f25e35…` | `18f89f4807de5493…` | 708356900 | `e2243ac4ed4391ff…` |
|
||||
| 1 | `74df102e4edb4509…` | `05aec12ceb4c94bd…` | `3d05b7fe48d5a06c…` | 902143161 | `26a162991b7cbde9…` |
|
||||
| 2 | `061c094d1913f942…` | `d9631ea6b2d3907e…` | `fc758d060dc51dec…` | 218373793 | `e287bff5ca71f8d2…` |
|
||||
|
||||
Splits: train/validation/test = 851,123 / 121,589 / 243,177 for every seed (10 clients, 9 classes,
|
||||
`Benign`). `train_support_Nij.csv`, `characterization.csv`, `rare_unseen.csv` identical across each
|
||||
seed's runs. `manifest_reload_ok = true`, `smoke = false` everywhere. Pairing proven by content
|
||||
hashes.
|
||||
|
||||
## 5. Validation results (USED FOR SELECTION) and pooled selection
|
||||
|
||||
Primary metric: validation per-client-mean rare-class macro-F1. All 10 clients eligible in every
|
||||
seed. Full tables: `artifacts/analysis/5gnidd__alpha0.25__multiseed__validation_per_seed.csv`.
|
||||
|
||||
Per-seed validation per-client-mean rare-class macro-F1:
|
||||
|
||||
| beta | seed 0 | seed 1 | seed 2 |
|
||||
|---|---|---|---|
|
||||
| 0.00 | 0.736360 | **0.771268** | **0.776732** |
|
||||
| 0.25 | **0.747299** | 0.767300 | 0.733259 |
|
||||
| 0.50 | 0.717450 | 0.738223 | 0.721471 |
|
||||
| 0.75 | 0.713262 | 0.728478 | 0.724966 |
|
||||
| 1.00 | 0.734455 | 0.739297 | 0.739157 |
|
||||
|
||||
Pooled across seeds (`...__validation_pooled.csv`):
|
||||
|
||||
| beta | pooled mean | std (sample) | min | max | selected |
|
||||
|---|---|---|---|---|---|
|
||||
| **0.00** | **0.761453** | 0.021903 | 0.736360 | 0.776732 | **yes** |
|
||||
| 0.25 | 0.749286 | 0.017107 | 0.733259 | 0.767300 | |
|
||||
| 0.50 | 0.725715 | 0.011017 | 0.717450 | 0.738223 | |
|
||||
| 0.75 | 0.722235 | 0.007967 | 0.713262 | 0.728478 | |
|
||||
| 1.00 | 0.737636 | 0.002756 | 0.734455 | 0.739297 | |
|
||||
|
||||
- **PRIMARY selected beta (`beta_star_5gnidd_alpha025`) = 0.00**, max pooled validation
|
||||
per-client-mean rare-class macro-F1; **no tie-break used**; validation-only. Top-two gap ≈ 1.2e-2.
|
||||
- **SECONDARY per-seed diagnostics only:** seed 0 → 0.25, seed 1 → 0.00, seed 2 → 0.00.
|
||||
- Between-beta spread ≈ 0.039; between-seed std 0.003–0.022. At this condition the spread exceeds
|
||||
the seed noise, but the sample is only 3 project-defined seeds.
|
||||
|
||||
## 6. Final test evaluation of the pooled-selected beta
|
||||
|
||||
`beta_star = 0.00` was fixed from pooled validation before test was read.
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.859893 | 0.813889 | 0.849925 | 0.841236 ± 0.024202 |
|
||||
| rare-class recall | 0.985234 | 0.972659 | 0.925756 | 0.961216 ± 0.031347 |
|
||||
| macro-F1 | 0.829680 | 0.802987 | 0.817254 | 0.816640 ± 0.013357 |
|
||||
| accuracy | 0.744844 | 0.737779 | 0.708052 | 0.730225 ± 0.019525 |
|
||||
| weighted-F1 | 0.717404 | 0.712302 | 0.683684 | 0.704463 ± 0.018175 |
|
||||
|
||||
`original_protean` for the same seeds/condition:
|
||||
|
||||
| metric (test) | seed 0 | seed 1 | seed 2 | mean ± std (sample) |
|
||||
|---|---|---|---|---|
|
||||
| rare-class macro-F1 | 0.864903 | 0.814848 | 0.847072 | 0.842274 ± 0.025370 |
|
||||
| rare-class recall | 0.985444 | 0.972333 | 0.923890 | 0.960556 ± 0.032423 |
|
||||
| macro-F1 | 0.835043 | 0.808373 | 0.814800 | 0.819405 ± 0.013919 |
|
||||
| accuracy | 0.745280 | 0.741295 | 0.706329 | 0.730968 ± 0.021431 |
|
||||
| weighted-F1 | 0.718041 | 0.716327 | 0.681939 | 0.705435 ± 0.020367 |
|
||||
|
||||
Descriptive difference for test rare-class macro-F1 (`beta_star` − `original_protean`) is
|
||||
−0.005010 / −0.000959 / +0.002853 (mean −0.001039). Because beta_star = 0, this difference is just
|
||||
floating-point divergence (see §7). The global rare evaluation support differs by seed
|
||||
(**141,677 / 129,824 / 147,630**), so cross-seed test levels are **not** directly comparable at
|
||||
this condition. Test weighted-F1 (`≈0.704`) again sits **below** accuracy (`≈0.730`), as at
|
||||
alpha=0.50/0.75.
|
||||
|
||||
## 7. `beta = 0` equivalence result (all three seeds)
|
||||
|
||||
For seeds 0, 1, 2, `support_weighted beta=0` vs frozen `original_protean`:
|
||||
|
||||
- Same manifest/split/hashes, round-0 snapshot byte-identical, support/char/rare_unseen
|
||||
identical: **yes** for all three.
|
||||
- Independent aggregation (`ΣC/n` vs `(1/n)ΣC`): 4.4e-16 / 4.4e-16 / 6.7e-16.
|
||||
- History rare-macro-F1 max diff (rounds ≥ 1): 2.17e-2 / 6.85e-2 / 2.74e-2; test metric max diff
|
||||
5.36e-3 / 5.39e-3 / 2.85e-3.
|
||||
- Classification: **B** in all seeds — semantic equivalence + floating-point divergence; no
|
||||
semantic discrepancy. Machine-readable:
|
||||
`artifacts/analysis/5gnidd__alpha0.25__multiseed__beta0_equivalence.csv`.
|
||||
|
||||
Conditional on the shared `S_j` reproduction assumption; not proof `beta=0` reproduces original
|
||||
PROTEAN. Since the selected beta is 0, the selected-run test metrics differ from original only by
|
||||
this divergence.
|
||||
|
||||
## 8. Partition characterization (realized) and comparison with alpha=0.50/0.75
|
||||
|
||||
5G-NIDD `alpha=0.25` (training split), `M = 10`:
|
||||
|
||||
| seed | `|S_j| < M` cells | missing client-class cells | client-size ratio | union rare | union unseen | max unseen/client |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 0 | 6 / 9 | 10 | 9.93 | 6 | 6 | 3 |
|
||||
| 1 | 5 / 9 | 10 | 27.83 | 6 | 5 | 3 |
|
||||
| 2 | 5 / 9 | 10 | 15.05 | 8 | 5 | 2 |
|
||||
| **sum/mean** | **16 / 27** | **30** | mean **17.60** | 6/6/8 | 6/5/5 | 3 |
|
||||
|
||||
Mean class-support CV = **1.302**; max per-class `max/min` = 185,504. All 10 clients have exactly
|
||||
two rare classes and are eligible. The rare-class union **differs across seeds** (6/6/8), so test
|
||||
rare support is not constant (141,677 / 129,824 / 147,630). This is the most heterogeneous of the
|
||||
three 5G-NIDD conditions. Full tables:
|
||||
`artifacts/analysis/5gnidd__cross_alpha__partition_heterogeneity_*` and
|
||||
`...__partition_characterization.csv`.
|
||||
|
||||
### 5G-NIDD three-alpha comparison
|
||||
|
||||
| statistic | a0.25 | a0.50 | a0.75 |
|
||||
|---|---|---|---|
|
||||
| pooled-selected beta | **0.00** | **1.00** | **0.50** |
|
||||
| `|S_j| < M` cells (sum/27) | **16/27** | 4/27 | 4/27 |
|
||||
| missing client-class cells | **30** | 4 | 5 |
|
||||
| client-size ratio mean | **17.60** | 9.59 | 6.31 |
|
||||
| mean class-support CV | **1.302** | 1.225 | 0.916 |
|
||||
| union rare per seed | 6/6/8 | 6/5/6 | 6/6/6 |
|
||||
| union unseen per seed | 6/5/5 | 2/0/2 | 2/2/0 |
|
||||
| test rare support per seed | 141,677 / 129,824 / 147,630 | 104,843 / 13,375 / 128,995 | 28,000 / 28,000 / 28,000 |
|
||||
| mean displacement `‖Cbar(1)−Cbar(0)‖` | **2.919** | 2.340 | 1.385 |
|
||||
| `beta=0` class | B/B/B | B/B/B | B/B/B |
|
||||
|
||||
Realized imbalance and mean displacement decrease as alpha rises; the selected beta is **not
|
||||
monotone** in alpha (0.00 → 1.00 → 0.50). No causal/monotone claim (three alphas, three seeds, one
|
||||
dataset).
|
||||
|
||||
## 9. Direct prototype-effect findings (fixed round-0 snapshots, no retraining)
|
||||
|
||||
`||Cbar_j(1) − Cbar_j(0)||`, cross-seed mean (std), 5G-NIDD alpha=0.25:
|
||||
|
||||
| class | mean disp | std |
|
||||
|---|---|---|
|
||||
| SYNScan | 3.815 | 1.402 |
|
||||
| HTTPFlood | 3.760 | 1.539 |
|
||||
| TCPConnectScan | 3.258 | 2.574 |
|
||||
| SlowrateDoS | 3.179 | 1.157 |
|
||||
| SYNFlood | 3.176 | 1.007 |
|
||||
| UDPScan | 2.816 | 1.604 |
|
||||
| Benign | 2.648 | 0.519 |
|
||||
| UDPFlood | 2.408 | 0.578 |
|
||||
| ICMPFlood | 1.215 | 0.151 |
|
||||
|
||||
Mean across classes **2.919** at beta=1.0 (monotone: 1.295 / 2.055 / 2.559 / 2.919 for beta
|
||||
0.25/0.50/0.75/1.00), the largest of the three 5G-NIDD alphas (a0.50 = 2.340, a0.75 = 1.385).
|
||||
Displacement is a direct-aggregation measure only, not a detection result. Outputs:
|
||||
`artifacts/analysis/5gnidd__alpha0.25__multiseed__prototype_effect_pooled.csv` and per-seed
|
||||
`...__prototype_effect.csv` / `...__prototype_weights.csv`.
|
||||
|
||||
## 10. Client-level findings (validation, vs beta = 0)
|
||||
|
||||
Detail: `...__client_validation_detail.csv`; summary: `...__client_validation_summary.csv`.
|
||||
|
||||
At the selected `beta=0.00` the delta is trivially 0/0/10 in every seed. At a common
|
||||
`beta=0.75`: **3/4/3** (seed 0, mean Δ −0.023), **2/5/3** (seed 1, mean −0.043), **1/4/5** (seed 2,
|
||||
mean −0.052): at this low-alpha condition support weighting at beta=0.75 **degrades** the
|
||||
rare-class validation metric for most clients, consistent with the pooled selection of beta=0.
|
||||
`N_ij` vs ΔF1 is non-monotonic. Client ids are per-seed realisations and are not pooled.
|
||||
|
||||
## 11. Runtime / resources
|
||||
|
||||
CPU only, no GPU, no swap, exit 0 on all 18 runs. Per-run wall-clock 62:00–64:48; CPU 739–743 %;
|
||||
peak RSS ~13.2 GiB (seed 0), ~11.9 GiB (seed 1), ~16.1 GiB (seed 2). New compute: 18 runs,
|
||||
**68,798 s ≈ 19.11 h** wall-clock, sequential (8 cores). Machine-readable:
|
||||
`artifacts/analysis/5gnidd__alpha0.25__multiseed__runtime.csv`.
|
||||
|
||||
## 12. Cross-dataset comparison (descriptive; completed this session)
|
||||
|
||||
Six-condition, validation-only pooled selections:
|
||||
|
||||
| dataset | alpha=0.25 | alpha=0.50 | alpha=0.75 |
|
||||
|---|---|---|---|
|
||||
| 5G-NIDD | **0.00** (0.7615) | **1.00** (0.6924) | **0.50** (0.8091) |
|
||||
| X-IIoTID | **0.00** (0.5917) | **0.75** (0.5794) | **0.50** (0.4767) |
|
||||
|
||||
- Matched-alpha selections coincide at alpha=0.25 (both 0.00) and alpha=0.75 (both 0.50) and
|
||||
differ at alpha=0.50 (1.00 vs 0.75). The selected beta takes values `{0.00, 0.50, 1.00}` across
|
||||
the six conditions — **not universal, not a function of alpha alone, not monotone**.
|
||||
- Realized heterogeneity is close but not identical at matched alpha (X-IIoTID has more missing
|
||||
cells at alpha=0.25/0.50; 5G-NIDD has a higher client-size ratio). Dataset structural
|
||||
differences and **rare-set composition differences are kept explicit**: 5G-NIDD rare classes are
|
||||
low-volume flood/scan types; X-IIoTID rare classes are low-volume behavioural categories. A
|
||||
"rare class" is not a comparable object across datasets.
|
||||
- Mean displacement at beta=1 falls with alpha on both datasets (5G-NIDD 2.919/2.340/1.385;
|
||||
X-IIoTID 3.422/2.099/1.305).
|
||||
- No causality, no universal beta, no cross-dataset superiority, no monotone law. Machine-readable:
|
||||
`artifacts/analysis/cross_dataset__matched_alpha_summary.csv` / `.json` and
|
||||
`cross_dataset__beta_curve.csv`. Report: `docs/research/CROSS_DATASET_BETA_ANALYSIS.md`.
|
||||
|
||||
## 13. Methodological decisions / assumptions
|
||||
|
||||
1. **`S_j` participant/denominator rule** (`S_j = {i | N_ij > 0 and valid C_ij}`, equal weights in
|
||||
the baseline) is a **REPRODUCTION ASSUMPTION**; material in **16/27** class-condition cells at
|
||||
5G-NIDD alpha=0.25 (and 19/27 at X-IIoTID alpha=0.25). Carry this caveat in all reporting.
|
||||
2. `N_ij` is class-specific sample support, not prototype quality.
|
||||
3. Beta is one value for all classes/clients; candidates frozen at `{0,.25,.5,.75,1}`.
|
||||
4. **PRIMARY selection = pooled validation across clients and seeds**; per-seed selection is
|
||||
secondary stability diagnostics; test used only after the pooled beta is fixed.
|
||||
5. No new source scripts this session; analysis used the existing read-only
|
||||
`analyze_partition_heterogeneity.py`, `analyze_prototype_effect.py`, `analyze_multiseed_beta.py`,
|
||||
`verify_beta0_equivalence.py`, `analyze_cross_alpha.py`, plus the ad-hoc Session-11 helpers only
|
||||
under `/tmp/opencode/s11/` (outside the repo).
|
||||
6. `lambda=1.0`, Adam, lr 1e-3, batch 256 remain PROJECT DECISIONS. No `equal_available` mode.
|
||||
|
||||
## 14. Unresolved questions / blockers
|
||||
|
||||
1. `S_j` reproduction ambiguity — unresolved; no official PROTEAN code exists.
|
||||
2. Pooled beta selection remains statistically weak at all six conditions; only three seeds.
|
||||
3. `beta = 0` is not bitwise identical to `original_protean`; documentation decision still open.
|
||||
4. Session-2 manifests still store `preprocessing_hash = null` / `config_hash = null`.
|
||||
5. At both alpha=0.25 conditions the selected beta is 0, so those conditions cannot demonstrate a
|
||||
support-weighting benefit.
|
||||
6. No inferential trend test; selected beta differs across conditions but no monotone/causal claim.
|
||||
|
||||
## 15. Git status
|
||||
|
||||
No commit or push was made in Session 12. `HEAD = d30e9831e6c17fc6b28383ae4592070ccd83657e`,
|
||||
branch `main`, in sync with `origin/main`. `git diff` / `git diff --cached` are empty (no tracked
|
||||
modification); only untracked docs/analysis/figures artifacts exist. `planningawal/` and both raw
|
||||
datasets are unchanged (SHA-256 match). `src/`, `tests/`, `configs/`, `scripts/`, `pyproject.toml`
|
||||
are unchanged. `results/` is gitignored and holds exactly 108 non-smoke run directories plus 1
|
||||
`smoke`. `equal_available` absent. No test-based selection. `matplotlib 3.11.2` was installed into
|
||||
the git-ignored `.venv` only; no dependency file was modified.
|
||||
|
||||
## 16. Session 11 audit result
|
||||
|
||||
- Verdict: **PASS** (`docs/research/SESSION11_AUDIT.md`).
|
||||
- Run inventory 18/18, pairing hash-identical, manifests reproduced from raw data.
|
||||
- Validation recomputed exactly; **pooled `beta_star(5gnidd, alpha=0.25) = 0.00`** (validation
|
||||
only). Selected-beta test and `original_protean` test means/stds exact.
|
||||
- Leakage audit PASS; `test_used_for_selection=false` upheld.
|
||||
- Rare/unseen integrity PASS (0 mismatches / 1,620 cells). `beta=0` equivalence class **B**.
|
||||
- Direct-effect, client-level, cross-alpha, cross-dataset labels: PASS.
|
||||
- Source/data integrity PASS. pytest: **75 passed, 0 failed, 0 skipped**.
|
||||
|
||||
## 17. Files created / modified (Session 12)
|
||||
|
||||
Created:
|
||||
|
||||
- `docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md` — final six-condition synthesis, RQ1/RQ2/RQ3
|
||||
answers, threats to validity, contribution, unsupported claims, gap assessment.
|
||||
- `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md` — precise Chapter-4 blueprint (purpose / artifact /
|
||||
key numbers / allowed / avoided per subsection).
|
||||
- `docs/thesis/CLAIMS_REGISTER.md` — claim / evidence / status / limitation / thesis-safe wording.
|
||||
- `artifacts/analysis/final__six_condition_master.{csv,json}`
|
||||
- `artifacts/analysis/final__pooled_validation_curves.csv` (30 rows)
|
||||
- `artifacts/analysis/final__realized_heterogeneity.csv`
|
||||
- `artifacts/analysis/final__prototype_displacement_by_beta.csv`
|
||||
- `artifacts/analysis/final__prototype_displacement_by_class.csv`
|
||||
- `artifacts/analysis/final__client_effect_summary.csv` (90 rows)
|
||||
- `artifacts/analysis/final__selected_vs_original.csv`
|
||||
- `artifacts/analysis/final__selection_strength.csv`
|
||||
- `artifacts/figures/final/figA–G*.png` — 8 thesis-friendly figures (matplotlib only).
|
||||
|
||||
Modified:
|
||||
|
||||
- `docs/handoff/SESSION_HANDOFF.md` (this file).
|
||||
|
||||
No source, test, config, model or aggregation file was changed. The synthesis generator is kept
|
||||
outside the repository at `/tmp/opencode/s12/synthesize_final_analysis.py` (Session-11 convention
|
||||
for ad-hoc analysis helpers). `matplotlib 3.11.2` was installed into the git-ignored `.venv`.
|
||||
|
||||
## 18. Exact recommended next phase
|
||||
|
||||
1. **The experiment grid is COMPLETE** (2 datasets × 3 alphas × 3 seeds × (5 betas + baseline) =
|
||||
108 runs). Do **not** run any new condition; no new training is required (gap classification A).
|
||||
2. **Synthesis is complete and verified.** All six selections, selected/original test summaries,
|
||||
heterogeneity values, displacement summaries and client counts were independently recomputed
|
||||
from raw artifacts with zero mismatches.
|
||||
3. Remaining work is **writing and formatting**, not experiments:
|
||||
- draft Chapter 4 from `docs/thesis/CHAPTER4_RESULTS_OUTLINE.md` using the `final__*` tables
|
||||
and `figA–G` figures, respecting `docs/thesis/CLAIMS_REGISTER.md`;
|
||||
- optionally add appendices with the full per-seed and per-class tables;
|
||||
- if the examiners require inferential support, add a clearly post-hoc, explicitly
|
||||
underpowered analysis and label it as such (not done by default).
|
||||
4. Do not infer a monotone alpha→beta law; do not claim dataset causality, superiority,
|
||||
generalization, or a global optimum.
|
||||
5. Update `EXPERIMENT_PROTOCOL.md` only if a real ambiguity appears; update this handoff.
|
||||
6. Do not commit or push unless asked. When checkpointing, stage only intended docs/analysis/
|
||||
figure artifacts; never commit `results/` or raw datasets.
|
||||
|
||||
## 19. Important commands
|
||||
## 13. Integrity / important commands
|
||||
|
||||
```bash
|
||||
# Tests (75 expected)
|
||||
# Tests (expected: 75 passed)
|
||||
.venv/bin/python -m pytest
|
||||
|
||||
# One full paired beta run (CPU, ~62-65 min, ~12-17 GiB for 5G-NIDD)
|
||||
.venv/bin/python scripts/run_baseline.py --dataset 5gnidd --alpha 0.25 --seed 1 \
|
||||
--clients 10 --rounds 10 --local-epochs 3 \
|
||||
--aggregation-mode support_weighted --beta 1.0
|
||||
# Repository status
|
||||
git status --short
|
||||
git diff --stat
|
||||
|
||||
# Pooled multi-seed validation analysis (read-only, no retraining)
|
||||
# Read-only analyses (no retraining); see docs/research/EXPERIMENT_PROTOCOL.md
|
||||
.venv/bin/python scripts/analyze_multiseed_beta.py --dataset 5gnidd --alpha 0.25 --seeds 0 1 2
|
||||
|
||||
# Direct prototype-effect analysis (no retraining)
|
||||
.venv/bin/python scripts/analyze_prototype_effect.py \
|
||||
--snapshot artifacts/analysis/5gnidd__alpha0.25__seed0__support_weighted__beta0__prototype_snapshot.json \
|
||||
--class-names "Benign,HTTPFlood,ICMPFlood,SYNFlood,SYNScan,SlowrateDoS,TCPConnectScan,UDPFlood,UDPScan" \
|
||||
--output-dir artifacts/analysis
|
||||
|
||||
# Realized partition characterization (cross-alpha)
|
||||
.venv/bin/python scripts/analyze_partition_heterogeneity.py --dataset 5gnidd --alphas 0.25 0.5 0.75
|
||||
|
||||
# beta=0 equivalence audit
|
||||
.venv/bin/python scripts/verify_beta0_equivalence.py --dataset 5gnidd --alpha 0.25 --seeds 0 1 2
|
||||
|
||||
# Cross-alpha comparison table (5G-NIDD)
|
||||
.venv/bin/python scripts/analyze_cross_alpha.py --dataset 5gnidd --alphas 0.25 0.5 0.75
|
||||
.venv/bin/python scripts/verify_beta0_equivalence.py --dataset 5gnidd --alpha 0.25 --seeds 0 1 2
|
||||
```
|
||||
|
||||
## 20. Recommended next-session prompt
|
||||
## 14. Recommended next-session prompt
|
||||
|
||||
```
|
||||
You are continuing the perdalam-protean research repository (Session 13).
|
||||
You are continuing the perdalam-protean research repository (Session 17).
|
||||
|
||||
FIRST, read completely:
|
||||
1. docs/handoff/SESSION_HANDOFF.md
|
||||
2. docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md
|
||||
3. docs/thesis/CHAPTER4_RESULTS_OUTLINE.md
|
||||
2. docs/proposal/THESIS_PROPOSAL_DRAFT.md
|
||||
3. docs/proposal/PROPOSAL_SOURCE_MAP.md
|
||||
4. docs/thesis/CLAIMS_REGISTER.md
|
||||
5. docs/research/CROSS_DATASET_BETA_ANALYSIS.md
|
||||
6. docs/research/EXPERIMENT_PROTOCOL.md
|
||||
5. docs/thesis/CHAPTER4_RESULTS_OUTLINE.md
|
||||
6. docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md
|
||||
|
||||
Critical rules:
|
||||
- planningawal/ is immutable.
|
||||
- ~/datasets/5G-NIDD/5G-NIDD.csv and ~/datasets/X-IIoTID/X-IIoTID-dataset.csv are read-only.
|
||||
- planningawal/ is immutable; raw datasets under ~/datasets are read-only.
|
||||
- The S_j participant rule is a REPRODUCTION ASSUMPTION.
|
||||
- Do NOT add an executable equal_available mode.
|
||||
- Beta selection uses validation only; PRIMARY is pooled across clients and seeds.
|
||||
- The planned 2-dataset x 3-alpha experiment grid is COMPLETE. Do NOT run new conditions.
|
||||
- Scientific gap classification is A (NO MATERIAL GAP); no new experiment is required.
|
||||
- The 108-run 2-dataset x 3-alpha x 3-seed grid is COMPLETE. Do NOT run new training.
|
||||
- The proposal Markdown is finalized and frozen unless academic feedback requires revision.
|
||||
|
||||
Context: Session 12 completed the thesis synthesis with no training. Six pooled selections:
|
||||
X-IIoTID 0.25/0.50/0.75 = 0.00/0.75/0.50 and 5G-NIDD = 0.00/1.00/0.50; no universal/monotone beta;
|
||||
all top-two gaps below selected-beta across-seed std; beta=0 equivalence class B everywhere.
|
||||
Authoritative artifacts: artifacts/analysis/final__*.{csv,json}; figures artifacts/figures/final/.
|
||||
All synthesis numbers independently verified; pytest 75 passed; no tracked file changed.
|
||||
Context: Session 15 finalized and committed the proposal Markdown
|
||||
(docs/proposal/THESIS_PROPOSAL_DRAFT.md) at HEAD e27c5a8. Session 16 refreshed this handoff.
|
||||
"Optimasi" in the title means empirical beta selection among predefined candidates, not a global
|
||||
optimum. All six conditions are synthesized; no material scientific gap remains (classification A).
|
||||
|
||||
Task: write/format thesis chapters from the existing synthesis. Use the final__* tables and
|
||||
figA-G figures, obey CLAIMS_REGISTER.md wording, keep all limitations explicit, and do not run
|
||||
training or add inferential tests by default. Use .venv/bin/python. Do not commit or push unless
|
||||
asked.
|
||||
Task: perform the requested writing/formatting or review work using the final synthesis documents
|
||||
and artifacts/analysis/final__*. Obey CLAIMS_REGISTER.md wording, keep all limitations explicit,
|
||||
and do not run training or add inferential tests by default. Use .venv/bin/python. Do not commit
|
||||
or push unless asked.
|
||||
```
|
||||
Reference in new issue
Block a user