docs: validate baseline and add deterministic regression coverage

Session 3 baseline validation (no beta/support-weighted code added).

- Two measured full-data original_protean conditions (X-IIoTID and 5G-NIDD,
  alpha=0.5, seed=0); frozen manifests reused and reload-verified.
- Partition consistency, per-class/per-client inspection, round-0, loss audit,
  inference validation, reproducibility and a deterministic end-to-end test.
- S_j participant rule remains an explicitly labelled REPRODUCTION ASSUMPTION;
  |S_j| < M quantified in 53/162 class-condition cells (32.7%), 15/18 conditions.
- Adds docs/baseline/BASELINE_VALIDATION.md, tests/test_baseline_validation.py,
  scripts/validate_sj_participant_rule.py and small analysis artifacts.
- Test suite: 46 passed.

planningawal/ and raw datasets unchanged.
This commit is contained in:
willtanoe committed 2026-09-24 20:42:56 +00:00
1 parent 374651a0fe
commit 16b3add2cf
9 files changed
+24265 -200

No files matched your search

+182 -196
View File
@@ -3,9 +3,11 @@
This is the canonical continuation point for the next agent session. It is written to be
readable without prior chat history, terminal scrollback, or git history.
Last updated: 2026-09-24 (Session 2, baseline-only checkpoint).
Session 2 type: minimum reproducible baseline implementation + smoke test, then cleanup so
the checkpoint contains **baseline only**. **No full-scale training and no beta sweep.**
Last updated: 2026-09-24 (Session 3, baseline validation and reproduction verification).
Session 3 type: validation-only. No beta / support-weighted / `equal_available` code was
added. Two measured full-data baseline conditions were run; partition consistency, per-
class/per-client behaviour, the `S_j` ambiguity, round-0, the loss, inference,
reproducibility and a deterministic end-to-end regression test were all checked.
---
@@ -22,260 +24,244 @@ S_j = { i | N_ij > 0 and a valid local prototype C_ij exists }
beta in {0.0, 0.25, 0.5, 0.75, 1.0}
```
**This checkpoint contains the baseline only.** The proposed beta-weighted aggregation and
the `equal_available` control are intentionally NOT implemented yet; they are deferred to
the later beta/controls phase, which must not start until the baseline is verified.
**The repo still contains the baseline only.** The beta-weighted aggregation and the
`equal_available` control are still NOT implemented. Session 3 did not add them.
## 2. Authoritative source
## 2. Authoritative source and immutability
- `planningawal/PROTEAN_Revisi_Lurus.md` — authoritative research specification.
- `planningawal/` is **immutable / read-only**. Verified unchanged:
SHA-256 `9ba1d7beb94354db0c4b588199fb1e2823aec8a23be1625ec09a756507fbcb89`
(only file in `planningawal/`).
- `planningawal/` is immutable. Re-verified unchanged this session:
SHA-256 `9ba1d7beb94354db0c4b588199fb1e2823aec8a23be1625ec09a756507fbcb89`.
## 3. Dataset locations (read-only source data)
## 3. Dataset locations (read-only external source data)
- `~/datasets/X-IIoTID/X-IIoTID-dataset.csv` — SHA-256
`7b9290057ee42e784da3c0d84b781815502c9205c74175c96374e71a5ffd98a0` (mtime 2021-10-07)
`7b9290057ee42e784da3c0d84b781815502c9205c74175c96374e71a5ffd98a0` (re-verified unchanged).
- `~/datasets/5G-NIDD/5G-NIDD.csv` — SHA-256
`fa36f80859585f474504ca69eb951a079b35145537976c86b00aae3aab46ee59` (mtime 2023-02-18)
`fa36f80859585f474504ca69eb951a079b35145537976c86b00aae3aab46ee59` (re-verified unchanged).
Re-verified unchanged. Paths resolve from `configs/datasets/datasets.yaml` with env
overrides (`PROTEAN_XIIOTID_RAW_PATH`, `PROTEAN_5GNIDD_RAW_PATH`). Raw files are never
modified or copied into the repo.
## 4. Session 3 objective and outcome
## 4. Current repository state
Objective: determine whether the frozen `original_protean` baseline is coherent,
reproducible and consistent with the best available PROTEAN evidence, and explicitly
identify unresolved reproduction assumptions. **All acceptance items were met.**
Baseline code:
## 5. Full-data runs completed
```
src/protean_research/
├── data/ loading, preprocessing, splits, partition, manifest,
│ characterization, rare_unseen, pipeline
├── models/ protean_cnn, prototypes
├── aggregation/ prototype_aggregation (baseline original_protean ONLY)
├── federated/ client, server, round
├── evaluation/ metrics
└── utils/ config, seeding, integrity
scripts/ run_baseline.py, generate_partitions.py
tests/ 34 baseline-focused tests
artifacts/partitions/ 18 full-data manifests (tracked, small, content-hashed)
artifacts/analysis/ 18 characterization + 18 rare_unseen CSVs (tracked, small)
artifacts/processed/ ignored: preprocessor state + smoke partitions/snapshots
results/ ignored: smoke run outputs
```
Both with the frozen protocol (`original_protean`, 10 clients, 10 rounds, 3 local epochs,
batch 256, lr 1e-3, mu 0.1, lambda 1.0), CPU-only (`torch 2.14.0+cpu`, CUDA unavailable).
Environment: `.venv/` exists (gitignored) with numpy 2.5.3, pandas 3.0.6,
torch 2.14.0+cpu, pyyaml 6.0.3, pytest 9.1.1. Use `.venv/bin/python`.
| condition | wall | peak RSS | accuracy | macro-F1 | weighted-F1 | rare macro-F1 | rare recall | exit |
|---|---|---|---|---|---|---|---|---|
| X-IIoTID α=0.50 seed 0 | 42:07 | 6.21 GiB | 0.8904 | 0.7077 | 0.9176 | 0.6096 | 0.8715 | 0 |
| 5G-NIDD α=0.50 seed 0 | 1:04:52 | 10.37 GiB | 0.7344 | 0.8369 | 0.7088 | 0.8720 | 0.9743 | 0 |
## 5. What was completed this session
Both re-used their frozen manifest (`manifest_reload_ok = true`), completed all 10 rounds
with all 10 clients active and no failures, and produced 9/9 global prototypes. Artifact
dirs: `results/xiiotid__alpha0.5__seed0__original_protean/`,
`results/5gnidd__alpha0.5__seed0__original_protean/` (ignored by git). Full detail:
`docs/baseline/BASELINE_VALIDATION.md`.
1. Froze every previously-PROVISIONAL protocol value in
`configs/experiments/experiment_protocol.yaml` and `docs/research/EXPERIMENT_PROTOCOL.md`.
2. Dataset config: explicit `categorical_columns`, `drop_duplicate_rows`, missing tokens,
fixed 5G-NIDD `Unnamed: 0` (quoted).
3. Data foundation (`data/`): read-only loading, deterministic canonicalisation, leak-free
train-fitted preprocessing, deterministic stratified 70/10/20 split, Dirichlet partition
with content-hashed manifests and `N_ij`, characterization, rare/unseen metadata.
4. Model (`models/`): baseline CNN, parameter averaging, embedding extraction, local
prototypes, global prototype collection, nearest-prototype inference.
5. Baseline aggregation (`aggregation/`): `original_protean` only. The proposed
`support_weighted` and the `equal_available` control were **removed** so the checkpoint
is baseline only; an unavailable-mode guard raises `ValueError`.
6. Federated loop (`federated/`): local update (CE + prototype alignment + model
alignment), server model/prototype aggregation, round orchestration, round-0 snapshot.
7. Evaluation (`evaluation/`): accuracy, macro/weighted F1, per-class P/R/F1, rare/unseen.
8. Scripts: `generate_partitions.py` (18 manifests, full data, reload-verified);
`run_baseline.py` (one baseline condition; no beta/mode options).
9. Tests: 34 baseline-only tests, all passing.
10. Smoke test: baseline-only on both datasets.
11. Docs: `docs/baseline/BASELINE_IMPLEMENTATION.md` rewritten with provenance tags.
Resource sizing for the future grid: ~42 min and ~6 GiB for X-IIoTID, ~65 min and ~10 GiB
for 5G-NIDD, per `(dataset, alpha, seed)` condition. A full grid
`2 datasets × 3 alphas × 3 seeds = 18` conditions is therefore on the order of
15–20 CPU-hours plus beta multiplicity; measure before launching anything large.
## 6. Baseline decisions and provenance
## 6. Tests added / test status
Full detail in `docs/baseline/BASELINE_IMPLEMENTATION.md`. Summary:
- New file `tests/test_baseline_validation.py` (12 tests): round-0 handling (spy confirms
round 0 receives no global prototypes and round 1 does), independent replication of the
documented loss formula (`atol=1e-5`), prototype-alignment skip for classes without a
global prototype, frozen-coefficient guard (`lambda=1.0`, `mu=0.1`, rounds=10, epochs=3,
batch=256, lr=1e-3), equal-weight model averaging, non-contiguous class-id inference,
brute-force squared-L2 cross-check, empty-prototype errors, local prototype class
mapping, and a deterministic end-to-end regression test.
- Total suite: `.venv/bin/python -m pytest -q` -> **46 passed** (34 pre-existing + 12 new).
- Note: `BASELINE_IMPLEMENTATION.md` previously said "39 tests"; the accurate Session-2
count was 34. Corrected this session.
- **No official PROTEAN code exists**, so no decision is source-confirmed by code.
- **PAPER**: architecture, `mu = 0.1`, 10 rounds / 3 local epochs, 80/20 test, equal `1/M`
model averaging, end-of-round mean-embedding prototypes, nearest-prototype inference.
- **REPRODUCTION ASSUMPTION (project decision)**: `original_protean` participant rule is
`S_j` with equal `1/|S_j|` weights. The paper writes `1/M` but is ambiguous for classes
missing on a client; this assumption follows the thesis sentence that prototypes are
shared only for locally observed classes. It is **not** treated as verified.
- **PROJECT DECISION**: `lambda = 1.0`; Adam `lr = 1e-3`; batch size 256; kernel size 3;
stratified 70/10/20 split with an added validation set; seeds `{0,1,2}`;
preprocessing recipe; per-class Dirichlet allocation; per-client eval partitioning.
- **INFERRED**: per-sample prototype alignment (FedProto-style reading of Eq. 3); round-0
handling (no prototypes, prototype term skipped).
- **UNRESOLVED**: whether a literal `1/M` missing-class reading differs from `S_j`.
## 7. `S_j` stress-test findings (central methodological result)
## 7. Seeds (explicit)
- `|S_j| < M` occurs in **53/162 class-condition cells (32.7 %)** across **15/18**
`(dataset, alpha, seed)` conditions (`artifacts/analysis/sj_participant_rule_analysis.csv`).
- Affected classes include high-frequency ones and even the benign class in two conditions;
the effect is strongest at `alpha = 0.25`.
- On the fixed X-IIoTID round-0 snapshot, the hypothetical literal `1/M` reading with
missing prototypes replaced by the zero vector is displaced by **~10–20 % of the
prototype norm** for the three affected classes (Exfiltration 0.724, Exploitation 1.131,
Tampering 0.510). Hypothetical-only, never used for training:
`artifacts/analysis/sj_vs_literal_1m_hypothetical_xiiotid__alpha0.5__seed0__original_protean.csv`,
reproducible via `scripts/validate_sj_participant_rule.py`.
- Conclusion: the ambiguity is **material, not rare**, but **cannot be resolved** without
original code (none exists). The current `S_j` equal-weight rule remains a documented
REPRODUCTION ASSUMPTION, defensible from the thesis sentence that prototypes are shared
only for locally observed classes. It is **not** verified original behaviour. The
`equal_available` control is unnecessary: the baseline already *is* equal-over-`S_j`, so
it is arithmetically identical to the proposed `beta = 0` rule.
- Exact values: **`{0, 1, 2}`**.
- Rationale: the paper averages over three random seeds but names none, so the values are
**not source-confirmed**. `{0, 1, 2}` is the minimal three-seed set and is an explicit
**project decision**.
- Each seed deterministically controls the split, Dirichlet partition, model
initialisation and batch order, via `protean_research.utils.seeding.derive_seed`.
- The set is identical across all baseline conditions.
## 8. Round-0, loss and inference validation
## 8. Manifests (18)
- **Round 0**: no global prototypes exist; the prototype-alignment term is skipped; the
first global prototypes come from round-0 aggregation and are used from round 1.
Provenance INFERRED / PROJECT DECISION. Covered by a focused test.
- **Loss**: `nll_loss(mean) + lambda * mean_matched ||phi(x)-Cbar_y||^2 + (mu/2)*sum||w-w_global||^2`.
Prototype term is per-sample and only for classes with a global prototype; missing
prototypes add no term and do not enter the normaliser. Coefficients `lambda=1.0`
(PROJECT DECISION), `mu=0.1` (PAPER). Independently replicated to `atol=1e-5`.
- **Inference**: nearest global prototype by squared L2 (expanded norm), returns true
class ids, restricts to classes with a global prototype, raises on empty prototype sets.
Cross-checked against brute force.
`artifacts/partitions/` holds **18** manifests from `2 datasets × 3 alphas × 3 seeds`.
They are content-hashed, reload-verified, and reused by later variants for pairing. Sample
indices are regenerated deterministically from the stored recipe/RNG seed and checked
against the stored `N_ij` and hash, keeping manifests small.
## 9. Reproducibility findings
## 9. Verification performed this session
The same smoke condition was run twice into separate output roots. All result artifacts
(`history.json`, `test_global_metrics.json`, `test_confusion_matrix.json`,
`test_per_class.csv`, `per_client.csv`, `train_support_Nij.csv`, `characterization.csv`,
`rare_unseen.csv`, prototype snapshot) were **byte-identical**; `manifest_content_hash` and
`preprocessing_hash` were identical. Only differences were the run-specific manifest
`created_utc` timestamp and the explicit `manifest_path`. The deterministic regression test
also produced exactly equal parameters/prototypes/history across two in-process runs.
No nondeterminism source was observed on this CPU backend; tolerance clause for future
accelerated backends is `atol=1e-5 / rtol=1e-4`.
- `python -m pytest -q` -> **34 passed**.
- Full partition generation on both datasets: 18 manifests, each reload-verified.
- Non-IID check (seed 0): X-IIoTID `alpha=0.75` max 1 missing class/client, `alpha=0.25`
up to 3; 5G-NIDD similar. Qualitatively consistent with the paper setting.
- Baseline-only smoke on X-IIoTID (3000 rows, 4 clients, 2 rounds, 1 epoch): full path
succeeds, ~4.5 s.
- Baseline-only smoke on 5G-NIDD (4000 rows): succeeds.
- No beta/support_weighted code is invoked or present.
- Integrity: `planningawal/` and both raw datasets retain their recorded SHA-256 values.
## 10. Partition consistency (X-IIoTID α=0.50 seed 0)
## 10. Files added this session
All PASS: manifest reproduces (split hash + partition hash + `N_ij`); splits disjoint and
cover the cleaned dataset; each train sample assigned exactly once with no cross-client
duplication; `N_ij` matches actual assigned samples; client totals sum to `num_train`
(569,909) and class totals match train label counts.
Source: `src/protean_research/data/{__init__,loading,preprocessing,splits,partition,manifest,
characterization,rare_unseen,pipeline}.py`,
`src/protean_research/models/{__init__,protean_cnn,prototypes}.py`,
`src/protean_research/federated/{__init__,client,server,round}.py`,
`src/protean_research/evaluation/{__init__,metrics}.py`,
`src/protean_research/utils/{seeding,integrity}.py`.
## 11. Files created / modified this session
Scripts: `scripts/run_baseline.py`, `scripts/generate_partitions.py`.
Created:
Tests: `tests/test_data_pipeline.py`, `tests/test_partition.py`, `tests/test_prototypes.py`,
`tests/test_baseline_aggregation.py`, `tests/test_metrics.py`, `tests/test_federated_smoke.py`.
- `docs/baseline/BASELINE_VALIDATION.md` (new authoritative validation record).
- `tests/test_baseline_validation.py` (12 tests).
- `scripts/validate_sj_participant_rule.py` (read-only analysis, no experiment mode).
- `artifacts/analysis/sj_participant_rule_analysis.csv` (all 18 conditions, |S_j| analysis).
- `artifacts/analysis/sj_vs_literal_1m_hypothetical_xiiotid__alpha0.5__seed0__original_protean.csv`.
- `artifacts/analysis/xiiotid__alpha0.5__seed0__original_protean__prototype_snapshot.json`.
- `artifacts/analysis/5gnidd__alpha0.5__seed0__original_protean__prototype_snapshot.json`.
Docs: `docs/baseline/BASELINE_IMPLEMENTATION.md`.
Modified:
Artifacts (tracked, small): `artifacts/partitions/*.json` (18),
`artifacts/analysis/*.csv` (36).
- `docs/baseline/BASELINE_IMPLEMENTATION.md` (validation link, corrected test count,
updated remaining-assumptions §14).
- `docs/handoff/SESSION_HANDOFF.md` (this file).
## 11. Files modified this session
Ignored/generated (not tracked): `results/xiiotid__alpha0.5__seed0__original_protean/`,
`results/5gnidd__alpha0.5__seed0__original_protean/`,
`artifacts/processed/*` (preprocessor state).
- `configs/datasets/datasets.yaml`
- `configs/experiments/experiment_protocol.yaml`
- `docs/research/EXPERIMENT_PROTOCOL.md`
- `docs/research/IMPLEMENTATION_PLAN.md`
- `README.md`
- `src/protean_research/aggregation/__init__.py`
- `src/protean_research/aggregation/prototype_aggregation.py`
- `src/protean_research/models/prototypes.py`
- `src/protean_research/federated/server.py`
- `src/protean_research/federated/round.py`
- `scripts/run_baseline.py`
- tests (baseline-only aggregation suite; beta tests removed)
`planningawal/` and both raw datasets were not modified.
`planningawal/` was not touched.
## 12. Git status
## 12. Removed from this checkpoint (deferred to beta/controls phase)
No commit was requested or made. Session 2 baseline commit `0aa8e2b` and the docs commit
`374651a` are on `main` at `origin/main`. Working tree has one modified tracked file
(`docs/baseline/BASELINE_IMPLEMENTATION.md`) and the new untracked files listed above.
`results/`, `artifacts/processed/`, `.venv/`, `__pycache__/` remain ignored. No secrets,
no raw datasets. `AGENTS.md` is tracked (commit `374651a`).
- `support_weighted` aggregation code and `beta` parameters.
- `equal_available` mode (redundant with the baseline `S_j` rule; only needed if a later
investigation shows the baseline participant rule differs).
- `--mode` / `--beta` CLI options in `run_baseline.py`.
- Beta-specific tests (`beta = 0/1`, single-contributor/equal-support beta invariance,
normalised beta weights, the `original_protean == support_weighted(beta=0)` equivalence
test). The former `tests/test_prototype_aggregation.py` was removed and folded into a
baseline-only `tests/test_baseline_aggregation.py`.
- Any claim that `original_protean == support_weighted(beta=0)` demonstrates equivalence
with original PROTEAN. That was at most an internal implementation-equivalence check and
is no longer present in code or docs.
## 13. Unresolved assumptions / blockers
## 13. Git status
1. **`S_j` participant/denominator rule** is a reproduction assumption, quantified but
unresolved (see §7). All future between-method claims must carry this caveat.
2. `lambda = 1.0`, optimiser (Adam), lr (1e-3), batch size (256) are PROJECT DECISIONS.
3. Prototype-alignment form (per-sample) and round-0 initialisation are INFERRED.
4. Model aggregation is equal `1/M` (PAPER); a FedAvg-style weighted wording in the thesis
intro conflicts and is not implemented (documented conflict).
5. Per-client evaluation partitioning is a PROJECT DECISION.
6. Session-2 manifests store `preprocessing_hash = null` / `config_hash = null`; they do
not bind the preprocessing recipe. Split/partition hashes and `N_ij` reproduce exactly,
and run metadata records the live preprocessing hash. Binding the hash into future
manifests would close this provenance gap.
7. Only one full-data condition per dataset has been validated; no reproduction success is
claimed from two runs.
Not committed (no commit was requested). `git status --short` shows modified tracked files
(configs, docs, aggregation package, federated modules, README, run script) plus untracked
source/tests/scripts/docs, the 18 partition manifests and 36 analysis CSVs. `AGENTS.md` was
already untracked before this session. `.venv/`, `artifacts/processed/`, `results/`,
`logs/`, and `*.log` are ignored. No checkpoints or raw datasets are tracked.
## 14. Exact next task
## 14. Unresolved issues / blockers
**Recommended: (B) freeze the validated baseline and prepare the paired beta
aggregation implementation — not a full grid.**
1. **Baseline is implemented but not yet VALIDATED.** The `original_protean` participant
rule is a reproduction assumption, not verified behavior.
2. `lambda`, optimiser/schedule, and round-0 handling are unverified.
3. No full-scale run has been timed; compute feasibility of a real sweep is unmeasured
(X-IIoTID ~570k train rows, 5G-NIDD ~851k; CPU-only).
4. Per-client evaluation partitioning is a project decision.
5. Prototype-alignment form (per-sample vs class-mean) is a documented inference.
6. Whether the literal `1/M` missing-class reading differs from the `S_j` assumption is
open; a deliberate control is needed in the verification phase.
Justification from evidence:
## 15. Exact next task
- Implementation correctness, determinism, partition consistency and the full-data paths
are validated on both datasets (Sessions 2–3).
- Option (A), "resolve remaining baseline reproduction ambiguity", is **not achievable
with available evidence**: no official PROTEAN code exists and the paper is internally
inconsistent. The ambiguity has been quantified; further "resolution" would only be
another documented decision.
- Therefore freeze the baseline definition *including* the documented `S_j` equal-weight
assumption and its quantified materiality, then implement the proposed beta weighting
as a new aggregation mode **without altering the baseline** and run a **single paired
`(dataset, alpha, seed)` condition first** (e.g. X-IIoTID α=0.5 seed 0, reusing the
frozen manifest) across `beta in {0, 0.25, 0.5, 0.75, 1}` before any grid.
**Baseline validation and reproduction verification — NOT beta weighting.**
Next session must:
Next session should:
1. Add `support_weighted` (`w_ij = N_ij^beta`) as a new mode only; keep
`original_protean` byte-identical and add the deferred beta-invariant tests
(`beta=0` == equal mean over `S_j`; `beta=1` == support weighting; single-contributor
and equal-support invariance; normalised weights sum to 1).
2. Do **not** add an executable `equal_available` mode (baseline already is equal-over-`S_j`).
3. Keep validation/test separation: beta selection on validation rare-class macro-F1,
test only after selection.
4. Run one paired condition end to end and inspect; measure before any grid.
5. Report the `S_j` reproduction assumption explicitly alongside any beta result.
1. Run one full-data baseline condition (e.g. X-IIoTID, `alpha=0.5`, `seed=0`,
`original_protean`) with measured wall-clock and memory; record it.
2. Inspect the run: per-class precision/recall/F1, per-client rare metrics, confusion
matrix. Check plausibility and missing-class-client behavior.
3. Stress-test the `S_j` reproduction assumption: compare against a literal `1/M`
omitted-class reading to quantify the effect of the assumption (introduce the control
only for this investigation, clearly marked).
4. Add a deterministic end-to-end baseline regression test pinning a small result.
5. Only after the baseline is demonstrably verified and recorded, begin the paired beta
comparison on a single `(dataset, alpha, seed)` before any grid.
Do **not** launch the full grid until the single paired condition is verified.
Do **not** launch the full grid, and do not change the architecture, loss coefficients, or
inference rule.
## 16. Important commands
## 15. Important commands
```bash
# Tests
# Tests (46 expected)
.venv/bin/python -m pytest -q
# Generate / verify all partition manifests (full data, ~30 s)
.venv/bin/python scripts/generate_partitions.py
# Baseline smoke on both datasets (writes only to ignored dirs)
.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.5 --seed 0 \
--max-rows 3000 --sample-from 60000 --clients 4 --rounds 2 --local-epochs 1
.venv/bin/python scripts/run_baseline.py --dataset 5gnidd --alpha 0.25 --seed 1 \
--max-rows 4000 --sample-from 60000 --clients 4 --rounds 2 --local-epochs 1
# Full-data baseline for one condition (next session; expect long CPU runtime)
# Full-data baseline for one condition (measure first; ~40-65 min, CPU)
.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.5 --seed 0 \
--clients 10 --rounds 10 --local-epochs 3
.venv/bin/python scripts/run_baseline.py --dataset 5gnidd --alpha 0.5 --seed 0 \
--clients 10 --rounds 10 --local-epochs 3
# Smoke (writes only to ignored dirs)
.venv/bin/python scripts/run_baseline.py --dataset xiiotid --alpha 0.5 --seed 0 \
--max-rows 3000 --sample-from 60000 --clients 4 --rounds 2 --local-epochs 1
# Read-only S_j participant-rule validation (writes analysis artifacts only)
.venv/bin/python scripts/validate_sj_participant_rule.py \
--snapshot artifacts/analysis/xiiotid__alpha0.5__seed0__original_protean__prototype_snapshot.json
# Regenerate / verify all partition manifests (full data, ~30 s)
.venv/bin/python scripts/generate_partitions.py
```
## 17. Recommended next-session prompt
## 16. Recommended next-session prompt
```
You are continuing the perdalam-protean research repository (Session 3).
You are continuing the perdalam-protean research repository (Session 4).
FIRST, read completely:
1. docs/handoff/SESSION_HANDOFF.md
2. docs/baseline/BASELINE_IMPLEMENTATION.md
3. docs/research/EXPERIMENT_PROTOCOL.md
Then read as needed:
- planningawal/PROTEAN_Revisi_Lurus.md
- docs/baseline/PROTEAN_BASELINE_INVESTIGATION.md
- docs/research/IMPLEMENTATION_PLAN.md
2. docs/baseline/BASELINE_VALIDATION.md
3. docs/baseline/BASELINE_IMPLEMENTATION.md
4. docs/research/EXPERIMENT_PROTOCOL.md
Critical rules:
- planningawal/ is immutable. Do not edit/create/move/rename/delete anything inside it.
- ~/datasets/X-IIoTID/X-IIoTID-dataset.csv and ~/datasets/5G-NIDD/5G-NIDD.csv are
read-only external source data. Do not modify or copy them into the repo.
- This checkpoint is BASELINE ONLY. Do NOT add the proposed support_weighted/beta code
or the equal_available control until the baseline is verified.
- Do NOT launch the full alpha x seed x dataset x beta sweep.
- The baseline is validated but the S_j participant rule is a REPRODUCTION ASSUMPTION
(material in 32.7% of class-condition cells). Carry this caveat in all reporting.
- Do NOT add an executable equal_available mode.
- Do NOT launch the full alpha x seed x dataset x beta grid.
Your task is Section 15 of the handoff: BASELINE VALIDATION AND REPRODUCTION
VERIFICATION. Run one full-data baseline condition with measured runtime, inspect its
per-class/per-client behavior, stress-test the S_j participant-rule assumption, and add a
deterministic end-to-end baseline regression test. Record everything in
docs/baseline/BASELINE_IMPLEMENTATION.md and update docs/handoff/SESSION_HANDOFF.md.
Task (Section 14 of the handoff): freeze the validated baseline and add the proposed
support_weighted aggregation (w_ij = N_ij^beta) as a NEW mode without changing
original_protean. Add the deferred beta-invariant tests. Then run ONE paired
(dataset, alpha, seed) condition across beta in {0, 0.25, 0.5, 0.75, 1}, select beta on
validation rare-class macro-F1, and evaluate the selected beta on test. Measure runtime
before scaling. Update docs and docs/handoff/SESSION_HANDOFF.md.
Use .venv/bin/python for all Python commands.
Use .venv/bin/python for all Python commands. Do not commit or push unless asked.
```