diff --git a/AGENTS.md b/AGENTS.md new file mode 100755 index 0000000..dab7f12 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,831 @@ +# PROTEAN Prototype Aggregation Research + +## Project Purpose + +This repository contains a master's research implementation based on: + +`planningawal/PROTEAN_Revisi_Lurus.md` + +Research focus: + +**Server-side prototype aggregation in PROTEAN using class-specific sample-support weighting for rare attack detection under non-IID federated data.** + +The proposed aggregation is: + +$$ +\bar{C}_j(\beta) += +\frac{ +\sum_{i \in S_j} N_{ij}^{\beta} C_{ij} +}{ +\sum_{i \in S_j} N_{ij}^{\beta} +} +$$ + +with planned candidate values: + +```text +beta ∈ {0.00, 0.25, 0.50, 0.75, 1.00} +``` + +where: + +* `C_ij` is client `i`'s local prototype for class `j` +* `N_ij` is the number of local training samples of class `j` +* `S_j` is the set of clients contributing a valid prototype for class `j` + +This repository is a research codebase. Scientific traceability and experimental isolation are more important than implementation convenience. + +--- + +# 1. AUTHORITATIVE RESEARCH SOURCE + +The primary research specification is: + +```text +planningawal/PROTEAN_Revisi_Lurus.md +``` + +Read this file before making methodological decisions. + +Do not silently reinterpret or broaden the research objective. + +If implementation evidence conflicts with an assumption in the planning document: + +1. preserve the source document unchanged +2. document the discrepancy +3. distinguish paper behavior, code behavior, inference, and project decision +4. do not silently choose one interpretation + +--- + +# 2. `planningawal/` IS IMMUTABLE + +The entire directory: + +```text +planningawal/ +``` + +is read-only research source material. + +Agents MAY: + +* read files +* cite them internally +* use them to guide implementation + +Agents MUST NOT: + +* edit files inside `planningawal/` +* create files inside `planningawal/` +* rename files inside `planningawal/` +* move files into or out of it +* delete files +* autoformat its contents +* rewrite Markdown formatting +* place reports, logs, code, or handoffs there + +All generated project material must live elsewhere. + +Before major checkpoints, verify that `planningawal/` remains unchanged. + +--- + +# 3. RAW DATASETS ARE EXTERNAL AND IMMUTABLE + +Raw datasets exist outside the repository: + +```text +~/datasets/5G-NIDD/5G-NIDD.csv +~/datasets/X-IIoTID/X-IIoTID-dataset.csv +``` + +These are source data. + +Do NOT: + +* modify them +* overwrite them +* rename them +* move them +* copy full raw datasets into the repository +* commit them to Git + +Dataset locations must be supplied through configuration, environment variables, or CLI arguments rather than duplicated throughout source code. + +Generated derivatives may be stored inside the repository under appropriate ignored artifact directories. + +Raw data and generated data must always remain distinguishable. + +--- + +# 4. PROJECT STRUCTURE + +Prefer the following separation of concerns: + +```text +perdalam-protean/ +├── AGENTS.md +├── README.md +├── planningawal/ # immutable research source +├── docs/ +│ ├── research/ +│ ├── baseline/ +│ └── handoff/ +├── configs/ +│ ├── datasets/ +│ ├── experiments/ +│ └── models/ +├── src/ +│ └── protean_research/ +│ ├── data/ +│ ├── federated/ +│ ├── aggregation/ +│ ├── models/ +│ ├── evaluation/ +│ └── utils/ +├── scripts/ +├── tests/ +├── artifacts/ +│ ├── partitions/ +│ ├── processed/ +│ ├── checkpoints/ +│ └── analysis/ +└── results/ +``` + +This is a design guideline, not an instruction to create empty directories without purpose. + +Do not introduce new top-level structures without a clear reason. + +--- + +# 5. SESSION CONTINUITY + +The canonical session continuation file is: + +```text +docs/handoff/SESSION_HANDOFF.md +``` + +At the beginning of a new substantial session: + +1. read `AGENTS.md` +2. read `docs/handoff/SESSION_HANDOFF.md` if it exists +3. read relevant project documentation +4. consult `planningawal/PROTEAN_Revisi_Lurus.md` when research semantics matter +5. inspect actual repository state before making assumptions + +Do not reconstruct state from chat history when repository documentation is available. + +The handoff must reflect actual repository state. + +Do not copy stale plans forward as if they were completed work. + +--- + +# 6. HANDOFF REQUIREMENT + +At the end of any substantial development/research session, update: + +```text +docs/handoff/SESSION_HANDOFF.md +``` + +The handoff should contain: + +* current objective +* completed work +* important findings +* files created +* files modified +* commands/tests performed +* current architecture +* methodological decisions +* unresolved questions +* known failures or incomplete work +* Git state where relevant +* exact next task +* a concise ready-to-paste next-session prompt + +Do not write huge terminal-only reports that future sessions cannot recover. + +Long analysis should be written into project Markdown files. + +--- + +# 7. BASELINE FIRST + +Do not implement the proposed beta weighting as a replacement for PROTEAN before baseline semantics are understood. + +The project must preserve a faithful original baseline. + +Important unresolved or verifiable baseline properties include: + +* local prototype construction +* missing-class handling +* server aggregation participant set +* aggregation denominator +* class support/count availability +* global prototype redistribution +* prototype alignment behavior +* inference behavior + +Never assume: + +```text +beta = 0 == original PROTEAN +``` + +This equivalence is only valid if aggregation participants and missing-class handling are also identical. + +If needed, distinguish: + +```text +original_protean +equal_available +support_weighted +``` + +`equal_available` should exist only if methodological evidence shows it is required as a control. + +--- + +# 8. RESEARCH SCOPE + +The intended modification is narrowly scoped to: + +```text +server-side prototype aggregation +``` + +Do NOT independently add research dimensions such as: + +* client rejection +* Byzantine or poisoning defense +* adaptive loss functions +* prototype clustering +* alternative distance metrics +* cosine similarity experiments +* dynamic architectures +* personalized FL +* attention modules +* client scoring +* prototype-quality scoring +* additional aggregation algorithms + +unless required to faithfully reproduce the verified baseline. + +A technically interesting addition is not automatically relevant to this thesis. + +--- + +# 9. SCIENTIFIC INTERPRETATION + +`N_ij` represents: + +```text +class-specific sample support +``` + +It does NOT represent: + +```text +ground-truth prototype quality +``` + +Do not state or encode assumptions that a larger `N_ij` automatically means a better prototype. + +Low-support clients may contain useful variation unavailable at higher-support clients. + +Use neutral terminology such as: + +```text +support_weighted +sample_support_weighting +beta_weighted +``` + +Avoid names implying proven superiority, such as: + +```text +improved_aggregation +optimal_aggregation +better_prototype +``` + +until supported by evidence. + +--- + +# 10. BETA SEMANTICS + +The proposed weight is: + +```text +w_ij = N_ij ** beta +``` + +with: + +```text +0 <= beta <= 1 +``` + +Expected properties: + +## beta = 0 + +For an identical contributor set `S_j`: + +```text +N_ij ** 0 = 1 +``` + +therefore aggregation becomes equal weighting. + +This must not automatically be labeled the original PROTEAN baseline. + +## beta = 1 + +Weight is directly proportional to class-specific support. + +Because each local prototype is a local class embedding mean, this should correspond to the combined mean of included embeddings when all assumptions match. + +## One contributor + +If `|S_j| = 1`, aggregation is unchanged for every beta. + +## Equal support + +If all `N_ij` values are equal for fixed local prototypes, changing beta must not change the aggregate. + +These invariants should eventually have tests. + +--- + +# 11. MISSING CLASS SEMANTICS + +For the proposed formulation: + +```text +S_j = clients with N_ij > 0 and a valid local prototype C_ij +``` + +Do not fabricate a prototype for a missing class. + +Do not use a zero vector unless this is specifically required to reproduce the original baseline in `original_protean` mode. + +Missing-class handling must remain explicit in code. + +Do not hide it inside generic tensor padding. + +--- + +# 12. RARE CLASS DEFINITION + +For each client, rare classes are defined from TRAINING DATA. + +Among attack classes satisfying: + +```text +N_ij > 0 +``` + +select the two attack classes with the smallest training support. + +If only one attack class is present: + +* evaluate that one class + +If no attack class is present: + +* exclude that client from rare-class aggregation + +Tie behavior must eventually be deterministic and documented. + +--- + +# 13. UNSEEN CLASS DEFINITION + +Attack classes satisfying: + +```text +N_ij = 0 +``` + +during client training are unseen classes. + +Rare classes and unseen classes are different evaluation categories. + +Never merge them into one metric. + +--- + +# 14. DATA PARTITIONING + +Target Dirichlet non-IID settings are: + +```text +alpha = 0.75 +alpha = 0.50 +alpha = 0.25 +``` + +Multiple deterministic seeds are required. + +For a given: + +```text +(dataset, alpha, seed) +``` + +all compared aggregation configurations must reuse exactly the same partition. + +Persist partition manifests. + +Do not regenerate a different Dirichlet split for every beta candidate. + +Paired comparison requires identical: + +* client partition +* train split +* validation split +* test split +* class mapping +* model initialization policy +* training hyperparameters + +unless a deviation is explicitly justified and documented. + +--- + +# 15. VALIDATION AND TEST SEPARATION + +Beta selection must use validation data only. + +Primary planned selection metric: + +```text +rare-class macro-F1 +``` + +The test set must not influence beta selection. + +Maintain a clear separation between: + +```text +candidate evaluation +beta selection +final test evaluation +``` + +Do not use test results to tune: + +* beta +* training epochs +* thresholds +* seed choice +* class selection +* model configuration + +Do not call the selected beta globally optimal. + +Correct wording is: + +```text +best among the tested beta candidates under the defined validation procedure +``` + +--- + +# 16. REQUIRED EVALUATION GRANULARITY + +Do not report only global accuracy. + +Preserve enough information for analysis at: + +* dataset level +* alpha level +* seed level +* client level +* class level + +Important outputs include: + +* rare-class macro-F1 +* rare-class recall +* overall macro-F1 +* per-client metrics +* per-class precision/recall/F1 where relevant +* evaluation support +* unseen-class results +* `N_ij` +* number of clients containing a class +* number of contributing prototypes + +Actual realized class imbalance must be recorded. + +Dirichlet alpha alone is not sufficient to characterize a partition. + +--- + +# 17. DIRECT PROTOTYPE ANALYSIS + +The research distinguishes: + +```text +direct aggregation effect +``` + +from: + +```text +end-to-end training effect +``` + +Where instrumentation is implemented, preserve fixed: + +```text +C_ij +N_ij +``` + +from the same communication round and recompute candidate global prototypes without retraining. + +Useful measures include: + +```text +||Cbar_j(beta) - Cbar_j(0)|| +``` + +as well as: + +* normalized client weights +* prototype displacement +* distance to local prototypes +* pairwise differences between beta settings + +Do not interpret direct displacement alone as improved detection. + +--- + +# 18. CODE QUALITY + +Prefer: + +* small cohesive modules +* typed interfaces where practical +* deterministic behavior +* explicit configuration +* clear validation errors +* minimal global state +* no duplicated experiment logic +* reusable dataset abstractions +* reproducible random-state handling + +Avoid: + +* magic constants +* hardcoded absolute user paths inside research logic +* giant scripts containing preprocessing, training, evaluation, and plotting together +* silent fallback behavior +* hidden data mutation +* unnecessary dependencies +* unrelated refactoring during methodological changes + +Every important research parameter should be visible in configuration or experiment metadata. + +--- + +# 19. TESTING EXPECTATIONS + +Before large experiments, establish tests for critical research invariants. + +Expected tests eventually include: + +* beta=0 arithmetic-mean property +* beta=1 support-weighted property +* single-contributor invariance +* equal-support invariance +* missing-class exclusion +* normalized weight sum +* finite numerical output +* deterministic partitions +* partition save/reload identity +* baseline behavior preservation +* validation/test separation + +Prefer focused tests before expensive training runs. + +--- + +# 20. COMPUTE DISCIPLINE + +Do not launch a full experimental sweep merely because the runner exists. + +Before expensive experiments: + +1. unit tests must pass +2. data pipeline must be validated +3. one small smoke run must succeed +4. baseline behavior must be checked +5. output artifacts must be inspected +6. experiment metadata must be reproducible + +Do not invent runtime estimates. + +Measure first. + +--- + +# 21. ARTIFACTS AND RESULTS + +Generated outputs should be structured and machine-readable. + +Prefer: + +* CSV +* JSON +* JSONL +* Parquet where appropriate + +over terminal-only text. + +Do not overwrite previous experiment results silently. + +Each experiment should be identifiable from metadata such as: + +```text +dataset +alpha +seed +aggregation_mode +beta +training configuration +code revision +``` + +Checkpoint/model files should not be committed unless explicitly required. + +--- + +# 22. DOCUMENTATION + +Research decisions that affect interpretation must be documented. + +Expected documentation areas: + +```text +docs/research/ +docs/baseline/ +docs/handoff/ +``` + +Prefer updating an existing authoritative document over creating multiple overlapping reports. + +Do not create `final_v2_revised_latest.md` style document proliferation. + +--- + +# 23. GIT DISCIPLINE + +The intended remote repository is: + +```text +git@github.com:willtanoe/prototype-agregation.git +``` + +Before checkpointing: + +* inspect `git status` +* inspect staged diff +* verify no secrets +* verify no raw datasets +* verify no large accidental artifacts +* verify `planningawal/` remains unchanged + +Do not force push. + +Do not rewrite shared history unless explicitly instructed. + +Use checkpoint commits at meaningful research/development boundaries. + +--- + +# 24. SECURITY AND SECRETS + +Never commit: + +* API keys +* access tokens +* passwords +* SSH private keys +* `.env` secrets +* service credentials + +Do not print discovered secrets into reports. + +Use environment variables or ignored local configuration for secret material. + +--- + +# 25. SOURCE OF TRUTH HIERARCHY + +When deciding what to trust, use approximately this order: + +1. immutable research specification for intended thesis design +2. verified original PROTEAN paper/code for baseline behavior +3. actual repository implementation +4. documented project decisions +5. inference + +Do not present inference as source-confirmed fact. + +When sources conflict, document the conflict rather than hiding it. + +--- + +# 26. AGENT BEHAVIOR + +Before modifying code: + +* inspect relevant files +* understand current architecture +* identify the smallest required change +* preserve established behavior outside the target scope + +When uncertain: + +* investigate evidence first +* write down unresolved ambiguity if it cannot be resolved +* avoid silent assumptions + +Do not optimize for producing the largest amount of code. + +Optimize for correctness, reproducibility, and traceability. + +--- + +# 27. USE OF SUBAGENTS + +Subagents may be used for parallel READ-ONLY investigation where tasks are independent. + +Good subagent tasks: + +* inspect original PROTEAN source +* inspect dataset schema +* review experimental methodology +* review code/tests +* inspect reproducibility issues + +Avoid multiple agents concurrently editing the same files. + +A single primary agent should own integration and final edits. + +Subagent findings are advisory until verified by the primary agent. + +Do not merge contradictory findings without resolving their evidence. + +--- + +# 28. BEFORE DECLARING A PHASE COMPLETE + +Verify the phase against its actual acceptance criteria. + +Do not declare completion because code merely runs. + +For implementation phases, check: + +* relevant tests +* expected artifacts +* configuration reproducibility +* no unintended baseline changes +* no data leakage +* no source-folder modifications + +For research/planning phases, check: + +* evidence is documented +* unknowns remain marked as unknown +* the handoff is current + +--- + +# 29. TERMINAL OUTPUT + +Keep terminal summaries concise. + +Long reports belong in files. + +At the end of substantial work, terminal output should usually summarize: + +* what changed +* what was verified +* tests/checks performed +* known blockers +* handoff location +* exact next task + +Do not rely on terminal scrollback as project documentation.