thesis-prototype-agregation/README.md

PROTEAN Prototype Aggregation Study

Research code and audited artifacts for a master's thesis on server-side prototype aggregation in PROTEAN (Chennoufi et al., 2025), a federated intrusion-detection framework, under non-IID client data. The study evaluates a beta-controlled, class-specific sample-support weighting of local class prototypes for rare attack detection.

Research status. The experimental grid is complete (108 full federated runs), the results are synthesized and audited, and the thesis proposal Markdown is finalized. No further training is planned unless new academic feedback explicitly requires it. The repository is in a documentation and thesis-writing stage.

Final title. Optimasi Agregasi Prototype pada PROTEAN melalui Pembobotan Berbasis Dukungan Sampel Kelas untuk Deteksi Serangan Langka pada Data Non-IID. Here "Optimasi" means empirical selection of beta among the predefined candidate values, not a global mathematical optimum.

---

1. Research Overview

PROTEAN is a federated intrusion-detection method that shares class prototypes between clients and the server in addition to model parameters. Each client i builds a local prototype C_ij for class j from its own training samples; the server aggregates these into a global prototype Cbar_j that is used both for local prototype alignment during training and as the classifier at inference (nearest prototype in embedding space).

This thesis modifies only the server-side prototype aggregation. The original PROTEAN aggregation gives every contributing client equal weight. The proposed aggregation weights each client's prototype by N_ij^beta, where N_ij is that client's class-specific training support and beta is a single exponent in [0, 1]. Everything else — local prototype formation, model aggregation, architecture, loss, and inference rule — is unchanged relative to the reproduced baseline.

N_ij is class-specific sample support, not prototype quality. A client with few samples is not assumed to have a worse prototype.

2. Problem Statement and Motivation

Federated IDS data is non-IID: clients observe different traffic mixtures and different attack types, so class support varies strongly across clients. Rare attack classes appear in little of the data, so aggregate metrics can look acceptable while the model fails on exactly the classes that matter for detection.

Under the original equal-weight prototype aggregation, a prototype built from a handful of samples contributes as much as one built from thousands. This thesis investigates whether — and under which realized non-IID conditions — weighting contributions by class-specific sample support changes the global prototype and rare-class detection behavior.

The study deliberately isolates one mechanism so that any observed change can be attributed to the aggregation weighting rather than to a confounded set of architectural or training changes.

3. Original PROTEAN versus Proposed Modification

A. Local prototype (unchanged). For class j with N_ij > 0 samples on client i, the local prototype is the mean embedding of that client's class-j samples:

C_{ij} = \frac{1}{N_{ij}} \sum_{k=1}^{N_{ij}} z_{ijk}

where z_ijk is the embedding of the k-th sample of class j on client i.

B. Original server-side prototype aggregation (reference). PROTEAN Eq. (2) averages the M contributing clients equally (equal contribution, no N_ij weight):

\bar{C}_j = \frac{1}{M} \sum_{i=1}^{M} C_{ij}

C. Proposed beta-controlled aggregation. Only the prototype aggregation is modified:

\bar{C}_j(\beta) = \frac{\sum_{i \in S_j} N_{ij}^{\beta}\, C_{ij}}{\sum_{i \in S_j} N_{ij}^{\beta}}, \qquad \beta \in \{0,\ 0.25,\ 0.50,\ 0.75,\ 1.00\}

D. Normalized contribution weight. Client i's normalized weight for class j is

w_{ij}(\beta) = \frac{N_{ij}^{\beta}}{\sum_{k \in S_j} N_{kj}^{\beta}}

E. Symbol meanings.

Symbol	Meaning
N_ij	number of local training samples of class j on client i (class-specific sample support, not prototype quality)
C_ij	client i's local class-j prototype (mean embedding)
S_j	participant set: clients with N_ij > 0 and a valid local prototype for class j
M	number of participants in the reference Eq. (2) (equal average)
beta	scalar weighting exponent in [0, 1], one value for all classes and clients
w_ij	normalized contribution weight (sums to 1 over S_j)
global prototype	Cbar_j(beta); used for alignment and nearest-prototype inference

F. Exact difference between original and modified PROTEAN.

• With beta = 0, N_ij^0 = 1, so the proposed aggregator becomes the equal mean over S_j.
• With beta = 1, the contribution is proportional to class support (equivalent to the mean of all included local class embeddings when the assumptions hold).
• beta = 0 is equal weighting over S_j; it is equivalent to the original formulation only if the participant and missing-class rules are also equivalent. That equivalence is not verified against official PROTEAN code.
• Model aggregation, local prototype formation, architecture, loss, and inference are not modified.

Novelty boundary. Sample-count prototype weighting is not new by itself; related work such as FedProto (Tan et al., 2022) already involves class sample counts in prototype-based federated learning. The contribution here is the controlled beta parameterization inside PROTEAN's server-side prototype aggregation, its condition-wise validation-based selection, the paired multi-seed multi-condition evaluation, and the resulting behavioral findings — not the invention of support weighting.

Analysis chain (not a causal proof by itself).

support imbalance -> contribution weight -> global-prototype displacement
                  -> rare-class detection -> per-client effect

4. Research Questions and Objectives

ID	Research question
RQ1	How are class-support imbalance, differences between local prototypes, and rare-class detection performance related in PROTEAN?
RQ2	How does changing beta affect the global prototype and rare-class detection performance under different non-IID conditions?
RQ3	Does the validation-selected beta change across non-IID levels, and what is its effect on overall and per-client performance?

Objectives: (1) relate support imbalance to prototype behavior and rare-class detection (RQ1); (2) characterize the effect of beta on the global prototype and rare-class detection across conditions (RQ2); (3) select beta by validation at different imbalance levels and quantify overall and per-client consequences (RQ3).

5. Experimental Design

Item	Value
Base method	PROTEAN federated intrusion detection
Modified component	server-side prototype aggregation only
Mechanism	w_ij = N_ij^beta, one beta for all classes/clients
Datasets	X-IIoTID (class2, benign Normal, K=9); 5G-NIDD (Attack Type, benign Benign, K=9)
Clients	10
Non-IID conditions	symmetric Dirichlet alpha = 0.25, 0.50, 0.75
Seeds	0, 1, 2 (project-defined)
Beta candidates	0, 0.25, 0.50, 0.75, 1.00
Baseline	original_protean (equal-weight prototype aggregation)
Rounds / local epochs / batch	10 / 3 / 256
Optimizer / lr / mu / lambda	Adam / 1e-3 / 0.1 / 1.0
Splits	stratified train / validation / test = 70 / 10 / 20
Selection metric	pooled validation per-client-mean rare-class macro-F1 across seeds
Test usage	read only after the selected beta is fixed

Expected experimental matrix: 2 datasets × 3 alpha × 3 seeds × (1 baseline + 5 betas) = 108 runs. Within each (dataset, alpha, seed) all six configurations share one manifest, split, class mapping, and round-0 initialization; only the aggregation mode changes.

6. Completed Research Phases

Documented phases, in order. Session numbers are used only where the repository documents them.

Phase / session	Objective	Key outputs	Validation
Planning and architecture	freeze scope, protocol, and project structure	AGENTS.md, planningawal/PROTEAN_Revisi_Lurus.md, docs/research/IMPLEMENTATION_PLAN.md	checkpoint commit
Baseline reproduction	implement reproducible PROTEAN baseline pipeline	docs/baseline/BASELINE_IMPLEMENTATION.md, src/protean_research/	docs/baseline/BASELINE_VALIDATION.md
Baseline validation + regression tests	verify baseline behavior and invariants	tests/, docs/baseline/PROTEAN_BASELINE_INVESTIGATION.md	unit/integration tests
Support-weighted aggregation	implement beta-controlled aggregator	docs/research/BETA_AGGREGATION_IMPLEMENTATION.md, src/protean_research/aggregation/	aggregation invariance tests
Session 5 — one paired condition	first full paired beta run (single seed)	docs/research/PAIRED_BETA_VALIDATION.md	val_selection.json
Session 6 — multi-seed (X-IIoTID a0.50)	pooled validation across seeds {0,1,2}	docs/research/MULTISEED_BETA_VALIDATION.md, SESSION6_AUDIT.md	audit PASS with minor corrections
Session 7 — X-IIoTID a0.25	third X-IIoTID alpha condition	docs/research/ALPHA025_BETA_VALIDATION.md, SESSION7_AUDIT.md	audit PASS with minor corrections
Session 8 — X-IIoTID a0.75	complete X-IIoTID three-alpha study	docs/research/ALPHA075_BETA_VALIDATION.md, SESSION8_AUDIT.md	audit PASS
Session 9 — 5G-NIDD a0.50	first 5G-NIDD condition	docs/research/5GNIDD_ALPHA050_BETA_VALIDATION.md, SESSION9_AUDIT.md	audit PASS
Session 10 — 5G-NIDD a0.75	second 5G-NIDD condition	docs/research/5GNIDD_ALPHA075_BETA_VALIDATION.md, SESSION10_AUDIT.md	audit PASS
Session 11 — 5G-NIDD a0.25	final condition; completes the grid	docs/research/5GNIDD_ALPHA025_BETA_VALIDATION.md, SESSION11_AUDIT.md	audit PASS
Session 12 — synthesis	consolidate all six conditions	docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md, CROSS_DATASET_BETA_ANALYSIS.md, docs/thesis/*, artifacts/analysis/final__*, artifacts/figures/final/*	independent recomputation, zero mismatches
Deep paired comparison	original vs proposed per condition	docs/research/ORIGINAL_VS_PROPOSED_DEEP_COMPARISON.md, artifacts/analysis/deepcmp__*.csv, scripts/deep_comparison.py, scripts/make_comparison_figures.py	manifest-hash pairing checks in code
Post-hoc statistical audit	exploratory paired tests, selection stability	docs/research/POSTHOC_STATISTICAL_AUDIT.md, artifacts/analysis/posthoc__*.csv, scripts/posthoc_statistical_audit.py	manifest-hash pairing re-verified in code
Proposal phase	write the thesis proposal and audit aids	docs/proposal/*	committed e27c5a8
Defense preparation	robustness audit and examiner Q&A aids	kisikisithesis/ROBUSTNESS_AUDIT_DAN_PERSIAPAN_PENGUJI.md, kisikisithesis/robustness_checks.py	read-only checks over the actual implementation and persisted snapshots
Handoff / README	keep repository state current	docs/handoff/SESSION_HANDOFF.md, README.md	this document

7. Experiment Coverage and Reproducibility

Verified run inventory. Counted from results/ metadata, not from filenames alone:

• non-smoke run directories: 108
• distinct (dataset, alpha, seed) cells: 18 (2 × 3 × 3)
• configurations per cell: 1 original_protean + 5 support_weighted (beta 0/0.25/0.50/0.75/1.00)
• paired comparison: within every cell, all six runs share one manifest content hash (verified); original_protean and support_weighted beta=0 additionally share a byte-identical round-0 snapshot
• failed runs: none; one unrelated smoke directory is excluded

Known artifact gap. One run, xiiotid__alpha0.5__seed0__original_protean, lacks its entire validation-side artifact set (val_global_metrics.json, val_confusion_matrix.json, val_per_class.csv, val_per_client.csv, val_selection.json). Its test artifacts and history are present, and it is not a selection run, so all 108 test evaluations and all six selections are unaffected. Every other run has the complete test and validation artifact set. This is recorded as an evidence gap rather than silently omitted.

Reproducibility controls.

• Frozen protocol: docs/research/EXPERIMENT_PROTOCOL.md (machine-readable configs/experiments/experiment_protocol.yaml). Values are either FROZEN from the paper or explicit PROJECT DECISIONs.
• Frozen manifests: 18 deterministic Dirichlet partition manifests under artifacts/partitions/; generated once and reused. Regenerating a fresh split for a run is a protocol violation.
• Raw data: external and read-only (~/datasets/…); never modified, copied, or committed. SHA-256 hashes are recorded in the audit documents.
• Validation/test separation: beta is selected from validation only; every run records test_used_for_selection = false.
• Seeds control split, Dirichlet partition, model initialization, and batch order.

8. Evaluation Metrics

All metrics below are implemented in src/protean_research/evaluation/metrics.py and persisted per run. No scikit-learn classification_report is used; metrics are pure NumPy.

Metric	Definition	Level	Split	Where stored
Accuracy	trace(CM) / sum(CM)	global	val, test	{split}_global_metrics.json
Macro-F1	unweighted mean of per-class F1 over classes with support > 0	global	val, test	{split}_global_metrics.json
Weighted-F1	support-weighted mean of per-class F1	global	val, test	{split}_global_metrics.json
Per-class precision/recall/F1/support	one-vs-rest from the confusion matrix	per class (global)	val, test	{split}_per_class.csv
Rare-class macro-F1	mean per-class F1 over the evaluated rare classes (union of client rare classes with support)	global	val, test	{split}_global_metrics.json
Rare-class recall	mean recall over the evaluated rare classes	global	val, test	{split}_global_metrics.json
Rare support	number of eval samples in evaluated rare classes	global	val, test	{split}_global_metrics.json
Per-client accuracy/macro-F1	standard metrics on each client's own eval partition	per client	val, test	per_client.csv, val_per_client.csv
Per-client rare-class macro-F1 / recall	rare metrics using that client's own rare class ids	per client	val, test	per_client.csv, val_per_client.csv
Selection score	mean over eligible clients of per-client validation rare-class macro-F1, pooled across seeds	per condition	validation	val_selection.json, aggregation artifacts
Prototype displacement	‖Cbar_j(beta) − Cbar_j(0)‖ and relative displacement, fixed round-0 snapshot	per class	n/a	*__prototype_effect*.csv
Normalized weight	N_ij^beta / Σ_k N_kj^beta	per client-class	n/a	*__prototype_weights.csv
Support CV / missing cells / client-size ratio	realized non-IID statistics from N_ij	per class / partition	training	characterization.csv, *partition_heterogeneity*

Averaging-level distinctions that must not be interchanged.

• Global rare-class macro-F1 pools all rare-class evaluation samples; per-client mean rare-class macro-F1 averages each client's own score. The selection metric is the per-client mean, pooled across seeds.
• Validation metrics choose beta; test metrics are reported only after selection.
• Per-seed values and pooled values are different quantities; the pooled score averages per-seed per-client-mean scores.

Not computed by this repository: micro-F1, ROC/AUC, and persisted per-class unseen-class recall (unseen-class evaluation is defined in code but only the unseen flags and union counts are saved).

Exploratory statistical audit (no inferential claims). scripts/posthoc_statistical_audit.py (read-only) computes paired per-seed deltas for the selected beta and exploratory exact tests (two-sided sign-flip permutation, exact Wilcoxon signed-rank) plus paired-t 95% CIs — all with explicit small-n limitations: three seeds give a minimum attainable sign-flip p of 0.25, so no statistical significance is claimed anywhere. Outputs: artifacts/analysis/posthoc__*.csv; full discussion in docs/research/POSTHOC_STATISTICAL_AUDIT.md.

9. Quantitative Experimental Results

All test values are means over seeds {0,1,2} for the selected beta, and the corresponding original_protean test means. Values are from artifacts/analysis/final__six_condition_master.csv and final__selected_vs_original.csv, and were independently re-checked against the per-run test_global_metrics.json.

9.1 Pooled validation selection (validation-only)

Pooled validation per-client-mean rare-class macro-F1 (selected in bold):

beta	X-IIoTID a0.25	X-IIoTID a0.50	X-IIoTID a0.75	5G-NIDD a0.25	5G-NIDD a0.50	5G-NIDD a0.75
0.00	0.5917	0.5444	0.4717	0.7615	0.6872	0.7781
0.25	0.5887	0.5554	0.4744	0.7493	0.6451	0.7905
0.50	0.5859	0.5673	0.4767	0.7257	0.6747	0.8091
0.75	0.5795	0.5794	0.4725	0.7222	0.6884	0.7915
1.00	0.5321	0.5735	0.4686	0.7376	0.6924	0.8043

9.2 Selected beta versus original PROTEAN (test)

Test rare-class macro-F1 (mean over 3 seeds), selected beta vs original_protean:

Dataset	alpha	selected beta	test rare-F1 (selected)	test rare-F1 (original)	delta	test macro-F1 (selected)	test accuracy (selected)
X-IIoTID	0.25	0.00	0.6321	0.6328	−0.0007	0.6813	0.8740
X-IIoTID	0.50	0.75	0.6655	0.6425	+0.0230	0.7312	0.9064
X-IIoTID	0.75	0.50	0.6052	0.6039	+0.0014	0.7576	0.9281
5G-NIDD	0.25	0.00	0.8412	0.8423	−0.0010	0.8166	0.7302
5G-NIDD	0.50	1.00	0.8720	0.8805	−0.0086	0.8386	0.7433
5G-NIDD	0.75	0.50	0.8967	0.8799	+0.0168	0.8531	0.7523

Reading of the table:

• The six conditions split into three mutually exclusive categories that total six: three positive deltas (X-IIoTID a0.50, X-IIoTID a0.75, 5G-NIDD a0.75), one negative delta (5G-NIDD a0.50), and two effectively unchanged (both alpha = 0.25 conditions, where the selected beta is 0, so the selected run is the equal-weight run and the difference is floating-point divergence only).
• The largest positive delta (+0.0230, X-IIoTID a0.50) is still smaller than that condition's between-seed sample std (0.0267). No statistical superiority is claimed.
• No single beta dominates across conditions, and the selection depends on both dataset and condition.

Cross-dataset metric levels are not comparable: 5G-NIDD rare classes are low-volume flood/scan types, X-IIoTID rare classes are low-volume behavioural categories, and their rare support differs.

9.3 Selected betas

Dataset	alpha=0.25	alpha=0.50	alpha=0.75
X-IIoTID	0.00	0.75	0.50
5G-NIDD	0.00	1.00	0.50

10. Beta Selection by Dataset and Alpha

Selection is by pooled validation per-client-mean rare-class macro-F1 (validation only); the test set is never used. artifacts/analysis/final__selection_strength.csv quantifies separation:

Dataset	alpha	selected beta	best	2nd best	abs gap	top-2 gap / selected std	per-seed diagnostic betas
X-IIoTID	0.25	0.00	0.5917	0.5887 (0.25)	0.0030	0.034	0.75 / 0.50 / 0.00
X-IIoTID	0.50	0.75	0.5794	0.5735 (1.00)	0.0059	0.078	0.50 / 1.00 / 0.75
X-IIoTID	0.75	0.50	0.4767	0.4744 (0.25)	0.0024	0.051	0.50 / 0.50 / 0.00
5G-NIDD	0.25	0.00	0.7615	0.7493 (0.25)	0.0122	0.556	0.25 / 0.00 / 0.00
5G-NIDD	0.50	1.00	0.6924	0.6884 (0.75)	0.0040	0.029	1.00 / 0.00 / 1.00
5G-NIDD	0.75	0.50	0.8091	0.8043 (1.00)	0.0048	0.081	0.50 / 1.00 / 1.00

In all six conditions the top-two gap is smaller than the selected beta's across-seed sample std (ratios 0.03–0.56), so each selection is a small numerical separation relative to seed variability. No tie-break was ever exercised. Per-seed diagnostic betas are shown only for stability; they are not the selection rule.

11. Prototype-Level Analysis

Direct aggregation effect, measured on fixed round-0 snapshots (C_ij, N_ij held constant, no retraining). artifacts/analysis/final__prototype_displacement_by_beta.csv.

Mean ‖Cbar_j(beta) − Cbar_j(0)‖ across classes:

Dataset	alpha	beta=0.25	beta=0.50	beta=0.75	beta=1.00
X-IIoTID	0.25	1.476	2.403	3.003	3.422
X-IIoTID	0.50	0.826	1.374	1.780	2.099
X-IIoTID	0.75	0.477	0.829	1.097	1.305
5G-NIDD	0.25	1.295	2.055	2.559	2.919
5G-NIDD	0.50	0.953	1.567	2.003	2.340
5G-NIDD	0.75	0.497	0.875	1.163	1.385

• Displacement is monotone increasing in beta for a fixed snapshot in all six conditions (mechanically expected: larger beta shifts weight toward high-N_ij clients).
• Displacement at beta = 1 decreases as alpha rises, tracking realized support imbalance.
• At matched alpha, X-IIoTID has larger displacement at a0.25 (3.422 vs 2.919) while 5G-NIDD is slightly larger at a0.50 (2.340 vs 2.099) and a0.75 (1.385 vs 1.305); per-class ordering is seed- and dataset-specific, and rare-class identities differ.
• Displacement is a representation-aggregation metric, not detection quality. No displacement-to-accuracy inference is made.

Normalized contribution weights are stored per client-class in artifacts/analysis/*__prototype_weights.csv; per-class displacement is in final__prototype_displacement_by_class.csv.

12. Rare-Class Detection Results

Rare classes are defined from each client's training support (the two attack classes with the smallest positive N_ij; one if only one; the client is excluded if none). Unseen classes (N_ij = 0) are treated separately and never merged with rare classes.

Key observations from the test results (Section 9.2) and the synthesis:

• Selected rare-class macro-F1 deltas are mixed in sign and small; the largest positive difference is within seed noise.
• Selected rare-class recall is slightly lower than original in most conditions while macro-F1 may be higher: weighting can trade a little rare-class recall for precision.
• At both alpha = 0.25 conditions the selected beta is 0, so those conditions cannot demonstrate any support-weighting benefit.
• Full rare-class curves and per-condition reports: docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md, the per-condition *_BETA_VALIDATION.md reports, and final__selected_vs_original.csv.

13. Per-Client Analysis

Per-client effects are computed on each client's own evaluation partition relative to beta = 0 (artifacts/analysis/final__client_effect_summary.csv; detail in the per-condition *__multiseed__client_validation_*.csv).

Improve / degrade / unchanged clients at the selected beta (validation, vs beta=0), by seed:

Condition (selected beta)	seed 0	seed 1	seed 2
X-IIoTID a0.25 (0.00)	0 / 0 / 10	0 / 0 / 10	0 / 0 / 10
X-IIoTID a0.50 (0.75)	7 / 3 / 0	8 / 1 / 1	8 / 1 / 1
X-IIoTID a0.75 (0.50)	6 / 4 / 0	9 / 0 / 1	6 / 4 / 0
5G-NIDD a0.25 (0.00)	0 / 0 / 10	0 / 0 / 10	0 / 0 / 10
5G-NIDD a0.50 (1.00)	6 / 3 / 1	3 / 5 / 2	7 / 2 / 1
5G-NIDD a0.75 (0.50)	4 / 6 / 0	6 / 3 / 1	4 / 2 / 4

• Weighting redistributes benefit across clients: within a condition some clients improve while others degrade.
• At beta = 1, mean per-client delta is most negative at both alpha = 0.25 conditions (e.g. X-IIoTID a0.25 ≈ −0.060, 5G-NIDD a0.25 ≈ −0.024), consistent with those conditions selecting beta = 0.
• N_ij versus per-client delta is non-monotonic: the worst-degrading client is not always the lowest-support client.
• Client identities are per-seed realizations and are not pooled across seeds.

14. Cross-Alpha and Cross-Dataset Findings

• Matched-alpha selections agree at alpha=0.25 (both 0.00) and alpha=0.75 (both 0.50) and differ at alpha=0.50 (X-IIoTID 0.75 vs 5G-NIDD 1.00). Across the six conditions the selected beta takes values {0.00, 0.50, 1.00} — not universal, not monotone in alpha, not a function of alpha alone.
• Realized heterogeneity (support CV, missing client-class cells, client-size ratio) decreases as alpha rises within each dataset, but not strictly per seed. Dirichlet alpha is a design knob, not the realized heterogeneity itself.
• Displacement at beta = 1 falls with alpha on both datasets.
• Rare-set composition differs by dataset, so absolute metric levels are not comparable across datasets; only within-condition differences are meaningful.
• No cross-dataset causality, superiority, or dataset-independent beta law is claimed.

Reports: docs/research/CROSS_DATASET_BETA_ANALYSIS.md, XIIOTID_CROSS_ALPHA_ANALYSIS.md, 5GNIDD_CROSS_ALPHA_ANALYSIS.md.

15. Confusion Matrix and Classification Breakdown

Confusion matrices do exist in the saved artifacts. They are written per run by the evaluation code (confusion_matrix in src/protean_research/evaluation/metrics.py) and saved as:

• results/<run>/test_confusion_matrix.json — 108 / 108 runs
• results/<run>/val_confusion_matrix.json — 107 / 108 runs (missing only for xiiotid__alpha0.5__seed0__original_protean, which also lacks the other validation artifacts)

Properties of the stored matrices.

Property	Value
Shape	9 × 9 (K = 9 classes)
Values	raw integer counts (not normalized)
Scope	global over the evaluation split, not per-client
Restriction	full multiclass; not restricted to rare classes
Split	separate test_ and val_ matrices
Class order	X-IIoTID: C&C, Exfiltration, Exploitation, Lateral_movement, Normal, RDOS, Reconnaissance, Tampering, Weaponization; 5G-NIDD: Benign, HTTPFlood, ICMPFlood, SYNFlood, SYNScan, SlowrateDoS, TCPConnectScan, UDPFlood, UDPScan (also in each run's metadata.json)
Aggregation	each client's eval partition concatenated; no seed/client mixing

Representative interpretation — 5G-NIDD, alpha=0.75, seed=0, test, original vs selected (beta=0.50) — using actual counts:

Class	eval support	original recall	selected-beta recall
Benign	95,547	0.3795	0.3825
HTTPFlood	28,162	0.9729	0.9923
ICMPFlood	231	1.0000	1.0000
SYNFlood	1,944	0.8570	0.8570
SYNScan	4,009	0.9968	0.9968
SlowrateDoS	14,625	0.9217	0.9054
TCPConnectScan	4,010	0.9948	0.9948
UDPFlood	91,468	1.0000	1.0000
UDPScan	3,181	0.9975	0.9981

Largest off-diagonal cells (true -> predicted):

true	predicted	original count	selected-beta count
Benign	UDPFlood	56,899	57,054
Benign	TCPConnectScan	2,015	1,495
SlowrateDoS	HTTPFlood	1,079	1,308
HTTPFlood	SlowrateDoS	759	212
SYNFlood	TCPConnectScan	278	278

What these counts support:

• The dominant error mode is Benign misclassified as UDPFlood (about 57k samples), which dominates the off-diagonal mass. It is essentially unchanged by beta, so beta does not fix this confusion. This is visible in low Benign recall (≈0.38) with very high UDPFlood recall (1.0000), i.e. UDPFlood is an attractor for Benign.
• The change from original to selected beta shifts a few specific off-diagonal cells (Benign → TCPConnectScan decreases; SlowrateDoS → HTTPFlood increases), consistent with the small, mixed-sign aggregate deltas, not with a systematic improvement.
• Diagonal versus off-diagonal entries must be read together: high diagonal on the rare/scan classes coexists with the large Benign confusion, which is why accuracy is moderate while macro-F1 is higher.

Reconstruction note: because only aggregate per-class metrics (precision/recall/F1/support) are persisted, a confusion matrix cannot be uniquely reconstructed from them; the raw y_true/y_pred arrays are not saved. Reconstruction is unnecessary here because the matrices themselves are already persisted as JSON.

Representative confusion-matrix files:

• results/5gnidd__alpha0.75__seed0__original_protean/test_confusion_matrix.json
• results/5gnidd__alpha0.75__seed0__support_weighted__beta0.5/test_confusion_matrix.json
• results/xiiotid__alpha0.25__seed0__original_protean/test_confusion_matrix.json

16. Final Figures and Visualizations

All figures live in artifacts/figures/final/ and are regenerated from the audited artifacts/analysis/final__* tables by scripts/make_final_figures.py and scripts/make_comparison_figures.py (no training; every number is loaded from an artifact). Each figure is saved as a high-resolution PNG (220 dpi) and a matching vector PDF, and its supporting source table is linked where useful.

Naming and legend conventions used throughout.

• Charcoal (original_protean): the frozen reproduced baseline.
• Teal (support_weighted, validation-selected beta): the proposed method at the beta selected by pooled validation for that condition.
• Gray (beta = 0 control): the equal-weight support_weighted run; a control, not automatically the same recorded configuration as original_protean.
• Alpha conditions are distinguished by colour and marker and line style, so colour is never the only cue.

Comparison types (so figures are not misread).

Figure	Comparison type	Baseline actually plotted
A (both)	Beta candidate comparison within support_weighted	none (support_weighted candidates only)
B	Beta selection surface	none (selection, not performance)
C	Dataset heterogeneity	none (data distribution)
D, E	Fixed-snapshot comparison vs beta = 0	beta = 0 control
F	Direct comparison vs original_protean (test)	original_protean
G	Per-client comparison vs beta = 0 (validation)	beta = 0 control
H	Paired original_protean vs selected beta confusion matrices	original_protean
I1, I2	Per-client test comparison vs original_protean	original_protean
J1, J2	Per-class test comparison vs original_protean	original_protean
K	Ranked confusion change vs original_protean (+ full matrix supplement)	original_protean

Fig A1 / A2 — Validation-based beta selection (support_weighted)

Compared: the five support_weighted beta candidates, separately for each alpha. Level/split: pooled validation, per-client mean across seeds {0,1,2}. Metric: rare-class macro-F1 (per-client mean). Observation: the validation response is non-monotone and the selected beta differs by condition. Limits: this is a selection figure; no original_protean run is plotted, and the error bars are ±1 sample std across three project-defined seeds, not a confidence interval.

Fig B — Validation-selected beta by dataset and alpha

Compared: none (selection only). Level: pooled validation. Metric: selected beta under the selection rule. Observation: selections take {0.00, 0.50, 1.00} and are not universal. Limits: selection surface, not a performance comparison; a higher beta is not automatically better. No original_protean comparison is implied.

Fig C — Realized class-support imbalance

Compared: none (data distribution). Level: training partitions. Metric: mean class-support CV of N_ij across contributing clients (std/mean), averaged over classes and seeds. Observation: realized imbalance decreases as the configured alpha rises. Limits: this describes the data, not the relative performance of original vs modified PROTEAN; Dirichlet alpha is the configuration knob and is not the same as the realized imbalance.

Fig D — Support-weighted aggregation: beta=1 vs beta=0 control

Compared: beta = 1 against the beta = 0 control (not original_protean). Level: fixed round-0 snapshots, same local prototypes and N_ij, no retraining. Metric: mean ‖Cbar_j(1) − Cbar_j(0)‖ across classes. Observation: displacement increases with beta and is largest where realized imbalance is largest. Limits: displacement is a representation metric, not detection quality.

Fig E — Realized support CV vs prototype displacement

Compared: beta = 1 displacement against the beta = 0 control, across conditions. Level: fixed snapshots. Metric: realized support CV (x) vs beta = 1 displacement (y). Observation: larger realized imbalance accompanies larger displacement. Limits: descriptive association only; the imbalance-displacement link is mechanical under the beta formula and is not causal evidence.

Fig F — Selected-beta PROTEAN vs ORIGINAL PROTEAN (test), two panels

Compared: original_protean against support_weighted at the validation-selected beta, per condition, test split. How to read: Panel A shows the absolute mean test rare-class macro-F1 (±1 sample std across seeds) and is deliberately near-flat; Panel B shows the signed difference proposed − original with the exact value printed at each bar and the selected beta annotated. Main result: three positive deltas (X-IIoTID a0.50 +0.0230, X-IIoTID a0.75 +0.0014, 5G-NIDD a0.75 +0.0168), one negative delta (5G-NIDD a0.50 −0.0086), and two beta=0 conditions that are effectively unchanged (−0.0007 / −0.0010). Limitation: no significance test (three seeds); a positive mean delta is not a proven improvement.

Links: PNG · PDF · source final__six_condition_master.csv.

Fig G — Per-client effect of selected beta relative to the beta=0 control

Compared: selected-beta support_weighted against the beta = 0 control (not original_protean). Level/split: validation, per client, per seed. Metric: per-client rare-class macro-F1 delta; a client is "improved"/"degraded" if its delta is above/below 0, "unchanged" otherwise. Observation: within a condition some clients improve and others degrade; at both alpha = 0.25 conditions the selected beta is 0, so all clients are unchanged. Limits: client ids are per-seed realizations and are not pooled.

Fig H — Paired test confusion matrices (original vs beta-tuned)

Compared: original_protean (left) against selected-beta support_weighted (right) for a valid paired condition (5G-NIDD, alpha = 0.75, seed = 0), test split. How to read: rows = actual class, columns = predicted class; the diagonal is correct and off-diagonal is misclassified. Raw counts, identical class order and colour scale; both matrices contain the same 243,177 test samples. Main result: the dominant residual error Benign → UDPFlood is essentially unchanged (56,899 → 57,054, +155). The summary below the figure lists the most important changes (HTTPFlood → SlowrateDoS −547; Benign → TCPConnectScan −520; SlowrateDoS → HTTPFlood +229; diagonal HTTPFlood +547, Benign +288, SlowrateDoS −239). Limitation: no significance test; the Benign → UDPFlood increase is not an improvement.

Links: PNG · PDF · source results/5gnidd__alpha0.75__seed0__original_protean/test_confusion_matrix.json and results/5gnidd__alpha0.75__seed0__support_weighted__beta0.5/test_confusion_matrix.json.

17. Deep Comparison: Original PROTEAN vs Proposed (Result)

A dedicated paired analysis compares original_protean against support_weighted at the validation-selected beta for every dataset-alpha condition, using the tests split and hard-coded pairing checks (same dataset, alpha, seed, and manifest_content_hash). Full report: docs/research/ORIGINAL_VS_PROPOSED_DEEP_COMPARISON.md. Derived tables: artifacts/analysis/deepcmp__*.csv (produced by scripts/deep_comparison.py). Figures: scripts/make_comparison_figures.py.

Global test overview (Δ = proposed − original, mean over seeds {0,1,2}; rare-class macro-F1 shown in full, other metrics as deltas):

Dataset	alpha	beta*	rare-F1 original	rare-F1 proposed	Δ rare-F1	Δ rare-recall	Δ macro-F1	Δ accuracy
X-IIoTID	0.25	0.00	0.6328	0.6321	−0.0007	−0.0007	−0.0007	−0.0014
X-IIoTID	0.50	0.75	0.6425	0.6655	+0.0230	−0.0144	+0.0222	+0.0069
X-IIoTID	0.75	0.50	0.6039	0.6052	+0.0014	−0.0077	+0.0058	+0.0075
5G-NIDD	0.25	0.00	0.8423	0.8412	−0.0010	+0.0007	−0.0028	−0.0007
5G-NIDD	0.50	1.00	0.8805	0.8720	−0.0086	−0.0058	−0.0038	−0.0007
5G-NIDD	0.75	0.50	0.8799	0.8967	+0.0168	−0.0015	+0.0117	+0.0013

Both alpha=0.25 conditions selected beta=0 and are numerically null. Among the four genuine beta>0 conditions, rare-class macro-F1 improved in three and degraded in one; overall macro-F1 moves with it, while rare-class recall usually falls (precision–recall trade-off). 5G-NIDD a0.50 is seed-dominated. No statistical significance is claimed.

Fig I1 / I2 — Paired per-client test rare-class macro-F1 (proposed vs ORIGINAL PROTEAN)

One figure per dataset, three alpha panels. Compared: original_protean against selected-beta support_weighted on the test split, each client on its own evaluation partition. How to read: x = client id, one marker per seed (circle = seed 0, square = seed 1, triangle = seed 2), y = Δ rare-class macro-F1; the shaded band is the ±0.02 effectively-unchanged threshold; the strongest positive and negative observation in each panel is labelled c<client> s<seed>. Main result: X-IIoTID α=0.25 essentially unchanged (1 / 1 / 28 improved/degraded/unchanged), α=0.50 12 / 1 / 17 (mean +0.030, range −0.257 … +0.324), α=0.75 8 / 2 / 20; 5G-NIDD α=0.25 1 / 0 / 29, α=0.50 15 / 6 / 9 (seed-dominated), α=0.75 8 / 1 / 21. Limitation: client ids are per-seed realizations (id k is not the same entity across seeds); at α=0.25 the selected beta is 0 so the residual spread is numerical, not a weighting effect.

Links: I1 PNG · I1 PDF · I2 PNG · I2 PDF · source deepcmp__per_client.csv.

Fig J1 / J2 — Per-class test F1 change (beta>0 conditions)

One figure per dataset, two alpha panels. Compared: original_protean against selected-beta support_weighted, test split. How to read: horizontal bars are the mean ΔF1 over seeds {0,1,2} (teal = increased, red = decreased) with the exact signed ΔF1 printed at each bar; test support n≈ is on a second label line; both panels share one x-scale so bar lengths are comparable. Main result: X-IIoTID largest gains Tampering +0.1438 and Reconnaissance +0.0542, largest loss C&C −0.0556; 5G-NIDD largest gain TCPConnectScan +0.0342 / ICMPFlood +0.0970 (tiny support), largest loss SYNFlood −0.0426. Limitation: the delta is F1, not precision or recall (e.g. Tampering +0.1438 F1 corresponds to +0.2120 precision and −0.0091 recall); no significance test.

Links: J1 PNG · J1 PDF · J2 PNG · J2 PDF · source deepcmp__per_class.csv.

Fig K — Ranked confusion change vs ORIGINAL PROTEAN (+ full-matrix supplement)

For the paired condition 5G-NIDD, alpha=0.75, seed=0 (identical manifest, split, class order, 243,177 test samples). How to read: the ranked view lists the largest off-diagonal changes as raw test counts; the upper panel shows more errors after weighting, the lower panel fewer. Main result: the largest reduced errors are HTTPFlood → SlowrateDoS (−547; 759 → 212) and Benign → TCPConnectScan (−520; 2,015 → 1,495); the largest increased errors are SlowrateDoS → HTTPFlood (+229) and Benign → UDPFlood (+155). The dominant error Benign → UDPFlood is essentially unchanged and is not an improvement. Limitation: no significance test; the figure is a single paired condition, not a condition average. The full difference matrices (raw count and row-rate) are kept as a supplemental figure for traceability.

Links: ranked PNG · ranked PDF · full-matrix PNG · full-matrix PDF · source deepcmp__confusion_delta.csv.

Per-client findings. The proposed method redistributes benefit: within a condition some clients improve and others degrade, and the low-rare-support quartile is not systematically disadvantaged (descriptive; Pearson Δ-vs-support-fraction ≈ −0.09, n=120). Client ids are per-seed realizations.

Per-class findings. Rare-class gains are partly precision-driven (recall often falls) and are not confined to rare classes; the largest regressions are on low-support attack classes (X-IIoTID C&C, 5G-NIDD SYNFlood).

Confusion-delta findings. The weighting does not fix the dominant Benign→UDPFlood error; the measurable benefit at the analysed pair is a reduced HTTPFlood↔SlowrateDoS confusion, with a small SlowrateDoS recall cost.

Practical implications and limitations. Effects are small (≤ ~2.3 pp rare-F1) and condition-dependent; the largest gain is comparable to seed noise. Practical benefit is modest, not a blanket improvement, and comes with rare-recall and some client- or class-level costs. Pairing, metric definitions, and threats to validity are detailed in the report.

18. Main Scientific Findings

1. Changing beta changes the global prototype. Contribution weights become proportional to support, and for a fixed snapshot mean prototype displacement grows monotonically with beta in every condition.
2. The rare-class response is condition-dependent and non-monotone in beta. The direct aggregation effect (geometry) and the end-to-end training effect (detection) are distinct.
3. No universal beta was found. The validation-selected beta differs across datasets and conditions and is not monotone in alpha. Every selection is weak relative to seed variability.
4. Selected-vs-original differences are small and mixed in sign. No statistical superiority is established; the largest positive difference is within seed noise.
5. Weighting redistributes benefit across clients. Within a condition some clients improve while others degrade; the effect is not simply "more support is better".
6. Where weighting does not help: both alpha = 0.25 conditions select beta = 0 (equal weighting), and at beta = 1 the per-client effect is most damaging there.
7. Realized support imbalance and prototype displacement co-vary descriptively; displacement is not a proxy for detection quality.
8. The dominant classification error (Benign → UDPFlood) is not addressed by beta.

19. Limitations and Novelty Boundary

Limitations.

• The S_j participant/denominator rule is a documented reproduction assumption (no verified official PROTEAN code). It is material in up to 19/27 class-condition cells; all results are conditional on it.
• beta = 0 is only arithmetically the equal mean over S_j; it is not bitwise identical to original_protean (floating-point divergence; equivalence class B in every condition).
• Two datasets, three alpha values, three project-defined seeds, 10 clients, one architecture.
• The beta grid is discrete; intermediate values were not tested.
• Three seeds are underpowered for inferential testing; no significance test is claimed.
• Prototype displacement is a representation metric, not detection quality.
• One validation artifact set is missing (Section 7); rare-class and unseen-class results cannot be merged, and cross-dataset absolute metric levels are not comparable.

Novelty boundary. Support-weighted prototype aggregation is not new (e.g. FedProto). The contribution is the controlled beta parameterization inside PROTEAN, the validation-only condition-wise selection, the paired multi-seed design, and the behavioral findings. No claim is made that the method universally improves PROTEAN.

20. Repository Structure

src/protean_research/   data loading/preprocessing/partitioning, models, aggregation, federated
                        round logic, evaluation metrics, analysis, utils
scripts/                training runner, partition generation, and read-only analysis scripts
configs/                dataset, experiment protocol, and model configuration (YAML)
docs/proposal/          thesis proposal and internal audit aids
docs/research/          frozen protocol, per-condition reports, cross-alpha/cross-dataset synthesis,
                        post-hoc statistical audit, session audits
docs/baseline/          PROTEAN baseline investigation, implementation, and validation
docs/thesis/            Chapter 4 blueprint and claims register
docs/handoff/           session handoff (start here for a new session)
kisikisithesis/         robustness audit and examiner-preparation aids (read-only checks)
planningawal/           immutable research specification (do not modify)
artifacts/partitions/   deterministic frozen Dirichlet partition manifests (18)
artifacts/analysis/     committed machine-readable analysis outputs
artifacts/figures/final/ thesis-ready figures
results/                local run outputs (git-ignored; 108 non-smoke run dirs + 1 smoke)

21. Reproduction and Tests

# unit + integration tests (expected: 75 passed)
.venv/bin/python -m pytest

# read-only pooled validation analysis (no retraining)
.venv/bin/python scripts/analyze_multiseed_beta.py --dataset 5gnidd --alpha 0.25 --seeds 0 1 2

# read-only realized-partition characterization
.venv/bin/python scripts/analyze_partition_heterogeneity.py --dataset 5gnidd --alphas 0.25 0.5 0.75

# read-only cross-alpha synthesis
.venv/bin/python scripts/analyze_cross_alpha.py --dataset 5gnidd --alphas 0.25 0.5 0.75

# beta=0 vs original equivalence audit (read-only)
.venv/bin/python scripts/verify_beta0_equivalence.py --dataset 5gnidd --alpha 0.25 --seeds 0 1 2

# post-hoc statistical audit (read-only; writes only posthoc__*.csv)
.venv/bin/python scripts/posthoc_statistical_audit.py

# deterministic robustness checks over the actual implementation (prints only)
.venv/bin/python kisikisithesis/robustness_checks.py

A full training run (not needed; the grid is complete) is:

.venv/bin/python scripts/run_baseline.py --dataset 5gnidd --alpha 0.25 --seed 1 \
    --clients 10 --rounds 10 --local-epochs 3 \
    --aggregation-mode support_weighted --beta 1.0

Runtime context: the 108 runs consumed ≈ 94.5 wall-clock hours on CPU (8 cores); per-run wall clock was ≈ 62–65 minutes for 5G-NIDD and ≈ 40 minutes for X-IIoTID (*__multiseed__runtime.csv). Do not rerun the grid.

Repository safety.

• planningawal/ is immutable; raw datasets are external and read-only.
• Do not modify or regenerate completed results, manifests, or figures.
• Do not add an equal_available mode; it is arithmetically support_weighted at beta = 0 under the shared S_j.
• beta selection uses validation only; the test set is never used for selection.

22. Complete Artifact Index

Per-run outputs (results/<dataset>__alpha<alpha>__seed<seed>__<config>/):

File	Content
metadata.json	run config, dataset/alpha/seed, mode, class names, hashes
history.json	per-round global accuracy/macro-F1/rare-class metrics
characterization.csv	per-class training support statistics (N_ij)
train_support_Nij.csv	N_ij matrix (client × class)
rare_unseen.csv	per client-class rare/unseen flags from training support
test_global_metrics.json, val_global_metrics.json	global metrics per split
test_per_class.csv, val_per_class.csv	per-class precision/recall/F1/support
per_client.csv, val_per_client.csv	per-client metrics
test_confusion_matrix.json, val_confusion_matrix.json	9×9 raw-count confusion matrices
val_selection.json	validation selection metric and test_used_for_selection flag

For the one incomplete validation run, see Section 7.

Analysis outputs (artifacts/analysis/), by function:

Category	Representative files
Final condition summary	final__six_condition_master.csv/.json
Beta-selection curves	final__pooled_validation_curves.csv, final__selection_strength.csv
Selected vs original	final__selected_vs_original.csv
Realized heterogeneity	final__realized_heterogeneity.csv, *__partition_heterogeneity_*.csv
Prototype displacement	final__prototype_displacement_by_beta.csv, ..._by_class.csv
Per-client effects	final__client_effect_summary.csv, *__multiseed__client_validation_*.csv
Per-condition validation	*__multiseed__validation_per_seed.csv, ..._validation_pooled.csv
Per-condition test	*__multiseed__test_descriptive.csv
Cross-alpha	{xiiotid,5gnidd}__cross_alpha__*.csv/.json
Cross-dataset	cross_dataset__matched_alpha_summary.csv/.json, cross_dataset__beta_curve.csv
Post-hoc audit	posthoc__paired_global_per_seed.csv, posthoc__paired_global_by_condition.csv, posthoc__paired_test_results.csv, posthoc__selection_stability.csv, posthoc__per_class_summary.csv, posthoc__per_client_summary.csv
beta=0 equivalence	*__multiseed__beta0_equivalence.csv
Runtime	*__multiseed__runtime.csv
Prototype snapshots & weights	*__prototype_snapshot.json, *__prototype_weights.csv, *__prototype_effect*.csv

Partition manifests: artifacts/partitions/<dataset>__alpha<alpha>__seed<seed>.json (18 files). Figures: artifacts/figures/final/ — 15 figures (figA–figK, including the split I1/I2 and J1/J2 and the K full-matrix supplement) as PNG (220 dpi) plus matching vector PDF; regenerated reproducibly by scripts/make_final_figures.py.

Key documents:

Document	Purpose
docs/research/EXPERIMENT_PROTOCOL.md	frozen reproduction protocol
docs/research/FINAL_EXPERIMENTAL_SYNTHESIS.md	authoritative six-condition synthesis
docs/research/ORIGINAL_VS_PROPOSED_DEEP_COMPARISON.md	paired original-vs-proposed deep comparison
docs/research/CROSS_DATASET_BETA_ANALYSIS.md	cross-dataset comparison
docs/research/POSTHOC_STATISTICAL_AUDIT.md	exploratory paired statistical audit of the completed grid
docs/research/XIIOTID_CROSS_ALPHA_ANALYSIS.md	X-IIoTID cross-alpha analysis
docs/research/5GNIDD_CROSS_ALPHA_ANALYSIS.md	5G-NIDD cross-alpha analysis
docs/thesis/CHAPTER4_RESULTS_OUTLINE.md	Chapter 4 blueprint
docs/thesis/CLAIMS_REGISTER.md	claim / evidence / status register
docs/proposal/THESIS_PROPOSAL_DRAFT.md	final proposal Markdown source
docs/proposal/PROPOSAL_SOURCE_MAP.md	claim-to-evidence map
docs/proposal/LITERATURE_MATRIX.md	literature matrix
docs/proposal/REFERENCES_VERIFIED.md	bibliography verification record
docs/handoff/SESSION_HANDOFF.md	current state and next steps
planningawal/PROTEAN_Revisi_Lurus.md	immutable research specification

23. References

• Chennoufi, S., Han, Y., Blanc, G., De Cristofaro, E., & Kiennert, C. (2025). PROTEAN: Federated Intrusion Detection in Non-IID Environments through Prototype-Based Knowledge Sharing. ESORICS 2025, pp. 103–125. DOI: 10.1007/978-3-032-07884-1_6. arXiv:2507.05524.
• Tan, Y., Long, G., Liu, L., Zhou, T., Lu, Q., Jiang, J., & Zhang, C. (2022). FedProto: Federated Prototype Learning across Heterogeneous Clients. AAAI 36(8), 8432–8440. DOI: 10.1609/aaai.v36i8.20819. arXiv:2105.00243.

The complete, verified bibliography (including the FL foundations, non-IID, class-imbalance, and evaluation-metric references used in the proposal) is maintained in docs/proposal/REFERENCES_VERIFIED.md and the proposal bibliography. This README cites only the two works directly relevant to the mechanism and novelty boundary.