feat: implement support-weighted prototype aggregation
This commit is contained in:
willtanoe committed
1 parent
16b3add2cf
commit
ad9aa53ba7
16 files changed
+1220
-234
No files matched your search
@@ -0,0 +1,176 @@
|
||||
# Beta-Weighted Prototype Aggregation — Implementation
|
||||
|
||||
Status: **IMPLEMENTED and unit/smoke-validated** (Session 4, 2026-09-24). The
|
||||
proposed class-specific sample-support weighting is now available as an
|
||||
independent aggregation mode. The frozen `original_protean` baseline path is
|
||||
unchanged. **No scientific beta sweep was run**; only small smoke tests.
|
||||
|
||||
Companion documents:
|
||||
|
||||
- `docs/baseline/BASELINE_IMPLEMENTATION.md` — baseline semantics and provenance.
|
||||
- `docs/baseline/BASELINE_VALIDATION.md` — Session-3 baseline validation evidence.
|
||||
- `docs/research/EXPERIMENT_PROTOCOL.md` — frozen protocol.
|
||||
- `docs/handoff/SESSION_HANDOFF.md` — continuation point.
|
||||
|
||||
---
|
||||
|
||||
## 1. Scope
|
||||
|
||||
The only change relative to the validated baseline is the **server-side prototype
|
||||
aggregation**. The data pipeline, partitions, splits, model, local loss, model
|
||||
averaging (`1/M`) and inference are reused unchanged. The proposed rule is:
|
||||
|
||||
```
|
||||
Cbar_j(beta) = ( sum_{i in S_j} N_ij^beta * C_ij ) / ( sum_{i in S_j} N_ij^beta )
|
||||
S_j = { i | N_ij > 0 and a valid local prototype C_ij exists }
|
||||
beta ∈ {0.00, 0.25, 0.50, 0.75, 1.00}, 0 <= beta <= 1
|
||||
```
|
||||
|
||||
`N_ij` is **class-specific sample support**, not a quality score. Larger `N_ij`
|
||||
is not assumed to mean a better prototype (AGENTS.md §9).
|
||||
|
||||
## 2. Code changes
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `src/protean_research/aggregation/prototype_aggregation.py` | Added `AggregationMode.SUPPORT_WEIGHTED`, `validate_beta`, `normalized_support_weights`; extended `aggregate_prototypes` and `aggregate_class_prototypes` with optional `supports`/`beta`. Baseline branch kept arithmetic-identical. |
|
||||
| `src/protean_research/aggregation/__init__.py` | Exports the new mode, helpers and `BETA_CANDIDATES`. |
|
||||
| `src/protean_research/models/prototypes.py` | `collect_global_prototypes` forwards `beta`. |
|
||||
| `src/protean_research/federated/server.py` | `aggregate_prototypes` forwards `mode`/`beta`. |
|
||||
| `src/protean_research/federated/round.py` | `TrainingConfig.aggregation_mode` + `.beta`; the round loop dispatches to the active mode. |
|
||||
| `src/protean_research/analysis/direct_prototype.py` | New direct prototype-effect instrumentation (fixed `{C_ij, N_ij}`). |
|
||||
| `scripts/analyze_prototype_effect.py` | New CLI: snapshot → effect CSV + weights CSV, no retraining. |
|
||||
| `scripts/run_baseline.py` | New `--aggregation-mode`/`--beta`; run name and metadata now record mode + beta. Default remains `original_protean`. |
|
||||
|
||||
### 2.1 Interface
|
||||
|
||||
```python
|
||||
aggregate_prototypes(prototypes, mode="original_protean", supports=None, beta=None)
|
||||
aggregate_class_prototypes(contributions, mode="original_protean", beta=None)
|
||||
normalized_support_weights(supports, beta) # weights summing to 1
|
||||
validate_beta(beta) # strict [0, 1] validation
|
||||
```
|
||||
|
||||
- `original_protean` computes the equal mean over `S_j`. Passing `beta` with it
|
||||
raises `ValueError` (no silent mixing).
|
||||
- `support_weighted` requires strictly positive, aligned `supports` and a
|
||||
`beta ∈ [0, 1]`; `None`, non-numeric, non-finite and out-of-range betas raise
|
||||
`ValueError`.
|
||||
- `S_j` is built once, identically, for both modes: entries with `proto is None`
|
||||
or `N_ij <= 0` are dropped; a class with no valid contributor is omitted. No
|
||||
zero-vector padding or prototype fabrication.
|
||||
|
||||
## 3. Beta semantics and invariants
|
||||
|
||||
| Property | Expected | Test |
|
||||
|---|---|---|
|
||||
| `beta = 0` | equal (arithmetic) mean over `S_j` | `test_beta0_is_equal_arithmetic_mean_over_s_j`, `test_beta0_matches_original_protean_on_identical_s_j` |
|
||||
| `beta = 1` | support-weighted mean | `test_beta1_is_support_weighted_mean` |
|
||||
| intermediate beta | closed-form weighted mean | `test_intermediate_beta_fixture`, `test_intermediate_beta_matches_closed_form` |
|
||||
| single contributor | beta-invariant | `test_single_contributor_is_invariant_to_beta` |
|
||||
| equal support | beta-invariant | `test_equal_support_is_invariant_to_beta` |
|
||||
| missing class | excluded from `S_j`, never fabricated | `test_missing_class_is_excluded_from_s_j`, `test_class_with_only_missing_contributors_is_omitted` |
|
||||
| weights | normalised, sum to 1, positive | `test_normalized_weights_sum_to_one`, `test_beta0_weights_are_uniform` |
|
||||
| invalid beta | rejected | `test_invalid_beta_is_rejected`, `test_none_and_non_numeric_beta_are_rejected` |
|
||||
|
||||
`beta = 0` is **not** labelled original PROTEAN. It is arithmetically identical
|
||||
to the baseline *only because both use the same documented `S_j` participant
|
||||
rule*; the `S_j` rule itself remains a reproduction assumption (§5).
|
||||
|
||||
## 4. Direct prototype-effect instrumentation
|
||||
|
||||
`analysis/direct_prototype.py` reads a round snapshot of fixed `{C_ij, N_ij}`
|
||||
(produced by the federated loop at round 0) and recomputes `Cbar_j(beta)` for the
|
||||
candidate betas **without retraining**. It reuses the production aggregation, so
|
||||
only the training feedback is removed, not the aggregation arithmetic.
|
||||
|
||||
Outputs (via `scripts/analyze_prototype_effect.py`):
|
||||
|
||||
- `*__prototype_effect.csv`: per `(class, beta)` — `n_contributors`,
|
||||
`support_min/max/mean`, `prototype_norm`, `reference_norm_beta0`,
|
||||
`displacement_from_beta0 = ||Cbar_j(beta) - Cbar_j(0)||`,
|
||||
`relative_displacement`.
|
||||
- `*__prototype_weights.csv`: per `(class, beta, client)` normalised weights.
|
||||
|
||||
Displacement is a *direct aggregation* measure only; it is **not** interpreted as
|
||||
improved or degraded detection (AGENTS.md §17).
|
||||
|
||||
## 5. Validation performed
|
||||
|
||||
### 5.1 Test suite
|
||||
|
||||
```
|
||||
.venv/bin/python -m pytest -> 75 passed
|
||||
```
|
||||
|
||||
Session-3 suite was 46 tests. Added: `tests/test_support_weighted_aggregation.py`
|
||||
(21), `tests/test_prototype_effect.py` (7), and one extra baseline-aggregation
|
||||
test (the old "support_weighted not available" guard was replaced by explicit
|
||||
availability/validation guards). The deterministic end-to-end baseline regression
|
||||
(`test_end_to_end_baseline_is_deterministic_and_learns_separable_fixture`) and all
|
||||
baseline arithmetic tests remain green.
|
||||
|
||||
### 5.2 Baseline preservation (integration evidence)
|
||||
|
||||
Smoke condition: `xiiotid`, `alpha=0.5`, `seed=0`, `max-rows=3000`,
|
||||
`sample-from=60000`, 4 clients, 2 rounds, 1 local epoch.
|
||||
|
||||
- `original_protean` and `support_weighted beta=0` produced **byte-identical**
|
||||
result artifacts (`history.json`, `test_global_metrics.json`, confusion matrix,
|
||||
per-class/per-client CSVs, `N_ij`, characterization, rare/unseen, round-0
|
||||
snapshot). Only `metadata.json` differed (`aggregation_mode`, `beta`,
|
||||
`run_name`).
|
||||
- The three round-0 snapshots (`beta = 0, 0.5, 1`) were **byte-identical**,
|
||||
confirming that the paired comparison reuses the same partition, split, model
|
||||
initialisation and round-0 training; beta only changes server aggregation.
|
||||
|
||||
### 5.3 Smoke runs (NOT scientific results)
|
||||
|
||||
| mode / beta | test accuracy | test macro-F1 | test rare macro-F1 | test rare recall |
|
||||
|---|---|---|---|---|
|
||||
| `original_protean` | 0.5167 | 0.2998 | 0.3140 | 0.4572 |
|
||||
| `support_weighted` 0.0 | 0.5167 | 0.2998 | 0.3140 | 0.4572 |
|
||||
| `support_weighted` 0.5 | 0.2333 | 0.1397 | 0.1797 | 0.3193 |
|
||||
| `support_weighted` 1.0 | 0.0383 | 0.0252 | 0.0324 | 0.1995 |
|
||||
|
||||
These are 3,000-row toy smoke runs with a random 4-client partition and a
|
||||
stratified 70/10/20 split; the split's class support at this scale is tiny and the
|
||||
metrics are unstable. They demonstrate that the beta path runs end to end and
|
||||
that beta **does** change training feedback (global prototypes enter the local
|
||||
loss from round 1). They must not be read as evidence about method quality. A
|
||||
full-scale paired validation/test protocol is required (§7).
|
||||
|
||||
Direct-effect instrumentation was exercised on the `beta=0` smoke snapshot
|
||||
(9 classes, max `||Cbar_j(1) - Cbar_j(0)|| = 7.65`, relative displacement up to
|
||||
1.51 for the most support-imbalanced class), written under the ignored smoke
|
||||
directory `artifacts/processed/smoke/analysis/`.
|
||||
|
||||
## 6. Provenance and assumptions
|
||||
|
||||
- `support_weighted` arithmetic (`N_ij^beta`, normalised over `S_j`) — **SPEC /
|
||||
PROJECT IMPLEMENTATION**; the research specification defines the formula.
|
||||
- `S_j = {i | N_ij > 0 and valid C_ij}` for **both** modes — **REPRODUCTION
|
||||
ASSUMPTION**, not verified original PROTEAN behaviour, carried from the
|
||||
baseline. It is **material, not rare**: `|S_j| < M` in **53/162 class-condition
|
||||
cells (32.7%)** across **15/18** `(dataset, alpha, seed)` conditions
|
||||
(`docs/baseline/BASELINE_VALIDATION.md` §D). It is not source-confirmed.
|
||||
- `beta = 0` equality with the baseline is **conditional on that shared `S_j`
|
||||
rule**; it is not an independent claim that `beta = 0` reproduces original
|
||||
PROTEAN.
|
||||
- No `equal_available` executable mode was added (the baseline already *is*
|
||||
equal-over-`S_j`, so it is arithmetically identical to `beta = 0`).
|
||||
- `N_ij` is treated strictly as sample support, not prototype quality.
|
||||
|
||||
## 7. Not done in this session
|
||||
|
||||
- No validation-based beta selection, no test evaluation of a selected beta.
|
||||
- No full or partial `(dataset, alpha, seed, beta)` sweep.
|
||||
- No `equal_available` mode, no client rejection, no new distance/metric.
|
||||
|
||||
## 8. Exact next step
|
||||
|
||||
Run **one** paired full condition (e.g. X-IIoTID `alpha=0.5`, seed 0, reusing the
|
||||
frozen manifest and frozen training settings) across
|
||||
`beta ∈ {0, 0.25, 0.5, 0.75, 1}` under the validation/test separation of
|
||||
`EXPERIMENT_PROTOCOL.md` §8, then inspect per-client/per-class artifacts before
|
||||
scaling. Measure runtime before any grid.
|
||||
@@ -70,9 +70,9 @@ A beta run MUST NOT derive a new partition; it loads the manifest for `(dataset,
|
||||
|
||||
## 5. Aggregation modes
|
||||
|
||||
**Baseline checkpoint: only `original_protean` is implemented.** The proposed
|
||||
support-weighted aggregation (`support_weighted`) and the `equal_available` control are
|
||||
**DEFERRED** to the later beta/controls phase and are not present in executable code.
|
||||
**Both `original_protean` and the proposed `support_weighted` are implemented.** The
|
||||
`equal_available` control is **not** implemented and is unnecessary (the baseline is already
|
||||
equal-over-`S_j`). See `docs/research/BETA_AGGREGATION_IMPLEMENTATION.md`.
|
||||
|
||||
- `original_protean` — reproduced baseline. **REPRODUCTION ASSUMPTION (project decision)**:
|
||||
because Eq. (2)/(5.5) is internally inconsistent about missing classes, the baseline
|
||||
@@ -80,12 +80,14 @@ support-weighted aggregation (`support_weighted`) and the `equal_available` cont
|
||||
**equal weights `1/|S_j|`**, following the thesis statement that prototypes are shared
|
||||
only for locally observed classes. This is not verified against code and must be
|
||||
validated in the baseline-verification phase.
|
||||
- `support_weighted` (deferred) — `w_ij = N_ij^beta`, normalised over `S_j`.
|
||||
- `equal_available` (deferred) — equal weights over `S_j`; only introduced later if the
|
||||
baseline participant rule is shown to differ from `S_j`.
|
||||
- `support_weighted` — `w_ij = N_ij^beta`, normalised over the **same** `S_j` as the
|
||||
baseline. `beta` must lie in `[0, 1]`; invalid or missing beta is rejected explicitly.
|
||||
- `equal_available` — deliberately not implemented; arithmetically identical to
|
||||
`support_weighted` with `beta = 0` given the shared `S_j`.
|
||||
|
||||
FROZEN candidates for the later phase: `beta ∈ {0.00, 0.25, 0.50, 0.75, 1.00}`, one beta for
|
||||
all classes and clients. They are **not** part of the baseline checkpoint.
|
||||
FROZEN candidates for the beta phase: `beta ∈ {0.00, 0.25, 0.50, 0.75, 1.00}`, one beta for
|
||||
all classes and clients. Implementation status and smoke validation:
|
||||
`docs/research/BETA_AGGREGATION_IMPLEMENTATION.md`.
|
||||
|
||||
## 6. Partition persistence
|
||||
|
||||
@@ -157,6 +159,10 @@ For every manifest, record at least: `N_ij`, class total, number of clients cont
|
||||
|
||||
Separate the direct mathematical aggregation effect from the end-to-end training effect.
|
||||
|
||||
**Implemented** in `src/protean_research/analysis/direct_prototype.py` and exposed through
|
||||
`scripts/analyze_prototype_effect.py`; it reuses the production aggregation on a fixed
|
||||
snapshot and requires no retraining.
|
||||
|
||||
- Persist at least one round's fixed `C_ij` and `N_ij` snapshot.
|
||||
- Reusing those snapshots, compute `Cbar_j(beta)` for several beta without retraining.
|
||||
- Report `|| Cbar_j(beta) - Cbar_j(0) ||`; optionally normalised aggregation weights,
|
||||
@@ -239,4 +245,8 @@ No statistic from validation or test data ever enters preprocessing.
|
||||
| Model aggregation | equal `1/M` | paper Eq. 2/5.5 |
|
||||
| Prototype computation | end-of-round mean embedding | paper Eq. 1 |
|
||||
|
||||
Until the baseline is verified, only smoke tests are permitted; no sweep may be launched.
|
||||
The baseline is validated and frozen (`docs/baseline/BASELINE_VALIDATION.md`), and the
|
||||
proposed `support_weighted` mode is implemented (`docs/research/BETA_AGGREGATION_IMPLEMENTATION.md`).
|
||||
Only smoke tests have been run for beta. Before any grid, one paired full
|
||||
`(dataset, alpha, seed)` condition across the beta candidates must be run and inspected
|
||||
(validation-based selection, then test), and runtime measured.
|
||||
Reference in new issue
Block a user