Skip to content

Auto-FL-Research: Agentic Search for Federated Learning Algorithms

🕒 Published (v1): 2026-07-01 18:28 UTC · Source: Arxiv · link

Why this paper was selected

Agentic search over FL algorithm space; meta-harness pattern applied to automated research

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Auto-FL-Research (AFR) is a constrained coding-agent workflow that automates federated learning algorithm search by letting agents propose and implement code-level changes (aggregation rules, local objectives, architecture variants) within a task-profile-defined mutation surface, validated through a fixed NVIDIA FLARE (NVFlare) FL contract. Evaluated on five FLamby healthcare tasks and six grouped-client LEAF profiles with five-seed repeats, AFR yields reproducible gains on 4/5 FLamby and 5/6 LEAF tasks while explicitly exposing seed-sensitive failures and separating genuine FL recipe changes from scalar HPO effects.

Problem

FL algorithm design spans a large, interacting surface—local optimizers, server aggregation, schedules, regularization, model architecture—that is too expensive to explore manually and unfairly compared when candidate changes can silently alter the metric, data split, or communication contract. Existing AutoML/NAS work covers only narrow sub-spaces (scalar HPO, federated NAS, learnable aggregation), and unconstrained coding agents risk confounding evaluation by changing the evaluation route, update type, or model-state schema.

Method

AFR wraps a coding agent in a campaign loop (Algorithm 1) structured around three invariant layers:

  1. Task profiles: each profile fixes the dataset, metric, model budget, client/site configuration, number of rounds, final evaluation policy, and the set of allowed mutation files. Candidates violating fixed budget fields are rejected before scoring.

  2. Federated contract: the NVFlare client contract (flare.init(), flare.receive(), flare.send(), strict state_dict loading, DIFF-typed updates with NUM_STEPS_CURRENT_ROUND metadata, shared final global-model evaluation) is enforced by an AST-based static validator plus Python compile checks and a per-task smoke command.

  3. Mutation surface: agents may modify task-local client training logic, job construction, registered model variants (under a \(\leq\) max_model_params cap), and shared custom aggregators. They may not alter data bridges, metrics, or communication protocol. Three search modes are defined: scripted scalar HPO (existing knobs only), architecture-open AFR (adds registered model variants and task-local code), and literature-enabled AFR (adds source-backed plateau recovery logged as non-scored rows).

Every candidate is appended to a tab-separated run log (score, runtime, budget, status, target file, description, artifacts). A plateau watchdog triggers a literature loop; literature events are recorded but not scored, keeping search cost and proposal timing visible. The full artifact trail (control prompt, task profile, mutation schema, candidate table, code diffs, seed evaluations) is committed to the experiment branch.

Key Contributions

  • A contract-preserving agentic FL search harness built on NVFlare task profiles, with explicit mutation boundaries checked by a static AST validator and smoke tests.
  • Evaluation across FLamby healthcare (cross-silo) and grouped-client LEAF (cross-device-approximation) benchmarks, with five-seed repeats and matched baselines for every reported gain.
  • A taxonomy of search outcomes: repeated FL mechanisms vs. fixed-surface scalar tuning effects vs. seed-selected single-run artifacts, operationalized via same-budget scripted HPO controls and held-out re-evaluation.
  • A reproducible artifact trail (not merely a best score) allowing post-hoc reconstruction of the search surface and causal attribution of gains.

Results

FLamby (five-seed repeat means over matched NVFlare baselines): - IXI: Dice \(+0.1981\) (+25.0%); mean 0.9895 vs. FedCompass calibration 0.9880 - ISIC2019: Balanced accuracy \(+0.1462\) (+29.6%); below FENS target of 0.750 - Camelyon16: ROC AUC \(+0.1634\) (+27.9%); mean 0.7494 above FENS target of 0.7150 - Heart Disease: Accuracy \(+0.074\) (+10.2%); matches FENS target within rounding - TCGA-BRCA: C-index \(+0.001\) (+0.1%); seed-sensitive, no robust lift (interval includes zero)

LEAF grouped-client (five-seed repeat): - Shakespeare: Next-char accuracy \(+0.1126\) (+24.4%) - Sent140: Accuracy \(+0.1032\) (+16.0%) - FEMNIST: Accuracy \(+0.0383\) (+4.6%) - Synthetic: Accuracy \(+0.0333\) (+3.5%) - Reddit: Next-token accuracy \(+0.0044\) (+2.9%) - CelebA: \(-0.001\) (−0.1%); baseline-saturated, campaign winner did not survive repeat

FEMNIST ablation (gains over matched baseline): - Fixed-model scalar HPO: +0.037 accuracy - Optimizer/scheduler search: +0.032 accuracy - Architecture-open AFR: +0.046 accuracy — code-level mutations add value beyond scalar HPO

Limitations

  • No cryptographic or sandbox-level proof that forbidden files were untouched; contract enforcement relies on AST validation, smoke tests, and post-hoc file-edit review.
  • LEAF experiments use a grouped-client approximation (NVFlare clients represent deterministic user groups), not the large-scale cross-device setup of the original LEAF paper.
  • \(n=5\) seeds is too small for confirmatory hypothesis tests; descriptive paired intervals should not be interpreted as statistical significance.
  • Validation-selected evaluation can overfit reported scores; the held-out re-evaluation is a partial check, not a substitute for externally defined test sets.
  • Causal attribution of gains to specific proposed mechanisms (e.g., the DSMIL-inspired model for Camelyon16) is not fully established; the paper explicitly flags this as a hypothesis from the search trace.
  • Candidates are run sequentially (one GPU each) to avoid concurrent edits; this limits throughput.
  • Agent-session token telemetry is not uniformly available across agent frontends and is excluded from the search budget definition.

Relevance to Harnesses / Meta-Harnesses

AFR is a direct instantiation of a research harness: it wraps a general-purpose coding agent in a structured meta-loop that enforces reproducibility constraints (fixed task profiles, AST-validated contracts, immutable scoring paths) and produces a structured artifact trail rather than just a result. The design pattern of separating the mutation surface from the evaluation contract, enforced at the harness layer rather than by the agent itself, is a concrete answer to the open problem of how to make agentic research loops auditable. The plateau-driven literature loop with explicit non-scored logging rows is a novel harness mechanism for blending automated search with literature-guided exploration while keeping the two distinguishable in post-hoc analysis. For researchers building meta-harnesses, AFR demonstrates how task profiles, static validators, run ledgers, and same-budget ablation controls can jointly decouple genuine algorithmic contributions from agent-induced confounders.