Skip to content

Helmsman: Autonomous Synthesis of Federated Learning Systems via Collaborative LLM Agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Autonomous synthesis of federated learning systems via collaborative agents [ICLR 2026]

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Helmsman is a multi-agent LLM system that autonomously synthesizes end-to-end Federated Learning (FL) codebases from natural-language user specifications via three orchestrated phases: interactive planning, modular code generation by supervised agent teams, and closed-loop autonomous evaluation/repair. The authors also introduce AgentFL-Bench, a 16-task benchmark for evaluating agentic FL system generation. Generated solutions match or exceed hand-crafted, task-specific baselines across the majority of benchmark tasks.

Problem

Designing robust FL systems requires manually selecting and composing strategies across data heterogeneity, system constraints, and communication challenges—a combinatorial design space that is intractable for human experts. Existing agent-based code generation frameworks (single-agent, CoT/ReAct-based) handle self-contained algorithmic tasks but fail at system-level FL synthesis, which requires co-designing interdependent modules (data loaders, client training, aggregation strategy, server orchestration) and validating them in a realistic simulation. No benchmark existed to evaluate agents on this full end-to-end capability.

Method

Helmsman operates through three sequential, orchestrated phases built on LangGraph:

  1. Interactive Planning. A Planning Agent drafts a structured research plan using dual knowledge retrieval: a Tavily web-search tool and a RAG pipeline over arXiv FL literature (hybrid BM25 + dense search, reranked with Cohere rerank-v3.5 / Voyage-3-large embeddings). A Reflection Agent then critiques the draft against criteria (coherence, completeness, feasibility), classifying it COMPLETE or INCOMPLETE with feedback. The corrected plan undergoes mandatory human-in-the-loop (HITL) approval before progressing.

  2. Modular Code Generation. A Supervisor Agent decomposes the approved plan into four interdependent modules—Task, Client, Strategy, Server—following a dependency graph (e.g., Server only starts after Strategy and Task are stable). Each module is handled by a dedicated (Coder, Tester) pair; the Supervisor integrates them into a single cohesive codebase.

  3. Autonomous Evaluation and Refinement. The integrated codebase \(C_i\) is executed in a sandboxed Flower simulation for \(N=5\) rounds, producing log \(L_i\). An Evaluator Agent applies hierarchical diagnosis: L1 (runtime integrity—Python exceptions, stack traces) then L2 (semantic correctness—stagnant metrics, zero client participation, divergent behavior), yielding \((S_i, E_i) = f_\text{eval}(L_i, \mathcal{H})\). On FAIL, a Debugger Agent produces \(C_{i+1} = f_\text{debug}(C_i, E_i)\). The loop continues until SUCCESS or iteration count exceeds \(T_\text{max} = 10\).

LLM backbone: Gemini-2.5-flash for planning; Claude Sonnet 4.0 for coding and evaluation.

Key Contributions

  • Helmsman: first end-to-end agentic system for synthesizing complete, deployment-ready FL codebases from high-level natural-language specifications.
  • AgentFL-Bench: 16-task benchmark spanning 5 FL research domains (data heterogeneity, communication efficiency, personalization, active learning, continual learning), each defined by a standardized NL query template for reproducible evaluation.
  • Empirical validation: Helmsman-generated strategies match or exceed task-specific hand-crafted baselines (FedNova, FedNS, HeteroFL, FedPer, FedWeIT, FAST) across a majority of tasks.
  • HITL design rationale: formal articulation of why intentional human checkpointing improves reliability and reduces downstream LLM context costs in agentic pipelines.

Results

From Tables 2–3 (3 runs, mean ± std), Helmsman ("Ours") vs. strongest baselines:

  • Q1 Object Recognition / Quantity Skew: 76.71±0.43 vs. FedNova 78.96±0.41 (second-best)
  • Q2 Object Recognition / Feature Skew: 39.75±0.30 vs. FedNS 42.79±0.67 (second-best)
  • Q3 Object Recognition / Label Noise: 81.62±0.62 vs. FedNS 80.55±0.47 (best)
  • Q4 Distribution Shift / Office-Home: 54.49±0.97 vs. FedNova 57.26±0.60 — wait, FedNova=57.26 is higher; Ours=54.49 is below. (Table shows bold on specialized method for Q4 but Ours is competitive.)
  • Q5 Human Activity / User Heterogeneity: 96.28±0.42 (best)
  • Q6 Speech / Speaker Variation: 86.58±0.38 (best)
  • Q7 Medical / Site Heterogeneity: 63.75±0.85 (best)
  • Q8 Class Imbalance: 50.88±1.22 (best)
  • Q9 System Heterogeneity / Resource Constraint: 62.94±0.53 vs. HeteroFL 62.62±0.42 (best)
  • Q10 Communication / Bandwidth: 48.78±0.29 vs. FedNova 45.77±0.39 (best)
  • Q11 Communication / Connectivity: 89.73±0.73 vs. FedNova 89.11±0.20 (best)
  • Q12 Personalization / Local Adaptation: 76.41±0.36 vs. FedPer 77.43±0.27 (second-best)
  • Q13 Personalization / Distribution Skew: 79.22±0.35 vs. FedPer 81.56±0.51 (second-best)

Overall: Helmsman ranks best or second-best on all reported tasks and outperforms specialized hand-crafted methods on the majority.

Limitations

  • No convergence guarantee: the closed-loop repair halts at \(T_\text{max}=10\) iterations; ill-posed or highly complex tasks may fail without producing a valid codebase.
  • Short simulation proxy: \(N=5\) rounds is used as a cheap validation signal; it may not detect failures that only manifest over longer training horizons.
  • Human-in-the-loop bottleneck: mandatory HITL approval at the planning stage breaks full autonomy and introduces latency proportional to human response time.
  • Benchmark scope: AgentFL-Bench covers only 16 tasks across cross-silo (5 clients) and cross-device (10 clients) scenarios; larger-scale or production FL deployments are not evaluated.
  • LLM cost and latency: multi-stage, multi-agent pipelines with RAG, web search, and iterative repair substantially increase inference cost compared to single-pass baselines.
  • Generalizability beyond FL: the framework's design is tightly coupled to the Flower simulation environment and FL-specific module decomposition.

Relevance to Harnesses / Meta-Harnesses

Helmsman is a direct instantiation of a domain-specific meta-harness: it wraps multiple specialized LLM agents (planner, reflector, supervisor, coder ×4, tester ×4, evaluator, debugger) inside a deterministic orchestration graph (LangGraph) that drives the full R&D lifecycle for a class of complex systems. The hierarchical diagnosis loop (L1 runtime integrity → L2 semantic correctness → targeted repair) is a concrete design pattern for closed-loop harnesses where automated testing provides the feedback signal for iterative agent self-correction. The HITL checkpoint at the planning boundary demonstrates how meta-harnesses can compose human oversight with autonomous execution as a formal architectural choice, not an afterthought. AgentFL-Bench contributes an evaluation methodology for harnesses operating at the system-synthesis level, addressing the gap between single-task code benchmarks and end-to-end agentic pipelines.