MetaAgent: Automatically Constructing Multi-Agent Systems Based on Finite State Machines¶
🕒 Published (v1): 2025-07-30 12:22 UTC · Source: Arxiv · Venue: ICML 2025 · link
Ask a follow-up
Open an assistant pre-loaded with this paper's context.
TL;DR¶
MetaAgent is a framework that automatically constructs multi-agent systems by designing and optimizing Finite State Machine (FSM) topologies from a plain-text task description. It eliminates manual system design by using an LLM "designer" to generate agents, states, and transition conditions, then iteratively merging redundant states. The resulting systems match or exceed human-designed baselines on text, ML, and software-development benchmarks.
Problem¶
Existing multi-agent systems are manually engineered for narrow, pre-defined scenarios, requiring substantial human effort to implement and tune. Prior auto-design methods (SPP, AutoAgents, EvoAgent, ADAS, Symbolic Learning) each have at least one of: case-specificity with no generalization across inputs, lack of tool use, dependence on external training data, or rigid linear/debate communication structures with no ability to traceback to earlier states on error.
Method¶
Given a natural-language task description, a designer LLM executes a three-stage construction pipeline:
- Agent design: The designer generates a minimal set of role-specialized agents, each with a name, system prompt, and assigned tools (code interpreter, search engine).
- FSM construction: States are enumerated to cover foreseeable situations in the task domain. Each state specifies: a task-solving agent, a state instruction, a condition verifier (same LLM as the task-solver but with transition conditions appended to its prompt), and listeners (agents whose memory receives the state's output on successful transition). Transition conditions are expressed in natural language; the condition verifier routes to the target state, loops via a null-transition for self-refinement, or tracebacks to a prior state when upstream errors are detected.
- FSM optimization: An adaptor LLM performs pairwise comparisons over all state pairs, merging states whose assigned agents have insufficiently distinct roles. The process restarts until no further merges are possible, reducing chain length and redundancy without external data.
Formally, the FSM is \(M = (\Sigma, S, s_0, F, \delta)\) where \(\Sigma\) is the set of task-domain cases, \(S\) the state set, \(s_0\) the initial state, \(F\) the final states, and \(\delta\) the LLM-evaluated transition function.
At deployment, the system executes Algorithm 2: the active state's agent acts on the query + memory, the condition verifier evaluates output, and \(\delta\) selects the next state, issues a null-transition (feedback loop), or signals task completion. Other communication topologies (linear, debate, orchestrator) are shown to be degenerate FSM variants lacking null-transitions, arbitrary traceback, or per-state verifiers.
Key Contributions¶
- FSM formalism as a unified, auto-designable structure that strictly generalizes linear, debate, and orchestrator multi-agent topologies.
- A data-free LLM-driven construction pipeline (agent design → state/transition design → FSM optimization) requiring no external training examples or iterative fine-tuning.
- Per-state condition verifiers enabling both null-transitions (in-state self-correction) and arbitrary state traceback (error recovery to any prior state).
- Iterative state-merging optimization algorithm that reduces FSM complexity without degrading functionality.
- Demonstrated unique combination of properties: auto-designed, generalizable across inputs in a domain, tool-enabled, and traceback-capable — no prior system achieves all four.
Results¶
- Trivial Creative Writing: MetaAgent 0.86 vs. SPP 0.79 (next-best auto-designed), CoT/Direct ~0.74–0.76.
- GPQA (Diamond): MetaAgent 0.60 vs. Self-Refine 0.55 (next-best), SPP 0.45, Direct 0.46.
- Machine Learning Bench: MetaAgent achieves SOTA on Titanic and House Prices datasets; reaches 97% of average performance of the best human-designed system (DataInterpreter); surpasses all other auto-designed frameworks (SPP, AutoAgents).
- Software development: MetaAgent passes 50% more objective checkpoints than the human-designed MetaGPT baseline.
- Ablations confirm that removing tool use, FSM optimization, or traceback each degrade performance across all task types.
Limitations¶
- Uses GPT-4o as the sole foundation model; generalization to weaker or open-weight LLMs is not evaluated.
- The tool pool is restricted to a code interpreter and a search engine; broader tool coverage is not explored.
- FSM construction cost (designer + optimizer LLM calls) scales quadratically with state count due to pairwise merging comparisons.
- Evaluation on software development uses author-defined checkpoints rather than established community benchmarks, limiting direct comparability.
- Maximum number of transitions \(M\) is a hard cutoff; tasks requiring longer reasoning chains will fail rather than gracefully degrade.
- Paper text is truncated; full ML Bench table details and ablation numbers are not available for review.
Relevance to Harnesses / Meta-Harnesses¶
MetaAgent is a direct instance of a meta-harness: a system whose job is to construct and configure other agentic systems rather than to solve domain tasks itself. The FSM-as-specification pattern — where structure, roles, routing logic, and tool assignment are all emitted by a designer LLM and then executed by a separate runtime — mirrors the harness/meta-harness separation of concerns (orchestration layer vs. execution layer). The data-free, LLM-driven optimization loop (iterative state merging) is a lightweight alternative to the external-data-dependent self-iteration approaches (ADAS, Symbolic Learning), relevant to anyone building meta-harnesses that must bootstrap without labelled corpora. The traceback and null-transition mechanisms are directly applicable to harness robustness design: they formalize error-recovery and retry semantics that are typically hand-coded in bespoke harness implementations.