Skip to content

Multi-Agent Collaboration via Evolving Orchestration

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Puppeteer frames multi-agent LLM collaboration as a sequential decision process governed by a centralized RL-trained orchestrator that dynamically routes which agent activates at each step. Unlike systems with static topologies, the orchestrator evolves its policy via REINFORCE, simultaneously improving task performance and reducing token consumption. On four benchmarks across closed- and open-domain tasks, Puppeteer outperforms static multi-agent baselines while using fewer compute resources.

Problem

Existing multi-agent LLM systems rely on predefined or statically generated agent topologies (chains, trees, DAGs) that cannot adapt as task complexity grows or as the agent pool diversifies. Static coordination introduces redundant computation, poor scalability, and ineffective communication—mesh systems with 50 agents can take 10+ hours for modest software tasks.

Method

The framework formalizes multi-agent reasoning as a Markov sequential decision process. A centralized "puppeteer" policy π selects one agent to activate per step conditioned on the global system state Sₜ and task τ; agents are abstracted as triples (model, reasoning pattern, tools). Collaboration is serialized: rather than searching combinatorial topology space, the orchestrator unfolds agent activations into a sequence that can be reconstructed post-hoc as a directed graph. The orchestrator is initialized from a Llama-3.1 Nemotron-70B-Reward variant and trained online with REINFORCE. The reward balances correctness (binary for closed tasks, continuous for open) and efficiency via a step-wise FLOPs/token cost term penalized by λ·Cₜ, with discount γ=0.99. Default hyperparameters: episode length 4, parallel exploration width 3, λ=0.1.

Key Contributions

  • Puppeteer paradigm: centralized, learnable orchestrator that serializes multi-agent reasoning into a sequential MDP, decoupling agent selection from agent internals.
  • RL-driven adaptive evolution: REINFORCE policy optimization that progressively prunes low-value agents and shortens reasoning chains without sacrificing accuracy.
  • Joint reward design: composite reward explicitly trading off solution quality against token cost, with tunable λ enabling application-specific efficiency targets.
  • Emergent topology analysis: empirical characterization of compaction (rising graph density, hub formation) and cyclicality (feedback loops enabling mutual verification) as the orchestrator evolves.
  • Heterogeneous agent support: Titan (large closed/open-source models) and Mimas (small open-source models) subspaces evaluated independently and in combination.

Results

  • Puppeteer (evolved, Titan subspace) achieves 0.7731 average across GSM-Hard / MMLU-Pro / SRDD / CommonGen-Hard vs. 0.6893 at initialization; best competing baseline (EvoAgent) reaches 0.4994–0.6899 depending on dataset.
  • Puppeteer (evolved, Mimas subspace) average 0.6324 vs. 0.5364 for next-best AFlow.
  • Token consumption monotonically decreases over training across nearly all settings, with the Titan orchestrator reducing active agents over time (earlier termination); Mimas achieves token savings by preferring cheaper agents rather than shorter chains.
  • Topology hyperparameter sweep (WxDy): default W4D2 optimally balances accuracy and token use; increasing depth or width beyond defaults causes redundancy and performance degradation.
  • Puppeteer-Mono (homogeneous agents) still outperforms MacNet, EvoAgent, Self-Refine, and AFlow using the same base model.

Limitations

  • Policy is initialized from a specific Nemotron reward model; sensitivity to other policy initializations is not ablated.
  • RL training requires completed task trajectories for feedback, adding online training cost that is not accounted for in the efficiency comparisons.
  • Episode depth (max 4) and width (max 3) are fixed hyperparameters; the non-monotonic accuracy-efficiency trade-off means suboptimal defaults could hurt both.
  • Open-domain reward (quality score ∈ [0,1]) requires a separate quality evaluator, introducing potential noise and evaluator bias.
  • Analysis of emergent topologies is descriptive; no theoretical guarantees on convergence or optimality of the evolved structure.

Relevance to Agentic AI / LLM Agents

Puppeteer directly addresses a central challenge in agentic AI: how to coordinate heterogeneous LLM agents at scale without hand-crafting communication topologies or suffering combinatorial overhead. By treating orchestration as an RL policy over a serialized MDP, the work offers a principled alternative to static graph search approaches (MacNet, GPT-Swarm, AFlow) and demonstrates that learned coordination can simultaneously improve quality and cut compute—a key concern for deploying production multi-agent systems. The emergent compaction and cyclicality phenomena suggest that iterative critique and hub-based reasoning are discovered as optimal by RL, connecting to reflexion-style and debate-style paradigms from a different angle. This work fits squarely in the line of research on automated MAS design (ADAS, MAS-GPT) but uniquely integrates online RL so the orchestrator adapts to new tasks without retraining agent components.