Learning to Orchestrate Agents in Natural Language with the Conductor

🕒 Published (v1): 2025-12-04 02:23 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

The Conductor is a 7B language model trained via reinforcement learning to dynamically orchestrate pools of larger worker LLMs by designing natural-language communication topologies, task decompositions, and targeted prompts. Without any manually designed scaffolds, it achieves state-of-the-art results on LiveCodeBench and GPQA-Diamond, surpassing frontier models including GPT-5 on several benchmarks. The approach demonstrates that effective multi-agent coordination strategies can emerge purely through end-to-end reward maximization.

Problem

Existing multi-agent coordination methods rely on manually designed topologies (e.g., fixed debate or refinement loops) or learned routers that select among a predefined set of human-engineered strategies. These approaches constrain the expressivity of coordination and fail to fully exploit the complementary strengths of specialized models. No single LLM is universally optimal across all tasks, yet there was no principled mechanism to automatically discover coordination strategies tailored to each input problem.

Method

The Conductor is a Qwen2.5-7B model fine-tuned with GRPO (a variant of PPO) to output three Python lists: subtask natural-language instructions, worker agent IDs, and per-step access lists defining inter-agent context visibility. The Conductor receives a problem, generates a chain-of-thought, then emits a full agentic workflow as structured text. Worker LLMs (Gemini 2.5 Pro, Claude Sonnet 4, GPT-5, DeepSeek-R1-Distill-Qwen-32B, Gemma3-27B, Qwen3-32B) execute steps sequentially; the reward is binary correctness of the final output. Training uses 960 problems across MATH, MMLU, RLPR, and LiveCodeBench V1 in 200 GRPO iterations with batch size 256, no KL regularization. Two extensions are added via short finetuning: (1) adaptive worker selection — training with randomly sampled k-model subsets to generalize to arbitrary user-specified pools; (2) recursive topologies — the Conductor can designate itself as a worker, enabling self-referential refinement as a tunable test-time compute axis.

Key Contributions

• RL-trained Conductor model that outputs free-form natural-language agentic workflows (subtasks + worker assignments + communication topology) via end-to-end reward maximization, with no human-designed coordination constraints.
• Demonstration that a 7B Conductor surpasses all individual worker models (including GPT-5) on LiveCodeBench and GPQA-Diamond and outperforms prior multi-agent baselines (MoA, MASRouter, RouterDC, Smoothie) while using fewer agent calls (~3 steps on average).
• Finetuning extensions unlocking adaptive worker-pool generalization and recursive self-calling topologies that provide a new test-time scaling axis.
• Emergent behaviors: difficulty-adaptive workflow length, spontaneous use of planning/verification/debate patterns, and prompt engineering quality that scales with Conductor model size.

Results

• LiveCodeBench V6: Conductor 83.93 vs. GPT-5 82.90, Gemini 2.5 Pro 67.24 (new SOTA at submission time, surpassing O-series models not in the worker pool).
• GPQA-Diamond: Conductor 87.5 vs. GPT-5 82.3, Gemini 2.5 Pro 84.8.
• AIME25: Conductor 93.3 vs. GPT-5 90.8, Gemini 2.5 Pro 78.3.
• Average across 7 benchmarks: Conductor 77.27 vs. GPT-5 74.78 (best prior single model).
• Outperforms all multi-agent baselines (MoA, MASRouter, RouterDC, Smoothie) on in-domain tasks with strictly fewer agent calls than most baselines.
• Recursive Conductor improves BigCodeBench from 37.8 → 40.0 using <2× the original call count.
• Adaptive-pool Conductor with open-source-only workers outperforms Claude Sonnet 4 by ~10% on constrained evaluation.

Limitations

• Relies on expensive frontier API calls during inference; cost scales with worker pool quality, which may exacerbate economic barriers to access.
• Training dataset is small (960 problems) and domain-limited (math, code, MMLU-style); generalization to tool-use, long-horizon, or embodied tasks is unproven.
• Recursive topologies are evaluated only on a subset of out-of-distribution tasks; infinite-recursion is prevented by a hard cap, not learned termination.
• Worker agents are treated as black boxes; the Conductor cannot observe intermediate chain-of-thought quality to adapt mid-workflow.
• 3B Conductor achieves competitive agent selection but inferior prompt engineering compared to 7B, suggesting prompt-engineering quality is a bottleneck sensitive to Conductor scale.

Relevance to Agentic AI / LLM Agents

This paper is directly relevant as one of the first demonstrations that meta-agent orchestration strategies — previously hand-coded in frameworks like LangGraph or AutoGen — can emerge autonomously from RL without any topology supervision. It reframes multi-agent system design as a learnable policy rather than an engineering artifact, establishing that natural language is a sufficient and expressive output medium for coordination. The recursive topology extension connects to test-time compute scaling literature and suggests that orchestrator models are a novel axis for inference-time scaling distinct from chain-of-thought length. For researchers tracking agentic AI, this provides a concrete RL training recipe (GRPO + workflow reward) that generalizes across heterogeneous closed- and open-source model pools.