Skip to content

AgentPO: Enhancing Multi-Agent Collaboration via Reinforcement Learning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

RL-based optimization of inter-agent interactions in multi-agent systems

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AgentPO trains a lightweight Collaborator agent via reinforcement learning (GRPO) to generate hints or critiques that guide a frozen, high-capability Actor agent, optimizing multi-agent collaboration without modifying the Actor. It consistently outperforms prompt-engineering and topology-search baselines on mathematical reasoning benchmarks while requiring only 500 training samples and 7.8% of EvoAgent's inference cost.

Problem

Multi-agent systems (MAS) struggle with two failure modes: manual workflow orchestration is brittle due to LLM sensitivity to prompt variations, and automated topology search faces combinatorial explosion. Both approaches neglect optimizing collaboration quality within a fixed topology. Existing methods also require fine-tuning large Actor models or incur high inference costs when ensembling multiple agents.

Method

AgentPO decouples collaboration from execution using two agent roles: a small, trainable Collaborator (\(\pi_\theta\)) and a frozen, high-performance Actor (\(\pi_\phi\)). The Collaborator generates an auxiliary signal \(z \sim \pi_\theta(\cdot \mid c_\theta(q))\) which is concatenated with the problem \(q\) to form the Actor's input; the Actor produces the final answer \(\hat{y} \sim \pi_\phi(\cdot \mid q, z)\). The binary reward \(R(\hat{y}, y) = \mathbb{I}(\hat{y} = y)\) trains the Collaborator via Group Relative Policy Optimization (GRPO):

\[J_{\text{GRPO}}(\theta) = \mathbb{E}\left[\frac{1}{G}\sum_{i=1}^G \frac{1}{|o_i|}\sum_t \left[\min\left(r_{i,t}\hat{A}_{i,t},\, \text{clip}(r_{i,t}, 1-\varepsilon, 1+\varepsilon)\hat{A}_{i,t}\right) - \beta D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}})\right]\right]\]

Two topology variants are evaluated: Feed-forward (Hint-Actor), where the Collaborator generates a proactive hint \(h\) conditioned only on \(q\); and Feedback (Critic-Actor), where the Collaborator generates a critique \(c\) conditioned on \((q, y_{\text{init}})\) and the Actor refines its answer. The framework also extends to Multi-Agent Debate and Role Assignment topologies by inserting a trained Collaborator into the existing structure.

Key Contributions

  • AgentPO framework: RL-based optimization of a lightweight Collaborator within a fixed MAS topology, leaving the Actor frozen.
  • GRPO-based training: binary success/fail reward with intra-group advantage normalization, no value function required.
  • Data efficiency: effective collaboration policy learned from as few as 100 samples, peaking at 500.
  • Hybrid system support: a locally fine-tuned Collaborator can guide black-box API-based Actor models that cannot themselves be fine-tuned.
  • Cross-topology generality: demonstrated on Hint-Actor, Critic-Actor, Multi-Agent Debate, and Role Assignment topologies.

Results

  • With Llama-3.2-3B-Instruct as Actor, AgentPO (Hint-Actor, Qwen2.5-3B Collaborator) achieves 24.5% avg accuracy across five benchmarks: +1.8% over Role Assignment, +7.2% over EvoAgent, +3.0% over CoT.
  • With Llama-3.1-8B-Instruct as Actor, AgentPO achieves 31.5% avg: +5.6% over Role Assignment, +11.3% over EvoAgent, +12.1% over CoT.
  • Collaborative vs. Actor optimization (Qwen2.5-Math-7B Actor): AgentPO with a 3B Hint model achieves 49.4% avg, surpassing SimpleRL-Zero-7B (36.0%), Prime-Zero-7B (31.6%), and OpenReasoner-Zero-7B (27.6%), all of which fine-tune the full 7B Actor.
  • Hybrid API setting (Qwen-Plus as frozen Actor, Qwen2.5-7B-Instruct Collaborator): 56.6% → 58.8% avg (+2.2%).
  • Critic-Actor topology with AgentPO: 58.9% avg vs. 56.6% standalone Qwen-Plus.
  • Multi-agent topologies: +3.3–5.8% on Math500/OlympiadBench over Debate and Role Assignment baselines.
  • AgentPO operates at 7.8% of EvoAgent's inference token cost while achieving higher accuracy (Figure 4).
  • Performance peaks at 500 training samples; slight regression at 700–1000 (mild overfitting to limited data).

Limitations

  • Evaluated exclusively on mathematical reasoning benchmarks; generalization to other domains (coding, planning, tool use) is not demonstrated.
  • The performance dip at 700–1000 training samples suggests overfitting risk in low-data regimes, and the optimal training size is dataset-dependent.
  • The Collaborator's effectiveness is sensitive to model family complementarity: Qwen-series Collaborators outperform Llama-series ones when paired with a Llama Actor, implying the framework benefits from cross-family pairing that may not always be available.
  • The fixed binary reward \(\mathbb{I}(\hat{y} = y)\) is straightforward for math but is hard to define for open-ended tasks.
  • Only pairwise (or shallow 2-level) topologies are tested; deeply nested or fully dynamic topologies are not explored.

Relevance to Agentic AI / LLM Agents

AgentPO directly addresses the core coordination problem in multi-agent LLM systems by replacing hand-crafted inter-agent prompts with a learned collaborative policy, offering a principled RL-based alternative to topology search. The Collaborator-Actor decoupling is a practically important design pattern: it enables upgrading collaboration quality without touching large, expensive, or black-box Actor models — directly relevant to production deployments that use proprietary API-based agents. The result that a 3B Collaborator fine-tuned on 500 samples can outperform full 7B Actor fine-tuning challenges the default assumption that capability improvements must come from scaling or fine-tuning the primary reasoner. This work complements role-specialization and debate-based MAS literature by showing that how agents communicate, not just what roles they play, is a learnable and high-leverage optimization target.