Skip to content

Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Combines MARL and RL to jointly optimize LLM agent specialization and performance

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Stronger-MAS introduces AT-GRPO, an Agent- and Turn-wise grouped RL algorithm that extends GRPO-style on-policy training to multi-agent LLM systems (MAS). Standard GRPO's grouping assumption breaks in MAS because prompts differ across roles and turns; AT-GRPO fixes this via tree-structured sampling and per-(agent, turn) grouping. On long-horizon planning, it pushes accuracy from a 14–47% single-agent RL baseline to 96–99.5%.

Problem

Applying on-policy RL (specifically GRPO-style optimization) to multi-agent LLM systems is underexplored and non-trivial for two coupled reasons: (1) GRPO's group-relative advantage requires all candidates in a group to share an identical prompt, but MAS prompts embed role-specific context and cross-agent interaction history that differs across agents and turns, invalidating the fairness assumption; (2) existing RL training frameworks (VERL, AReaL, TRL, OpenRLHF) support only a single model/policy, making concurrent on-policy training of role-specialized policies architecturally impossible.

Method

AT-GRPO algorithm adapts GRPO for MAS via three mechanisms:

  1. Tree-structured sampling: At each turn \(t\) for each agent \(i\), sample \(K\) candidate macro-actions from the current state, compute advantages within that group, then greedily select \(c^\star = \arg\max_c r_{t,i,e}^{(c)}\) as the executed action to continue the rollout. This ensures group size = \(K\) at every \((i, t)\) even in multi-turn trajectories, unlike parallel sampling where group size collapses to 1 after turn 1.

  2. Agent- and turn-wise grouping: A unique group key \(g = \text{hash}(e, i, t)\) is assigned per environment instance \(e\), agent \(i\), and turn \(t\), ensuring all \(K\) candidates in a group share an identical prompt. The relative advantage is computed as: $\(A_g\!\left(a_t^{(c)}\right) = \frac{R(a_t^{(c)}) - \text{mean}\!\left\{R(a_t^{(c)})\right\}_{c=1}^K}{F_\text{norm}\!\left\{R(a_t^{(c)})\right\}_{c=1}^K}\)$

  3. Agent-wise credit assignment: Each agent receives a mixed reward blending a global team reward and a role-local reward: $\(r_{t,i} = \alpha\, r_t^{\text{team}} + r_i^{\text{loc}}\)$ For a coder–tester MAS: \(r^{\text{team}}\) is the code pass rate on golden tests; \(r^{\text{loc}}_{\text{coder}}\) is the coder's own pass rate; \(r^{\text{loc}}_{\text{tester}}\) is the golden reference pass rate against generated tests.

MAS training system: Independent GPU resource pools (RolloutWorker + UpdateWorker per policy) support concurrent on-policy training of up to \(N\) distinct policies. CPU EnvWorkers run sandboxed environment instances; a Router dispatches trajectories to the appropriate UpdateWorker based on the role-to-policy mapping \(\sigma(i)\).

Key Contributions

  • AT-GRPO: First general on-policy RL algorithm for multi-turn, multi-agent LLM training with valid group-relative advantage estimation across diverse role structures.
  • MAS training system: Architecture supporting concurrent on-policy RL for both role-sharing (\(M=1\)) and role-specialized (\(M=N\)) policy regimes with per-model GPU pools and CPU environment fleets.
  • Empirical analysis: Shows that RL on MAS reinforces role-specific specialization; demonstrates that the choice between shared vs. per-role policies is task-dependent.
  • Open source: Code and environments released via PettingLLMs (https://github.com/pettingllms-ai/PettingLLMs).

Results

All experiments use Qwen3-1.7B and Qwen3-8B (no-thinking mode) on 8× H100 GPUs, \(K=4\) branches, turn horizon \(T=4\).

  • Long-horizon planning (Plan-Path, 1.7B): Single agent 5% → Single agent+GRPO 11% → MAS prompt-only 10% → MAS+AT-GRPO 82–96%; single-agent RL baseline of 14–47% (across model sizes) lifted to 96.0–99.5%.
  • Sokoban (8B): Single agent+GRPO ~14% → MAS+AT-GRPO ~98% (+84 pp absolute, +560% relative per Fig. 1).
  • LiveCodeBench (1.7B): Single agent+GRPO 18.8% → MAS+AT-GRPO 20.6% (+5 pp / +25% relative); average code benchmark gain 3.87–7.62%.
  • Math (AIME25, 1.7B): Single agent 9.8% → MAS+GRPO 16.7% (+6.9 pp); OlympiadBench: 22.2% → 35.9% (+13.7 pp); average math gain 9.0–17.93%.
  • Sudoku (1.7B): Single agent 7% → Single agent+GRPO 29% → MAS prompt-only 69% → MAS+GRPO 87%.
  • MAS+AT-GRPO consistently outperforms both single-agent GRPO and prompt-only MAS baselines across all four domains.

Limitations

  • Experiments are restricted to Qwen3 (1.7B and 8B); generalization to other model families is not evaluated.
  • Tree-structured sampling multiplies per-step inference cost by \(K\); the paper does not report wall-clock training overhead versus single-agent GRPO.
  • The \(\alpha\) reward mixing coefficient is set to 1 without tuning; sensitivity analysis is absent.
  • The choice between role-sharing and role-specialized policies requires empirical determination per task; no principled rule is derived.
  • Math gains, while positive, are considerably smaller than planning gains, suggesting limited benefit in domains where single-agent chain-of-thought is already strong.
  • Results on Sokoban show MAS prompt-only at 0%, suggesting the MAS workflow design for that game may be suboptimal without RL.

Relevance to Agentic AI / LLM Agents

This work is directly relevant to the core question of how to train multi-agent LLM systems end-to-end rather than relying solely on prompt engineering—a gap highlighted repeatedly in MAS surveys. AT-GRPO provides a principled credit assignment mechanism for cooperative agents with distinct roles and multi-turn interaction, addressing the fundamental mismatch between single-agent RL formalisms and collaborative agentic workflows. The dramatic gains on long-horizon planning (where single-agent RL plateaus at 47%) suggest that learned role specialization via RL unlocks capabilities that neither single-agent RL nor prompt-only MAS can achieve alone. The open-source PettingLLMs system also provides infrastructure for future MAS RL research, making this a foundational methodological contribution for the field.