MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Self-play with strategic LLMs improves cooperative and competitive multi-agent reasoning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

MARSHAL is an end-to-end RL framework that trains LLMs to reason in multi-agent settings by having them play cooperative and competitive strategic games via self-play. It introduces two algorithmic fixes to GRPO—turn-level advantage estimation and agent-specific normalization—that solve credit assignment and variance problems in multi-turn, multi-agent training. Skills learned in games generalize zero-shot to established MAS frameworks, yielding up to +10.0% on AIME and +7.6% on GPQA-Diamond.

Problem¶

RL fine-tuning has proven effective for single-agent reasoning, but extending it to multi-turn, multi-agent settings faces two unresolved challenges: (1) long-horizon credit assignment—accurately attributing outcomes to individual actions across many turns when rewards are sparse; and (2) advantage variance instability—heterogeneous agent roles with asymmetric payoff scales corrupt advantage estimates when normalized jointly, destabilizing training.

Method¶

MARSHAL builds on Group-Relative Policy Optimization (GRPO) and extends it to a hierarchical turn-level MDP. Two novel modifications are introduced:

Turn-level advantage estimator: Rather than normalizing intermediate rewards globally then computing cumulative sums (as in GRPO's process-supervision variant), MARSHAL inverts the order—first computing Monte Carlo returns \(R_k^i = \sum_{\hat{k}=k}^{K} r_{\hat{k}}^i\), then normalizing across the group batch: \(A_{k,t}^{p,i} = R_k^{p,i} - \text{mean}(R^p)\). This is equivalent to GAE with \(\gamma=1\), \(\lambda=1\) using an empirical mean baseline.

Agent-specific advantage normalization: Trajectories are partitioned by player role \(p\) into sub-groups \(G^p\); advantage normalization is applied independently within each sub-group, preventing cross-role baseline contamination.

Training uses self-play on six two-player games spanning three categories: perfect-information competitive (Tic-Tac-Toe → Connect Four), imperfect-information competitive (Kuhn Poker → Leduc Hold'em), and imperfect-information cooperative (Mini Hanabi → Simple Hanabi). Rewards include intrinsic game outcomes, a format reward (\(+0.05\) valid / \(-10.0\) invalid), and a length penalty for verbosity. The base model is Qwen3-4B; both specialists (per-game) and a generalist (all games simultaneously) are trained.

Key Contributions¶

MARSHAL: an end-to-end multi-agent RL framework using self-play across cooperative and competitive strategic games to develop generalizable MAS reasoning
Turn-level advantage estimator that inverts GRPO's normalize-then-cumsum to cumsum-then-normalize, providing stable learning signals across long trajectories
Agent-specific advantage normalization that partitions advantage estimation by player role to handle heterogeneous payoff scales
Demonstration that game-trained agents generalize zero-shot to practical MAS frameworks (MAD, AutoGen) on math and QA benchmarks

Results¶

Strategic games: MARSHAL generalist achieves up to +28.7% on held-out Leduc Hold'em and +22.9% on Simple Hanabi over Qwen3-4B baseline; outperforms SPIRAL (competitive-only baseline) across all six games
MAD (competitive): MARSHAL generalist achieves +3.51% average over Qwen3-4B across 7 benchmarks; +7.57% on GPQA-Diamond specifically
AutoGen (cooperative): MARSHAL generalist achieves +10.00% on AIME24, +3.5% average across all benchmarks
Ablation (failure mode analysis on GPQA-Diamond/MAD): MARSHAL significantly reduces Inter-Agent Misalignment failures (e.g., failure-to-clarify, conversational-reset, information-withholding) vs. Qwen3-4B baseline
Competitive game specialists excel in competitive MAS (MAD); cooperative specialist (Hanabi) excels in cooperative MAS (AutoGen), confirming game-type alignment matters

Limitations¶

Base model is Qwen3-4B only; generalization to larger models or different architectures not demonstrated
Game suite is restricted to two-player settings; multi-player coordination is not addressed
Generalization evaluation is zero-shot; fine-tuning MARSHAL agents on downstream tasks is not explored
Failure mode analysis covers GPQA-Diamond only; broader analysis across all benchmarks is absent
The cooperative Hanabi training environment (Mini Hanabi) is a simplified, reduced variant; complexity gap to real-world cooperation may be large
Self-play assumes a single shared policy across roles; asymmetric multi-agent settings with heterogeneous models are not covered

Relevance to Agentic AI / LLM Agents¶

MARSHAL directly addresses the open problem of how to train LLMs for robust multi-agent interaction—a prerequisite for deploying LLM agents in collaborative pipelines, debate frameworks, and tool-use environments. The insight that strategic game self-play instills transferable social reasoning capabilities (role awareness, Theory of Mind, intent recognition) rather than just game-specific skills is particularly significant, suggesting games as a scalable curriculum for MAS agent training. The turn-level GRPO modifications are directly applicable to any multi-turn agent training setup beyond games. The zero-shot gains in MAD and AutoGen demonstrate that improved individual agent reasoning policies compound in multi-agent systems.