Skip to content

SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

SPIRAL: self-play removes need for human-curated QA pairs in multi-agent RL training

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SPIRAL is a self-play framework that trains LLMs by having them compete in multi-turn zero-sum language games (TicTacToe, Kuhn Poker, Simple Negotiation) against continuously improving versions of themselves, generating an automatic curriculum without human supervision. The key technical contribution is Role-conditioned Advantage Estimation (RAE), which stabilizes multi-agent RL by maintaining per-role, per-game baselines to prevent "thinking collapse." Multi-game SPIRAL training improves reasoning benchmarks by up to 10.5% absolute across Qwen3 and Llama model families, outperforming SFT on 25,000 expert trajectories.

Problem

Existing RLVR approaches (o1, DeepSeek-R1) depend on human-curated problem-answer pairs and domain-specific reward engineering, creating a scalability bottleneck: every new reasoning domain requires expert-crafted metrics, problem curation, and trace validation. Prior self-play attempts for LLMs were limited to offline updates on single games, LoRA adaptations, or single-turn tasks, leaving multi-turn competitive dynamics unexplored as a reasoning training signal.

Method

SPIRAL formulates self-play over a collection \(\mathcal{G} = \{G_1, G_2, \ldots, G_n\}\) of two-player zero-sum Markov games. A single shared policy \(\pi_\theta\) plays both roles simultaneously (role-conditioned via system prompts), so improving at one role automatically strengthens the opponent. Rewards are sparse: zero at every non-terminal step, \(R_0(\tau) = \rho_i(s_T) \in \{-1, 0, +1\}\) and \(R_1(\tau) = -R_0(\tau)\) at game termination.

The standard REINFORCE gradient over shared parameters suffers from high variance because \(R_0\) and \(R_1\) are negatively correlated and positional asymmetries (first-move advantage, information asymmetry) bias per-role expected returns. Role-conditioned Advantage Estimation (RAE) addresses this by maintaining separate exponential-moving-average baselines \(b_{G,p}\) for each game \(G\) and role \(p\):

\[b_{G,p} \leftarrow \alpha b_{G,p} + (1-\alpha)R_p(\tau), \quad A_{G,p}(\tau) = R_p(\tau) - b_{G,p}\]

The variance-reduced policy gradient becomes:

\[\nabla_\theta J_{\text{SPIRAL}}(\theta) = \mathbb{E}_{G \sim \mathcal{G}} \mathbb{E}_{\tau \sim \pi_\theta \times \pi_\theta | G} \left[ \sum_{p \in \{0,1\}} \sum_{t \in T_p} A_{G,p}(\tau) \cdot \nabla_\theta \log \pi_\theta(y_t^{(p)} | s_t, p, G) \right]\]

The system is implemented as a fully online distributed actor-learner architecture (built on Oat), with actors using vLLM for inference and TextArena for game simulation, enabling continuous weight-sync and on-policy updates across 8 H100 GPUs.

Key Contributions

  • Fully online multi-agent multi-turn RL framework for LLMs: distributed actor-learner with full-parameter updates, not offline or LoRA-restricted
  • Role-conditioned Advantage Estimation (RAE): per-role, per-game baselines preventing "thinking collapse" (models abandoning reasoning traces after ~200 steps without RAE)
  • Empirical transfer discovery: self-play on zero-sum language games improves out-of-distribution academic reasoning benchmarks by up to 10.5% without any domain-specific data
  • Multi-game synergy: TicTacToe (spatial reasoning), Kuhn Poker (probabilistic/EV reasoning), Simple Negotiation (strategic optimization) develop complementary skills that combine superadditively
  • Automatic curriculum superiority: self-play's adaptive curriculum outperforms fixed opponents (Random, Mistral-Small-3, Gemini-2.0-Flash-Lite) by preventing exploitation of static strategies

Results

  • Qwen3-4B-Base + SPIRAL-Multi: 34.0% → 44.5% average across 8 benchmarks (+10.5% absolute); AIME24: 9.6 → 19.7 (+10.1%), AMC-23: 42.4 → 61.6 (+19.2%), MMLU-Pro: 47.2 → 58.5 (+11.3%)
  • Qwen3-8B-Base + SPIRAL-Multi: 39.5% → 49.6% (+10.1%); AIME24: 12.1 → 26.2 (+14.1%)
  • Octothinker-8B-Base: 25.8% → 33.8% (+8.0%); GPQA-Diamond: 22.1 → 33.8 (+11.7%)
  • Llama-3.1-8B-Instruct: 23.9% → 25.9% (+2.0%, despite being already instruction-tuned)
  • DeepSeek-R1-Distill-Qwen-7B (already RLVR-trained): 60.4% → 61.8% (+1.4%), confirming additive benefit
  • SPIRAL consistently outperforms SFT on 25,000 expert Qwen3-32B-generated game trajectories across all 8 benchmarks for all model families
  • Chain-of-thought analysis (290 game trajectories, 46,792 math solutions, GPT-4.1 judge): Case-by-Case Analysis transfers at 72%→71% fidelity, Pattern Recognition amplifies (35%→45%), Expected Value Calculation transfers selectively (78%→28%)
  • Fixed-opponent (Gemini) training: win rate climbs 0%→62.5% (exploitation); self-play win rate stays at ~50–52% (genuine learning)

Limitations

  • Gains are modest for already-strong or instruction-tuned models (Llama-3.1-8B-Instruct +2.0%, DeepSeek-R1-Distill +1.4%), suggesting diminishing returns when base capabilities are high
  • Games tested are relatively simple (TicTacToe, Kuhn Poker, Simple Negotiation); it is unclear whether more complex games would yield greater or different transfer
  • RAE uses Monte Carlo returns with no temporal difference bootstrapping; variance may still be an issue for longer game horizons
  • Transfer mechanism analysis relies on LLM-as-a-judge (GPT-4.1) classification of chain-of-thought patterns, which may not be a reliable ground truth for reasoning attribution
  • Training budget is fixed at 400 steps; scaling behavior with longer training is not characterized

Relevance to Agentic AI / LLM Agents

SPIRAL is directly relevant to the agentic AI agenda because multi-turn zero-sum self-play is structurally equivalent to the sequential decision-making loop central to LLM agents—maintaining state across turns, planning ahead, adapting to an evolving environment. The fully online multi-agent RL infrastructure developed here (distributed actor-learner, vectorized game environments, shared-parameter self-play) is a reusable foundation for training agents that must reason and act over long horizons without handcrafted reward signals. The finding that game-learned reasoning patterns (EV calculation, case decomposition) transfer to math benchmarks supports the hypothesis that domain-general reasoning is an emergent property of competitive pressure, not curriculum-specific knowledge—a principle applicable to tool-using or multi-agent orchestration settings. The "thinking collapse" failure mode and its RAE fix are a concrete warning and solution for anyone applying MARL-style training to LLM agents.