Search Self-Play: Pushing the Frontier of Agent Capability without Supervision¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Self-play for agent capability improvement without ground-truth supervision

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Search Self-Play (SSP) is a fully unsupervised RL training framework for deep search agents in which a single LLM simultaneously acts as a question proposer and a problem solver, co-evolving both capabilities without any human annotation. The proposer generates verifiable, progressively harder queries via multi-turn search; the solver attempts to answer them. SSP yields consistent, substantial gains on top of multiple strong baselines across seven QA benchmarks.

Problem¶

RLVR for LLM agents requires large quantities of human-curated task queries with verified ground-truth answers, creating a data bottleneck that limits training scalability. Existing query-synthesis methods partially alleviate annotation cost but remain offline (unable to adaptively adjust difficulty during training) and still require rigorous per-sample answer validation, leaving the core scalability problem unsolved.

Method¶

SSP frames training as a zero-sum adversarial game with a cooperative constraint, using a single shared LLM policy $\pi_\theta$ in two roles:

Proposer: Given a ground-truth answer $a^*$ drawn from a pre-defined answer set $D$, the proposer executes multi-turn search tool calls to discover implicit factual evidence, then generates a challenging question $q = Q(\tau)$. The proposer's reward is $\bar{r}_{\text{propose}} = 1 - \frac{1}{n}\sum_{j=1}^n r^j_{\text{solve}}$ — it wins when the solver fails.

Solver: Given $q$, the solver explores $n$ trajectories via standard deep search; its reward is a binary answer-correctness judgment $r(A(\rho), a^*)$.

RAG verification (correctness constraint): All search results $O(\tau) = (o_1,\ldots,o_T)$ from the proposer's trajectory are collected as RAG documents. A separate solver pass $\sigma \sim \pi_\theta(\cdot|x_{\text{solve}}, q, O(\tau))$ checks whether the question is answerable with those documents. Only questions satisfying $r(A(\sigma), a^*) = 1$ enter the adversarial training batch (rejection sampling), preventing reward hacking via unanswerable questions.

The joint objective is: $$\min_u \max_v \mathbb{E}_{a^* \sim D,\, \tau \sim u(\cdot|a^*),\, \rho \sim v(\cdot|q=Q(\tau))}[r(A(\rho), a^*)]$$ subject to the RAG verification constraint. The solver is updated with GRPO (advantage $\hat{A}^j_i = r^j_{\text{solve},i} - \frac{1}{n}\sum_k r^k_{\text{solve},i}$); the proposer is updated with REINFORCE. Additional rule-based pre-filtering removes malformed questions (empty string, no search calls, excessively short, answer leaked in question).

Key Contributions¶

First self-play training scheme for agentic RLVR, eliminating the need for human-annotated query–answer pairs in deep search.
RAG-based automatic verification pipeline that guarantees generated question correctness using the proposer's own search trajectory, avoiding reward hacking without external supervision.
Co-evolutionary training of both proposer and solver under a single shared model, adaptive difficulty, and an online (not offline) data generation loop.
Demonstrated effectiveness in both from-scratch training and continual RL training on top of already-specialized search agents (Search-R1, ZeroSearch, R-Search).
Model-agnostic: gains replicated across Qwen2.5-7B/32B, LLaMA-3.1, and Qwen3 family models.

Results¶

Qwen2.5-7B-Base + SSP: +26.4 average points across 7 benchmarks; +40.4 on TriviaQA alone.
Qwen2.5-7B-Instruct + SSP: +8.0 average improvement.
Continual training: SSP uniformly improves Search-R1-7B, ZeroSearch-7B, and R-Search-7B over already-strong search-trained checkpoints (e.g., ZeroSearch-7B PopQA 51.8 → 60.4, TriviaQA 73.4 → 78.0, 2Wiki 54.4 → 59.2).
Scaling: Qwen2.5-32B-Instruct + SSP achieves state-of-the-art on 5 of 7 evaluated benchmarks.
All results measured by pass@1 with LLM-as-a-judge (Qwen2.5-32B-Instruct) on 500-sample subsets of NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle.

Limitations¶

Evaluated exclusively on deep search agents using a static local retrieval corpus (E5 + Wiki-2018); generalization to live web search or other tool-use agents (coding, GUI) is not demonstrated.
Answer set $D$ is still drawn from existing public training sets, meaning the method is not fully unsupervised from a data provenance perspective — it requires a pool of ground-truth answers as seeds.
Difficulty adaptation is indirect (controlled via the proposer's SSP win rate) and not formally analyzed; there is no guarantee of curriculum monotonicity.
The RAG verification step adds significant compute overhead (an extra solver rollout per proposer trajectory), and the paper does not report total training FLOPs compared to supervised RLVR baselines.

Relevance to Agentic AI / LLM Agents¶

SSP directly attacks the data bottleneck in agentic RLVR — one of the central open problems for scaling LLM agent training — by enabling a model to generate its own training curriculum through tool-augmented self-play. The RAG-grounded verification mechanism is a concrete solution to the reward hacking pathology that arises when the proposer can fabricate unanswerable questions, offering a template applicable beyond search to any agent with a verifiable external oracle. The co-evolutionary proposer–solver dynamic provides an online difficulty-adaptive training signal absent from offline synthesis pipelines such as WebSailor or ASearcher. For researchers tracking agentic RL, this work establishes self-play as a viable and scalable alternative to human data collection, complementing RLVR literature (Search-R1, ZeroSearch, R1-Searcher) with a fully self-supervised outer loop.