Unlocking Long-Horizon Agentic Search with Large-Scale End-to-End RL¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Large-scale end-to-end RL for long-horizon search agents; addresses open-source capability gap

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

ASearcher is a 32B-parameter search agent trained purely via end-to-end RL (no commercial LLM APIs for data or tools), achieving competitive performance with OpenAI DeepResearch and Kimi-Researcher on deep information-retrieval benchmarks. The key unlocks are a two-stage curriculum with synthesized 25.6k multi-hop QA pairs and large-scale asynchronous RL with up to 128 actions per rollout, which causes emergent long-horizon search strategies including uncertainty analysis and conflict verification.

Problem¶

Open-source search agents depend on commercial LLMs either to generate supervised fine-tuning trajectories or to serve as sub-modules (e.g., audio/visual processing, complex reasoning), raising a dependency on proprietary systems. Prior RL-based search agents focused on simple multi-hop QA with short rollout horizons and did not demonstrate the depth of retrieval needed for real-world "deep research" tasks. Training efficiency under variable-length long-horizon trajectories is also unresolved.

Method¶

Agent Design. ASearcher wraps QwQ-32B in a single-model loop with two tools: a search engine (<search>) and a web browser (<access>). Each turn produces (1) internal reasoning (excluded from future context to keep history clean), (2) a structured analysis summary, and (3) an action. Long webpages are chunked at 10k characters and summarized by the agent itself.

Training Data. Two sources: (a) 16k samples filtered from HotpotQA + 2WikiMultiHopQA by retaining questions of intermediate difficulty (neither trivially easy nor unsolvable), and (b) 9.6k synthetic QA pairs from a data-synthesis agent that iteratively applies Injection (inserts related facts to increase context complexity) and Fuzzing (replaces specific entities with vague descriptors) to seed questions, yielding 25.6k total.

Two-Stage Curriculum. Stage 1 trains on the full difficulty range to establish basic tool-calling. Stage 2 removes questions solvable in fewer than 5 tool calls, forcing the agent to develop long-horizon strategies.

Large-Scale Asynchronous RL. GRPO is used with turn limit 128. To handle high variance in trajectory length (some trajectories span dozens of tool calls vs. others just a few), training fully decouples rollout collection from weight updates: trajectories are collected asynchronously in parallel, and a training step fires once a sufficient batch accumulates, achieving near-full GPU utilization. Dynamic filtering removes queries where all rollouts yield identical rewards (zero advantage). Reward is LLM-as-Judge (Qwen2.5-72B-Instruct) at trajectory completion (sparse reward).

Key Contributions¶

ASearcher: a single-model, RL-only search agent with no commercial LLM dependency at training or inference time.
A scalable data-synthesis agent using Injection+Fuzzing operations to create 9.6k high-quality multi-hop QA pairs without commercial APIs.
Two-stage difficulty curriculum that progressively shifts to long-horizon (≥5 tool call) tasks.
Fully asynchronous agentic RL training (built on AReaL) that decouples rollout and weight-update phases to handle trajectory-length variance at scale.
Empirical demonstration of emergent complex search behaviors (reflective keywords, external-reference citations) arising purely from RL, analogous to DeepSeek-R1's reasoning emergence.
Zero-shot transferability: plugging in DeepSeek-V3 as a summary tool and running \(K=16\) parallel rollouts (majority-vote aggregation) closes the gap to commercial systems without retraining.

Results¶

Base ASearcher (single model, basic tools only):
GAIA: 58.7 (vs. Search-R1-32B 28.6, WebSailor-32B 53.2, AFM-RL-32B 60.9)
xBench-DeepSearch: 51.1 (vs. Search-R1-32B 19.5, WebSailor-32B 53.3)
Frames: 74.5 (vs. Search-R1-32B 44.1, WebSailor-32B 69.8)
HLE: 21.5
RL training gain over base model: +15.0 GAIA, +22.4 xBench, +15.6 Frames
Average tool calls: rises from 1.67 at initialization to >20 during training
+ DeepSeek-V3 summary tool (zero-shot):
GAIA 60.3, xBench 56.4, Frames 76.6, HLE 23.4
+ Test-time scaling (\(K=16\) rollouts, zero-shot):
GAIA 71.8, xBench 75.0, Frames 83.4, HLE 24.6
Outperforms OpenAI DeepResearch (67.0 GAIA, 69.0 xBench, 78.8 Frames) and Kimi-Researcher
Training cost: ~16k H800 GPU hours

Limitations¶

Zero-shot transfer to summary tools and test-time scaling requires external commercial models (DeepSeek-V3), reintroducing the dependency it avoids during training.
Evaluation on GAIA uses only 103 text-only validation examples, which may not fully represent the benchmark.
The sparse LLM-as-Judge reward introduces potential reward-hacking risks not analyzed in the paper.
The data synthesis pipeline still uses QwQ-32B (the same base model) for question generation, creating potential distribution bias.
No ablation on the relative contribution of the two-stage curriculum vs. the 128-turn limit vs. the asynchronous training infrastructure.

Relevance to Agentic AI / LLM Agents¶

This paper directly advances the core question of whether RL alone—without expert demonstrations from stronger models—can instill long-horizon tool-use in agents, answering affirmatively for the deep-search domain. The emergent behaviors (uncertainty analysis, conflict verification, structured query decomposition) arising purely from reward signal mirror findings from DeepSeek-R1 in the reasoning domain, suggesting a general principle about RL-induced capability emergence in agentic settings. The fully asynchronous training infrastructure addresses a practical bottleneck in scaling agentic RL to long rollout horizons, which is a prerequisite for training agents on complex real-world tasks. The zero-shot transfer to additional tools and test-time scaling without fine-tuning is relevant to work on composable and modular agent architectures.