DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment

🕒 Published (v1): 2026-07-08 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

Self-distillation from own trajectories in verifiable env; directly applicable to agent training pipelines

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DeepSearch-World is a deterministic offline Wikipedia environment with 420K multi-hop QA tasks that enables process-level verification of tool-use steps without LLM judges. DeepSearch-Evolve is an iterative self-distillation framework that generates scaffold trajectories, filters them via entity-level environment verification, converts them to ReAct format, and fine-tunes the agent on its own verified rollouts — no stronger teacher model required. A 9B model trained this way reaches 31.2% on BrowseComp, 61.5% on GAIA, and 93.4% on HotpotQA.

Problem

Existing self-evolving pipelines for tool-use agents face three failure modes: (1) SFT on self-collected trajectories saturates quickly, bounded by backbone capability and trajectory diversity; (2) RL with sparse, trajectory-level rewards provides no signal about where failures occur (query formulation, tool selection, evidence extraction, or answer synthesis); (3) on-policy self-distillation (OPSD) requires a stationary, reliable teacher distribution at each tool-use step, but non-deterministic web environments make per-step teacher distributions noisy, destabilizing long-horizon optimization.

Method

DeepSearch-World (environment): A deterministic offline Wikipedia corpus (~10M entries) with two tools: web_search_wiki(x) — BM25 retrieval over a Pyserini/Lucene index returning ranked (title, caption, url) tuples — and visit_wiki(url) — a SQLite offset index mapping URLs to full articles. Tasks are 420K multi-hop QA instances built by sampling entity-level random walks \(\tau = (v_1, \ldots, v_H)\) over the Wikipedia hyperlink graph \(G = (V, E)\) and obfuscating entity mentions. Each instance has a ground-truth entity set \(T_i\); the environment maintains a completed set \(S_t \subseteq T_i\) and verifies each tool response by matching it against \(T_i \setminus S_t\) without calling an LLM. Failed retrievals trigger staged rule-based reflection (generic revision → canonical entity name on repeated failures).

Scaffold teacher agent: A three-phase rollout (PLAN → ACT → END) with structured working memory (completed_list, todo_list, information, experience). At each ACT step \(t\): $\(s_{t+1} = U(s_t, a_t, o_t, r_t)\)$ where \(r_t\) is the environment-grounded reflection on failure. The scaffold is not exposed to the student.

Scaffold-to-ReAct conversion: Each verified scaffold trajectory is converted by rewriting the progress state \(P_t\), environment reflection \(R_t\), and local rationale \(A_t\) into a standard <think> block: $\(\langle\text{think}\rangle_t = P_t \oplus R_t \oplus A_t\)$ This injects planning, memory tracking, and error recovery into the student while training a deployable ReAct policy.

Evolving SFT loop: 11 rounds; each round samples 10K instances, keeps 4K that pass rejection sampling and Qwen3.5-9B trajectory-level filtering, converts to ReAct, and fine-tunes for one epoch. Importance sampling with decay \(\gamma = 0.5\) mixes data across rounds to mitigate catastrophic forgetting. SFT objective is token-level negative log-likelihood: $\(\mathcal{L}_\text{SFT}(\theta) = \mathbb{E}_{\tilde\tau \sim \mathcal{D}_e(R)} \sum_{t=1}^{T} \mathrm{KL}\!\left(\delta_{y_t} \;\|\; \pi_\theta(\cdot \mid x, y_{<t})\right)\)$ A final GRPO stage on 1,600 real-tool instances (Google SerpAPI + Jina) closes the offline-to-real execution gap.

Key Contributions

• DeepSearch-World: Deterministic offline Wikipedia environment with entity-level process verification and 420K multi-hop QA tasks derived from hyperlink-graph random walks.
• Scaffold process supervision: Three-phase teacher rollout (PLAN/ACT/END) with working memory and grounded reflection, enabling structured "search–fail–reflect–retry" trajectories.
• Scaffold-to-ReAct distillation: Procedure to convert scaffold traces into standard ReAct supervision, injecting agentic cognitive behaviors into a deployable student without runtime scaffolding.
• Evolving SFT framework: Iterative self-improvement loop using importance-sampled, rejection-filtered, environment-verified trajectories — no proprietary or stronger teacher model required.

Results

• vs. backbone (Qwen3.5-9B-Instruct): +23.8 BrowseComp (7.4 → 31.2%), +22.9 BrowseComp-ZH (13.5 → 36.4%), +29.0 HLE (20.0 → 49.0%), +37.6 GAIA (23.9 → 61.5%), +29.0 xbench, +48.1 HotpotQA (45.3 → 93.4%).
• Competitive with best open-source agents: matches Marco-DR (31.4% BrowseComp) and MiroThinker-v1.0 (31.1%), without using frontier-model distillation.
• Still below proprietary ceiling: OpenAI Deep Research 42.9%, o3 58.1% on BrowseComp.
• Tool-use behavior: avg. 18.0 interaction rounds vs. 4.7 for backbone; 5.4 avg. visit calls vs. 0.9; Advanced Capability Score 70% vs. 19%.
• Data-pool scale: 420K pool reaches higher validation plateau and lower format-error rate than 100K pool across 20 evolving rounds; 100K saturates and fluctuates after ~8 rounds.

Limitations

• Environment is restricted to an offline Wikipedia corpus; no live web coverage or dynamic knowledge.
• Training uses English-only trajectories; BrowseComp-ZH gains (+22.9) lag behind English BrowseComp (+23.8) and cross-lingual transfer is partial.
• Final GRPO bridging step still requires 1,600 labeled real-tool instances, limiting fully self-contained training.
• OPSD instability analysis is theoretical/comparative; the paper does not include a direct OPSD ablation on long-horizon tasks.
• Performance gap to proprietary agents (e.g., o3) remains substantial on hard benchmarks (31.2% vs. 58.1% on BrowseComp).

Relevance to Agentic AI / LLM Agents

This work directly addresses the core bottleneck of scalable self-improving agents: how to obtain dense, reliable process-level supervision for long-horizon tool use without relying on stronger teacher models. The verifiable environment paradigm — entity-level intermediate verification without LLM judges — provides a principled alternative to both sparse-reward RL and static imitation, and is directly applicable to any agentic system that interacts with structured knowledge sources. The scaffold-to-ReAct conversion pipeline also demonstrates a transferable recipe for distilling structured cognitive behaviors (planning, working memory, failure recovery) into a standard deployable agent, which is relevant to the broader effort to make agentic reasoning emergent rather than prompt-engineered.