Skip to content

WebSeer: Training Deeper Search Agents through Reinforcement Learning with Self-Reflection

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

RL with self-reflection trains deeper web search agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

WebSeer is a 14B search agent trained with a two-stage framework combining supervised fine-tuning on self-reflective trajectories and Self-Reflective Reinforcement Learning (SRRL), enabling deeper multi-hop web retrieval through iterative self-verification and query reformulation. It achieves state-of-the-art results of 72.3% on HotpotQA and 90.0% on SimpleQA, outperforming prior RL-trained search agents by large margins. The key insight is that explicit reflection steps—backtracking, cross-validating, and resubmitting answers with F1 feedback—dramatically extend useful tool-use depth from ~3 calls to 5–8 strategic calls.

Problem

Existing agentic RAG systems suffer from three compounding failures: (1) insufficient search depth—models prematurely synthesize answers from partial evidence rather than continuing retrieval; (2) absence of spontaneous self-reflection—no mechanisms to cross-verify, rewrite queries, or backtrack when retrievals are incomplete; (3) neglect of real-world web complexity—most prior work trains on local Wikipedia vector DBs, not live web environments. Error accumulation across multi-hop chains further degrades accuracy with each step.

Method

WebSeer uses a unified two-stage training framework on a Qwen2.5-14B base:

Stage 1 — SFT via Multi-Turn Rejection Sampling. A reasoner \(G\) generates a tool-augmented path \(P_t\) culminating in answer \(\hat{y}_i^{(t)}\). A verifier \(V\) (same model, different prompt) assesses correctness and returns binary judgment \(J_t\) plus a tool-augmented verification path. A validity predicate \(\Psi(R_t, \hat{y}_i^{(t)}, y_i^*) = 1\) iff the judgment matches ground truth. Valid verifier outputs are concatenated into the history \(H_t\); the process iterates up to \(n_{\max}\) turns. Only trajectories reaching correct final answers are retained. SFT uses a masked NLL objective that excludes tool-observation tokens from the loss, supervising only the agent's own reasoning and tool-call decisions.

Stage 2 — SRRL. Within a single dialogue, the model can submit answers up to 20 times. Each submission yields a token-level F1 reward \(r^{(t)} = F_1(\hat{y}^{(t)}, y^*)\) returned as text feedback. The trajectory-wise correctness reward is discounted exponentially: \(R_{\text{correct}}(\tau) = r \cdot \alpha^T\) where \(T\) is the submission count, penalizing unnecessary resubmissions. Format is penalized linearly for outputs exceeding a safe-zone threshold \(L_{\text{expect}}\) up to hard limit \(L_{\max}\). Policy optimization uses GRPO advantage estimation with DAPO asymmetric clipping \((\epsilon_{\text{low}}, \epsilon_{\text{high}})\) to handle skewed reward distributions.

Three tools are provided: Google Search (keywords → title/URL/snippet), Webpage Reader (URL + question → model-mediated summarization), and Code Executor (Python → stdout).

Key Contributions

  • Two-stage unified framework (cold-start SFT + SRRL) that operates within a single-model architecture—no auxiliary controllers or stronger backbone models
  • Multi-turn rejection sampling data construction method producing long self-reflective trajectories with explicit backtracking and answer-refinement patterns
  • SRRL: RL with in-context F1 feedback and exponential resubmission discount, enabling iterative answer refinement within a single trajectory
  • Empirical demonstration that RL after SFT shifts tool-use from overuse (~50 calls post-SFT) to strategic deployment (5–8 calls post-RL) without hard-coded constraints
  • State-of-the-art on HotpotQA (72.3%) and SimpleQA (90.0%) with a single 14B model; strong OOD generalization trained on site-restricted Wikipedia search but tested on open web

Results

In-domain multi-hop QA (7 benchmarks, 512-sample splits, LLM-as-Judge evaluation): - HotpotQA: 72.3% (WebSeer web) vs. 66.9% (Search-r1 local RAG, best prior RL agent) - 2WikiMultiHopQA: 84.2% vs. 57.0% Search-r1 (+27.2 pts) - NQ: 82.8% vs. 66.9% Search-r1 (+15.9 pts) - In-domain average: 82.4% vs. 69.9% DeepResearcher (prior SOTA web agent), +12.5 pts over Search-r1

Out-of-distribution benchmarks: - Bamboogle: 81.6% (WebSeer local RAG) — +12.8 pts over prior best - FanoutQA: 55.4% vs. 12.6% Search-r1-14B; nearly matches GPT-4o (55.8%) - FRAMES: 56.1% vs. 29.5% Search-r1-14B - SimpleQA: 90.0% vs. 36.4% Search-r1-14B

Tool-use depth: pre-SFT avg ~3 calls; post-SFT peaks at 10, extends to 50; post-RL sharpens to 5–8 with rare extremes.

Scale matters: SFT+RL benefits only the 14B model consistently; 3B and 7B show unstable behavior, malformed tool calls, and reward collapse.

Limitations

  • Model scale requirement: gains only materialize reliably at 14B; 3B/7B models exhibit instability and malformed JSON tool calls after RL
  • Training restricted to Wikipedia via site-search API for cost and noise control, creating a distribution gap with open-web inference
  • SFT data composition ratio (single-pass vs. multi-refinement trajectories) is a sensitive hyperparameter not determined by a principled method
  • Ablation studies are cut off in the provided text; full analysis of SRRL components vs. GRPO/DAPO choices is not fully reproduced here
  • 480 A800 GPU hours for 100 training steps limits accessibility
  • Evaluation uses LLM-as-Judge rather than exact match, introducing evaluation variance

Relevance to Agentic AI / LLM Agents

WebSeer directly addresses a central challenge in LLM agent design: how to train agents that know when to keep searching versus when to commit to an answer, without requiring explicit stopping rules. The SRRL mechanism—returning scalar F1 feedback as in-context text and allowing resubmission within a single trajectory—is a practical instantiation of iterative refinement in agentic loops, and the exponential discount on resubmissions is a clean solution to the reward-hacking risk of unbounded retries. The finding that post-RL tool use converges to a strategic 5–8 call range despite no hard constraint echoes broader observations about RL shaping emergent efficiency in agent behavior. The masked SFT objective (excluding observation tokens) is a transferable design choice for any agent trained on environment-interaction trajectories, avoiding the agent from learning to "copy" environment outputs rather than reason about them.