Skip to content

WebDancer: Towards Autonomous Information Seeking Agency

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

WebDancer is an end-to-end trained web agent built on ReAct that combines scalable agentic data synthesis with a two-stage SFT→RL training pipeline to enable autonomous multi-step information seeking. It introduces two QA construction methods (CRAWLQA and E²HQA) and uses the DAPO RL algorithm to generalize beyond the cold-start SFT phase. On GAIA and WebWalkerQA, WebDancer (QwQ-32B backbone) achieves 51.5% and 47.9% average Pass@1 respectively, outperforming all open-source baselines at the same model scale.

Problem

Existing information-seeking agents either rely on prompt engineering without exploiting reasoning model capabilities, or apply SFT/RL on datasets too shallow (e.g., 2Wiki >80% solved) and too small (GAIA: 466 examples) to drive real-world generalization. No prior open-source work provides a complete, scalable pipeline—from training data synthesis through RL—for building deep research agents comparable to proprietary systems like OpenAI Deep Research.

Method

Data synthesis: Two pipelines produce complex, multi-hop QA pairs. CRAWLQA recursively crawls knowledgeable websites (arXiv, GitHub, Wikipedia) and uses GPT-4o to synthesize typed questions (COUNT, MULTI-HOP, INTERSECTION). E²HQA iteratively rewrites simple SimpleQA-style questions by retrieving information about entities and substituting them into the question, controlling difficulty by the number of rewriting iterations.

Trajectory sampling: A ReAct agent framework with actions {search, visit, answer} generates trajectories via rejection sampling. Short-CoT trajectories use GPT-4o directly; Long-CoT trajectories feed step-wise history to QwQ-Plus (excluding prior thoughts to avoid out-of-distribution inputs). A three-stage funnel (validity → correctness via GPT-4o judge → quality: non-redundancy, goal alignment, logical accuracy) filters trajectories.

Training: Stage 1 is supervised fine-tuning (RFT/cold start) on filtered ReAct trajectories with observation tokens masked from loss. Stage 2 applies on-policy DAPO RL on QA pairs unused during SFT. DAPO uses decoupled asymmetric clipping (εlow/εhigh) and dynamic sampling that discards prompts with accuracy 0 or 1, focusing learning on informative examples. Reward is binary: 0.1 × format score + 0.9 × LLM-as-Judge answer score.

Key Contributions

  • Two scalable agentic QA synthesis pipelines: CRAWLQA (web-crawl-based) and E²HQA (easy-to-hard iterative rewriting), producing 17,764 training instances total
  • Short-CoT / Long-CoT trajectory construction strategy using GPT-4o and QwQ-Plus respectively, with a three-stage funnel filter
  • Two-stage training pipeline: SFT cold start (observation-masked loss) followed by DAPO on-policy RL with dynamic sampling
  • Empirical analysis of RL's effect on reasoning length, action count, CoT knowledge transfer across model types, and web environment non-stationarity
  • First end-to-end trained deep research agent in the Tongyi DeepResearch open-source series

Results

  • GAIA (Pass@1): WebDancer QwQ-32B 51.5% vs. Vanilla ReAct QwQ-32B 37.8%; WebThinker-RL 48.5%; Simple DS 50.5%; OpenAI DR 67.4% (closed)
  • WebWalkerQA (Pass@1): WebDancer QwQ-32B 47.9% vs. Vanilla ReAct QwQ-32B 24.1%; WebThinker-RL 46.5%
  • GAIA Pass@3: Best model reaches 64.1%; WebWalkerQA Pass@3: 62.0%
  • BrowseComp (En./Zh.): WebDancer 3.8/7.9 (Pass@1/Pass@3) En., 18.0/31.5 Zh.
  • RL over SFT yields +5.80 pp Pass@1 gain on 7B and 32B models; gains are marginal for QwQ-32B (LRM), attributed to sparse rewards from long trajectories
  • SFT-only with no RL on QwQ achieves only ~5% Pass@3 on GAIA, confirming cold start alone is insufficient
  • Long-CoT trajectories raise invalid rates significantly for non-reasoning models (21.36% for 7B vs. 0.65% for Short-CoT)

Limitations

  • RL improvements for large reasoning models (QwQ-32B) are marginal in Pass@1/Pass@3; consistency gains are modest, likely due to sparse rewards from very long trajectories
  • Long-CoT reasoning patterns from LRMs do not transfer cleanly to instruction-tuned models, causing high invalid rates and repetition
  • Web environment non-stationarity causes agent instability unrelated to decoding temperature, making reliable evaluation difficult
  • Training and evaluation datasets (GAIA, WebWalkerQA) are relatively small and may not fully capture real-world diversity
  • All results use LLM-as-Judge (GPT-4o) for answer correctness, introducing potential evaluation noise for non-verifiable QA

Relevance to Agentic AI / LLM Agents

WebDancer provides a reproducible, open-source blueprint for the full stack of building deep research agents—data synthesis, trajectory filtering, cold-start SFT, and on-policy RL—which directly addresses a key gap in the community relative to proprietary systems like OpenAI Deep Research. The decoupled SFT+DAPO pipeline and the finding that RL improves consistency and action sophistication but has diminishing returns for already-capable LRMs are practically important design lessons for anyone training tool-augmented agents. The E²HQA synthesis method offers a general recipe for bootstrapping difficult agentic training data without human annotation. This work fits squarely in the emerging line of ReAct-style web agents trained end-to-end (alongside Search-o1, R1-Searcher, WebThinker), and the analysis of CoT transfer failure across model types is relevant to anyone composing reasoning models with smaller instruction models in multi-agent pipelines.