Skip to content

WideSearch: Benchmarking Agentic Broad Info-Seeking

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR; benchmarks wide-scale info-seeking workflows; exposes automation bottleneck

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

WideSearch is a benchmark of 200 manually curated tasks requiring LLM agents to exhaustively collect large-scale atomic information from the web and output it as a structured table. Best-performing multi-agent systems reach only 7.3% success rate (Avg@4), exposing a severe capability gap in completeness and fidelity at scale. The bottleneck is not item-level retrieval but ensuring zero omissions or errors across hundreds of facts simultaneously.

Problem

Existing agentic benchmarks (BrowseComp, DeepResearch Bench) target either hard-to-find single facts or deep synthesis/narration. Neither evaluates the "wide" regime: tasks where an agent must collect all entities satisfying a criterion and fill every attribute—work that is operationally overwhelming in volume but not cognitively complex. This class of task, common in professional research and planning, is entirely unrepresented in prior benchmarks, leaving critical agent deficiencies unmeasured.

Method

Each WideSearch task is a tuple \((Q, S)\): a natural-language query \(Q\) and a table schema \(S = \{C_1, \ldots, C_m\}\). The agent must produce a populated table \(T_\text{agent}\) by interacting with live web search and page-reader tools. Tasks are built through a five-stage pipeline: (1) human screening and query refinement from real user queries; (2) exhaustive gold-standard annotation with time/URL-count metrics; (3) parametric-knowledge filtering (questions answerable by LLM alone are discarded); (4) difficulty pruning (tasks completable in <10 min or <10 URLs are discarded); (5) iterative automated/human score alignment until consistency ≥95%.

Evaluation uses a hybrid automated pipeline: syntax validation → entity normalization → row alignment via primary keys → cell-wise scoring using exact match, numerical approximation, date matching, URL matching, or LLM-as-judge (GPT-4.1) depending on column type. Three metrics are reported: Success Rate (SR) (binary perfect match), Row-level F1, and Item-level F1, each under Avg@N, Pass@N, and Max@N aggregations.

Key Contributions

  • First benchmark explicitly targeting the "wide-context" information-seeking regime: exhaustive, multi-entity, attribute-filling tasks requiring live web search.
  • 200 bilingual (100 EN / 100 ZH) tasks spanning 15+ domains, grounded in real user queries, with a five-stage curation and quality-control pipeline.
  • Hybrid automated evaluation framework achieving >97.8% consistency with human judgment across multiple judge models.
  • Systematic benchmarking of 10+ systems: single-agent, multi-agent (divide-and-conquer), and end-to-end commercial systems, including human baselines.
  • Test-time scaling and failure-mode analysis distinguishing item-level retrieval ability from task-level completeness—key insight for future architecture design.

Results

  • Best single-agent SR Avg@4: GPT-5 at 6.9%; all others ≤4.5%.
  • Best multi-agent SR Avg@4: GPT-5 at 7.3%; top multi-agent Pass@4: GPT-5 12.0%, OpenAI o3 9.5%.
  • Human single-player SR: 20.0%; human cross-validated (multiple annotators) approaches ~100%.
  • Multi-agent consistently beats single-agent on F1: e.g., GPT-5 Item-level F1 Max@4 rises from 68.2% (single) to 74.5% (multi-agent).
  • End-to-end commercial systems (Gemini 2.5 Pro, Claude Sonnet 4, o3) cluster around 3–4.3% SR Avg@4, below API-access API-agent counterparts.
  • Item-level F1 can approach ~80% with sufficient retries (Pass@N), but SR remains near 0%: any single cell error fails the task entirely.
  • Automated pipeline consistency with human evaluation: >97.8% for all tested judge models (o4-mini 98.3%, GPT-4.1 98.0%).

Limitations

  • Binary SR metric is extremely strict; a table with one wrong cell fails entirely, which may not reflect practical utility.
  • Temporal invariance is asserted but web content changes; ground truth may drift as the live web evolves.
  • Only 200 tasks (100 per language); domain coverage, while broad, is hand-curated and may not be exhaustive.
  • Human cross-validation baseline is not directly comparable (multiple annotators vs. single agent), inflating the apparent human–agent gap.
  • Tool-use interface is standardized (search + page reader) and may not reflect diverse real-world agent tool stacks.
  • Truncated analysis section in the provided text; full failure-mode taxonomy and test-time scaling results are not fully shown.

Relevance to Harnesses / Meta-Harnesses

WideSearch directly quantifies where current agentic harnesses break down: not at the tool-call level, but at the orchestration level—planning sub-query coverage, reflecting on missed entities, and aggregating results without loss. The finding that multi-agent divide-and-conquer architectures consistently outperform single-agent loops points toward meta-harness designs (planner + parallel sub-agents + cross-validation) as the right structural response to breadth-first tasks. The benchmark's five-stage data pipeline and hybrid evaluation framework are themselves a meta-harness for benchmark construction, illustrating reusable patterns for grounding automated scoring in human judgment. For harness researchers, the test-time scaling analysis—showing item F1 saturates while SR stays near zero—is a sharp diagnostic: harnesses must enforce completeness guarantees across parallel workers, not just per-item correctness.