Skip to content

WideSearch: Benchmarking Agentic Broad Info-Seeking

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Benchmark for wide-scale agentic info-seeking; measures breadth vs depth tradeoffs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

WideSearch is a benchmark of 200 manually curated questions (100 English, 100 Chinese) designed to evaluate LLM agents on "wide-context" information-seeking tasks: collecting large sets of atomic facts from the web and populating a structured table. All tested systems achieve near-zero success rates (best: 7.3% Avg@4), revealing that current agents fail not at finding individual facts but at guaranteeing completeness and zero-error aggregation at scale.

Problem

Existing benchmarks for agentic search focus on either locating hard-to-find single facts (DeepSearch/"I can't find it") or synthesizing complex narratives (DeepResearch/"I can't write it well"). Neither covers the class of tasks whose bottleneck is operational scale and fidelity—exhaustively collecting all entities matching a set of criteria and organizing them into a verifiable structured output. This "broad-sourced, multi-entity" retrieval category is ubiquitous in professional workflows (financial screening, job searches, graduate admissions research) yet wholly unevaluated.

Method

Each WideSearch task is a tuple \((Q, S)\): a natural language query \(Q\) and a table schema \(S = \{C_1, C_2, \ldots, C_m\}\). An agent must identify the full entity set \(E = \{e_1, \ldots, e_n\}\) satisfying \(Q\)'s constraints via live web search, then fill every cell \(T_\text{agent}(i,j)\) with the value of attribute \(C_j\) for entity \(e_i\).

Data pipeline (5 stages): (1) human annotators screen and refine real user queries; (2) exhaustive gold-standard annotation with time/URL-count metrics recorded; (3) parametric knowledge filtering—questions answerable by LLMs without tools are discarded; (4) difficulty pruning—tasks completable in under 10 minutes or fewer than 10 URLs are rejected; (5) iterative refinement until automated scoring agrees with human scoring at ≥95% similarity.

Evaluation: A hybrid automated pipeline performs syntax validation, row alignment via primary-key matching, and cell-wise scoring using five methods: exact match, numerical approximation, date matching, URL normalization, and LLM-as-a-judge (GPT-4.1-2025-04-14 by default). Three metrics are reported: Success Rate (SR)—binary perfect-table match; Row-level F1—treating each row as a unit; Item-level F1—treating each cell as a unit. Aggregation uses Avg@N, Pass@N, and Max@N over \(N=4\) independent runs.

Evaluated systems: 9 single-agent LLMs (GPT-5, o3, Claude Sonnet 4 Thinking, Gemini 2.5 Pro, Kimi K2, Qwen3-235B-A22B-Thinking, DeepSeek-R1, Doubao-Seed-1.6 Thinking/Non-Thinking), same models in a multi-agent divide-and-conquer framework, 3 end-to-end commercial web-browse systems, and 10 human annotators.

Key Contributions

  • First benchmark specifically targeting wide-context, broad-sourced, multi-entity information-seeking with structured table output
  • 200-question bilingual dataset (EN/ZH) spanning 15+ domains, each grounded in real user queries and requiring ≥10 search queries and ≥10 URLs
  • Five-stage quality-control pipeline ensuring difficulty, completeness, temporal invariance, and verifiability
  • Hybrid automated evaluation system with >97.8% consistency with human judgment across multiple judge models
  • Systematic analysis distinguishing item-level retrieval capability (item F1 can reach ~80% with retries) from task-level completeness failure (SR near 0%)
  • Empirical evidence that multi-agent divide-and-conquer outperforms single-agent on this task class

Results

  • Best single-agent SR: GPT-5 at 6.9% Avg@4 / 13.5% Pass@4; DeepSeek-R1 lowest at 0.4% Avg@4
  • Best multi-agent SR: GPT-5 at 7.3% Avg@4 / 12.0% Pass@4; multi-agent consistently beats single-agent in F1 (e.g., GPT-5 item F1 Max@4: 74.5% vs. 68.2%)
  • End-to-end commercial systems: Gemini 2.5 Pro best at 4.3% Avg@4; all hover ~5%
  • Human single-player SR: 20% (with unlimited time and tool access), Row F1 69.2%, Item F1 82.4%—task is inherently hard even for humans working alone
  • Test-time scaling insight: Item-level F1 approaches ~80% with sufficient retries, confirming the bottleneck is completeness guarantees, not per-item findability
  • Automated evaluation consistency: >97.8% agreement with human annotation across all four tested judge models (o4-mini: 98.3%, Gemini 2.5 Pro: 98.1%, GPT-4.1: 98.0%, Doubao: 97.8%)
  • Multi-agent framework costs significantly more tool calls (e.g., Doubao-Seed-1.6 Thinking: 52.34 vs. 1.14 tool calls, single vs. multi)

Limitations

  • Success Rate metric is binary—any single cell error counts as total failure, making SR extremely sensitive and potentially underrepresenting meaningful partial progress
  • Ground-truth tables require multiple human annotators with cross-validation; ongoing maintenance as web content changes may degrade temporal invariance
  • Benchmark is static (200 questions); dynamic leakage into training data over time is a risk
  • Tasks with "Temporal and Contextual Invariance" by design exclude dynamic, rapidly-changing information domains
  • Error analysis of failure modes (incomplete planning, lack of reflection, faulty evidence use) is qualitative; no formal taxonomy with quantified attribution rates is provided in the excerpt
  • Human baseline is measured on a 20-question subset, not the full 200

Relevance to Agentic AI / LLM Agents

WideSearch directly targets the reliability and completeness gaps of tool-using LLM agents at operational scale—a frontier challenge that becomes critical as agentic systems move from demos to production workflows. The finding that item-level F1 (~80%) far exceeds task-level SR (~7%) has architectural implications: it suggests that current single-agent loops lack the planning depth and reflective self-correction needed to guarantee zero-omission coverage, and that multi-agent parallelism with cross-validation is a promising design direction. The benchmark also highlights that the standard DeepResearch framing (synthesize a report) is misaligned with a large class of real user needs—cataloguing and completeness—which require different agent architectures and evaluation criteria. For researchers building or evaluating LLM agents, WideSearch provides a rigorous, objectively scorable testbed that complements narrow-fact and synthesis benchmarks.