Skip to content

Demystifying Deep Search: A Holistic Evaluation with Hint-free Multi-Hop Questions and Factorised Metrics

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR; hint-free multi-hop queries with factorised metrics expose deep search agent gaps

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

WebDetective is a benchmark of 200 hint-free multi-hop questions over a controlled Wikipedia sandbox, designed to expose the true autonomous reasoning capabilities of deep-search agents. Existing benchmarks embed path or specification hints that reduce the task to execution rather than discovery. The paper also introduces EvidenceLoop, a multi-agent agentic workflow, and a factorised evaluation framework that decomposes pass rate into search sufficiency, knowledge utilisation, and refusal behaviour.

Problem

Two compounding flaws degrade existing multi-hop deep-search evaluation: (1) Path-Hinting (PH) benchmarks such as HotpotQA narrate the full reasoning chain in the question, turning reasoning into execution; Specification-Hinting (SH) benchmarks such as BrowseComp fingerprint the target entity via distinctive attributes, reducing the task to constraint satisfaction. (2) Evaluations collapse all behaviours into a single pass rate, obscuring whether failures stem from insufficient retrieval, weak synthesis, or absent refusal — preventing principled diagnosis and improvement.

Method

Benchmark construction. Starting from Wikipedia single-hop QA pairs (starting entity \(v_0\), answer \(v_n\)), the direct edge \(v_0 \to v_n\) is removed and BFS on the hyperlink graph finds the shortest alternative path \(v_0 \to v_1 \to \cdots \to v_n\). Each intermediate entity page is masked so \(v_i\) is only discoverable after visiting \(\text{page}(v_{i-1})\), eliminating corpus shortcuts. Three automated checks via Qwen-3-235B-A22B verify: (1) parametric inaccessibility (\(\text{LM}(q) \neq v_n\)), (2) evidence sufficiency (\(\text{LM}(q, E) = v_n\)), (3) evidence necessity (\(\text{LM}(q, E \setminus \{e_i\}) \neq v_n\) for all \(i\)). 200 questions survive human validation.

Factorised metrics. Performance is decomposed into: - Knowledge Sufficiency: whether the agent possesses all required evidence \(E = \{e_1,\ldots,e_n\}\) via search or verified parametric probing. - Search Score: extends sufficiency by crediting efficient use of parametric shortcuts (fewer hops than the reference path). - Good Refusal F1 (\(F1_{GR}\)): F1 over abstention when evidence is absent. - Knowledge Utilisation F1 (\(F1_{KU}\)): F1 over synthesis when evidence is present. - Generation Score: \(\frac{F1_{GR} + F1_{KU}}{2} \cdot \text{KnowledgeScore}\), weighted by sufficiency to prevent refusal-gaming. - Knowledge Degradation: Forget (parametric knowledge inaccessible in full context) and Lead-astray (noisy search context prevents synthesis).

EvidenceLoop baseline. A multi-agent workflow with \(R_{\max}\) refinement rounds: \(N\) parallel solver agents each explore up to \(B\) actions; an extraction agent distills findings; an aggregation agent builds \(C^{r+1}\). A persistent evidence memory \(M\) assigns unique Evidence IDs (EIDs) to every retrieved snippet. Answer proposals must decompose into atomic claims linked to EIDs, verified by a dedicated verification agent before acceptance.

Key Contributions

  • WebDetective: first benchmark eliminating both PH and SH hints via co-designed question formulation + entity-masked Wikipedia sandbox.
  • Factorised evaluation framework: separates search sufficiency, knowledge utilisation, and calibrated refusal, enabling per-failure-mode diagnosis.
  • EvidenceLoop: a diagnostically-derived agentic workflow with structured EID memory, parallel exploration, and explicit verification loops.
  • Large-scale evaluation: 25 frontier models compared under uniform conditions (40 tool calls, 32K context, 200 questions).
  • Behavioral profiling: six emergent model archetypes identified from metric interplay (e.g., high-search/low-synthesis, high-refusal/low-pass).

Results

  • Best Pass@1 is ~56%: o3-Pro 56.0%, GPT-5 and Grok-4 both 50.5%; Claude-Opus-4.1 44.5%; most models below 40%.
  • Search and synthesis are decoupled: GPT-5 achieves 80.0% Search Score but only 23.21% Generation Score; Qwen3-235B-Think posts 72.0% Search yet only 11.15% Generation Score and 21.5% Pass@1.
  • Good Refusal F1 is near-absent across the board: best is 53.95% (Doubao-1.6-Flash); GPT-5 scores 8.89%, o3-Pro 9.37%, o4-Mini 19.75%, Claude-Opus-4.1 28.57%.
  • EvidenceLoop (on DeepSeek-R1) yields Pass@1 = 25.0% vs. DeepSeek-R1 baseline's 20.0%, with improved Generation Score (12.61 vs. 10.57) but does not dominate all metrics.
  • Knowledge Degradation is substantial: many models show >40% Forget and >30% Lead-astray rates, indicating that possessing evidence does not guarantee correct synthesis.

Limitations

  • WebDetective contains only 200 questions, limiting statistical power for fine-grained cross-model comparisons.
  • The Wikipedia sandbox eliminates live-web variability; results may not transfer to open-web retrieval environments where masking is impossible.
  • EvidenceLoop is evaluated only with DeepSeek-R1 as the base model; generality across architectures is untested.
  • The parametric probing procedure (filling blanks in evidence chains) may itself be unreliable for very long chains or rare entities.
  • Domain distribution is skewed toward entertainment/humanities; scientific or technical multi-hop reasoning is underrepresented (6%).

Relevance to Harnesses / Meta-Harnesses

EvidenceLoop is a directly relevant example of a multi-agent harness architecture: it coordinates parallel solver agents, an extraction module, an aggregation module, and a verification agent under a shared memory store with structured identifiers (EIDs), forming a loop-based meta-harness pattern. The paper's broader contribution — using factorised diagnostic metrics to derive harness architecture from observed failure modes — exemplifies the "diagnostic-driven harness design" methodology where benchmark analysis prescribes which control-flow and memory components to add. The EID-based evidence buffer with on-demand full-document retrieval is a concrete pattern for evidence management in multi-hop harnesses. The distinction between exploration breadth and synthesis quality underscores why harnesses must treat retrieval and reasoning as separately schedulable sub-processes.