When Search Agents Should Ask: DiscoBench for Clarification-Aware Deep Search¶

🕒 Published (v1): 2026-06-26 02:57 UTC · Source: Arxiv · link

Why this paper was selected

Clarification-aware deep search benchmark; underexplored but critical for production search-agent reliability

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

DiscoBench is a benchmark for evaluating whether LLM-based search agents can proactively detect query ambiguity, ask targeted clarification questions, and recover correct multi-step reasoning trajectories through user interaction. The benchmark contains 211 samples with 463 ambiguity instances across 11 domains and four ambiguity types. Experiments on 11 frontier models reveal that ambiguity detection and effective clarification are distinct capabilities, and that over-searching without asking is often worse than guessing directly.

Problem¶

Existing deep search benchmarks (GAIA, BrowseComp, AgentBench) assume complete, explicit user queries, ignoring that real queries are frequently vague, underspecified, or factually incorrect. In multi-step retrieval, a single ambiguous node can propagate errors through the entire reasoning chain, wasting computation and producing wrong final answers. No prior benchmark models ambiguity as a dynamic phenomenon arising at intermediate retrieval checkpoints—earlier work either uses static corpora (AmbigQA, ASQA) or restricts interaction to closed sandbox environments (IN3, UserBench).

Method¶

DiscoBench constructs interactive multi-hop QA tasks via a two-phase semi-automatic pipeline:

Seed data preparation: 11 knowledge domains are sampled; multi-hop reasoning chains are built by LLM-assisted expansion over encyclopedic and search-engine sources, then human-verified.
Ambiguity injection: Deterministic hops are transformed into ambiguous checkpoints (CPs) by replacing specific constraints with shared attributes shared by multiple plausible candidate entities. Four ambiguity types are injected: Entity (multiple entities match the description), Version (temporally or version-specific state), Criteria (conflicting ranking/evaluation standards), and Factual Inaccuracy (description conflicts with objective fact). Discriminative clue facts are generated via retrieval-augmented LLMs and human-verified.

At each CP the agent must choose action \(a_i \in \{\texttt{SEARCH}, \texttt{ASK}, \texttt{ANSWER}\}\). An ambiguous CP requires the agent to invoke ASK(·), whereupon a user simulator (Gemini-3-Flash-Medium) releases a pre-defined discriminative clue; the agent then refines its search and resolves the ambiguity before answering.

Evaluation covers four dimensions: task utility (end-to-end accuracy, checkpoint pass rate), ambiguity detection (detection accuracy, detection F1), interaction strategy (clarification effectiveness CE-A/CE-B, average ask turns), and cost efficiency (token and tool-use turns). Two prompting settings are tested: Neutral (no hint about ambiguity) and Guided (explicit reminder to seek clarification).

Key Contributions¶

DiscoBench: 211 samples, 463 ambiguity instances, 11 real-world domains, four ambiguity types, difficulty stratified by number of ambiguity CPs (1/2/3).
An ambiguity-aware multi-turn evaluation framework with a user simulator that progressively releases discriminative clues, enabling unified measurement of detection, clarification quality, and interaction cost.
Empirical finding that ambiguity detection and effective clarification are distinct capabilities not reducible to retrieval or reasoning strength.
Identification of a dominant failure mode: agents repeatedly issue SEARCH instead of ASK, yielding lower success rates than simply guessing the answer directly.

Results¶

Best model (Doubao-Seed-2.0-Pro) achieves only 43.1% end-to-end accuracy under Neutral prompting; Gemini-3.1-Pro reaches 40.8%; most models fall below 40%.
Guided prompting raises average end-to-end accuracy from 28.6% → 33.7%, checkpoint pass rate from 50.1% → 57.6%, and detection F1 from 45.3% → 64.9% (averaged over 10 valid models).
Large gap between checkpoint pass rate and end-to-end accuracy (e.g., Claude-Opus-4.7: 57.0% CP vs. 39.8% accuracy), showing that local step success does not imply full trajectory recovery.
Factual Inaccuracy ambiguity is easiest to detect (explicit evidence conflict); Entity and Criteria types are hardest (no conflict signal, requires distinguishing among plausible candidates).
High tool-use frequency does not correlate with accuracy (Claude-Opus-4.7 has high tool calls but lower accuracy than Gemini and Doubao), confirming that strategic allocation matters more than search volume.

Limitations¶

Benchmark is primarily constructed in Chinese, limiting direct comparability with English-centric retrieval benchmarks and potentially introducing language-specific retrieval biases.
Only 211 samples, which may limit statistical power across 11 domains and four ambiguity types simultaneously.
Discriminative clue quality is manually curated and may not reflect the variability of real user responses (clues are pre-defined, not spontaneously generated by users).
The user simulator (Gemini-3-Flash-Medium) may not faithfully replicate diverse human clarification behaviors, potentially overestimating how well agents handle real interaction.
Difficulty is operationalized solely as the number of ambiguous CPs, not accounting for within-type difficulty variance.
Evaluation of GPT-5.4 was incomplete (37 neutral-prompt samples filtered by usage-policy), reducing comparability.

Relevance to Agentic AI / LLM Agents¶

DiscoBench directly targets a gap in agentic search: the ability to recognize when to stop retrieving and instead ask the user—a core decision-making skill for any interactive agent operating under incomplete information. The finding that repeated retrieval without clarification underperforms even naive guessing is a sharp practical signal for agent designers balancing tool-use cost against interaction cost. The explicit decomposition of ambiguity detection vs. effective clarification as separate capabilities provides a diagnostic framework for understanding where agent pipelines break down, beyond the usual retrieval-accuracy lens. This work connects to the broader line of proactive agent design, alongside benchmarks like GAIA2 and IDRbench, establishing that next-generation agentic systems need dedicated uncertainty-awareness and interaction-planning modules.