Skip to content

LiveResearchBench: A Live Benchmark for User-Centric Deep Research in the Wild

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Live benchmark with real-time web tasks for deep research agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LiveResearchBench introduces 100 expert-curated deep research tasks requiring real-time web search and synthesis, paired with DeepEval, a multi-protocol evaluation suite covering six quality dimensions. Evaluation of 17 frontier single- and multi-agent systems reveals that most current models fail to produce genuinely insightful research reports, with citation accuracy and analysis depth as the most persistent failure modes.

Problem

Existing deep research benchmarks suffer from one or more critical deficiencies: they are static (susceptible to pretraining contamination), domain-specific, require only short-form or closed-ended answers, or pose ambiguous queries that permit multiple valid interpretations—making fair cross-system comparison impossible. No benchmark simultaneously enforces user-centricity, unambiguity, time-variance, and multi-faceted search intensity.

Method

Benchmark construction (LiveResearchBench): A six-stage pipeline converts crowd-survey user intents into finalized queries: (1) user interviews and surveys, (2) expert-drafted initial questions aligned with domain/task distribution, (3) clarification question generation using two frontier models (OpenAI o3 Deep Research, Gemini Deep Research) to surface ambiguities, (4) human-expert query refinement, (5) GPT-5-generated unit-test checklists, and (6) a five-stage independent verification process with multiple annotator groups. Time-varying tasks embed a {{date}} placeholder replaced at evaluation time, ensuring queries remain contamination-resistant.

Evaluation suite (DeepEval): Six complementary dimensions assessed via four tailored protocols: - Checklist-based (binary 0/1 per item): ❶ Presentation & Organization (10-item error checklist), ❸ Coverage & Comprehensiveness (query-specific unit-test checklists) - Pointwise/additive (error-count deduction from 100): ❷ Factual & Logical Consistency, ❺ Citation Association - Pairwise comparison (5-dimension scoring vs. baseline): ❹ Analysis Depth (granularity of reasoning, multi-layer insights, critical evaluation, analytical evidence use, insight density) - Rubric-tree: ❻ Citation Accuracy (verifying ⟨statement, URL⟩ pairs)

Holistic 0–10 LLM scoring was ruled out after experiments showed <60% human agreement even with GPT-5/Gemini-2.5 Pro judges. An agent-ensemble-as-a-judge protocol uses Gemini 2.5 Pro and GPT-5 as independent judges (Claude 4 Sonnet showed inconsistent human alignment) and averages their scores.

Key Contributions

  • Four task-design principles for deep research benchmarks derived from user surveys: user-centric, unambiguous, time-varying, multi-faceted and search-intensive.
  • LiveResearchBench: 100 expert-curated tasks across 7 domains and 10 task categories (market analysis, literature review, policy evaluation, etc.), constructed with >1,500 hours of human labor, each with verified unit-test checklists.
  • DeepEval: six-dimension evaluation suite with per-metric protocol selection validated against human judgments; Presentation & Organization achieves 98.3% judge–human agreement.
  • Comprehensive evaluation of 17 SOTA systems (single-agent web search, single-agent deep research, multi-agent systems), exposing systematic failure modes and identifying components needed for reliable deep research.

Results

  • Most evaluated models fail to produce genuinely insightful reports; citation accuracy and analysis depth are the most persistent failure modes across all system classes.
  • Checklist-based Coverage evaluation achieves 100% human preference alignment (judges agree with or are tied with human assessors in all cases).
  • Holistic 0–10 scoring protocol yielded <60% human agreement even with best available judges, with analysis depth scores varying >50 points across runs—motivating the structured protocol design.
  • Single-agent vs. multi-agent system comparison reveals no consistent advantage for multi-agent architectures; which dimensions favor MAS remains an open question. (Specific per-system numeric scores are presented in the full paper but were not included in the provided text excerpt.)

Limitations

  • Benchmark is capped at 100 tasks; domain coverage may not be fully representative of all deep research use cases.
  • Evaluation relies on LLM judges (Gemini 2.5 Pro + GPT-5 ensemble); judge model availability and version drift can affect score reproducibility over time.
  • The {{date}} mechanism makes tasks time-varying but also means answers have no single fixed ground truth, complicating absolute score interpretation across evaluation dates.
  • Citation Accuracy requires fetching live URLs at evaluation time, introducing dependency on web availability and potential link rot.
  • Multi-agent system internals (Grok 4 Heavy, Manus) are proprietary, limiting analysis of what drives performance differences.

Relevance to Agentic AI / LLM Agents

LiveResearchBench directly targets the evaluation gap for long-horizon, multi-step agentic systems that must plan, retrieve from live web sources, and synthesize citation-grounded reports—a core capability frontier for AI agents. The systematic comparison of single-agent ReAct-style systems against multi-agent orchestrations with specialized planner/researcher/writer roles provides actionable signal on where MAS architectures do and do not add value. DeepEval's finding that holistic LLM-as-a-judge scoring is unreliable for open-ended agentic outputs, and that structured checklist/pairwise protocols are necessary, has direct implications for anyone building evaluation pipelines for agentic tasks. The contamination-resistant, time-varying task design addresses a fundamental reproducibility challenge in benchmarking agents that interact with live environments.