A Benchmark for Deep Information Synthesis

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Benchmark for deep information synthesis across tool use, browsing, and code

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DEEPSYNTH is a 120-task benchmark designed to evaluate LLM-based agents on real-world information synthesis tasks requiring multi-source web retrieval, structured reasoning, and insight generation across 67 countries. State-of-the-art models including GPT-5.2-Pro and dedicated deep research agents achieve at most F1 = 8.97 and LLM-judge = 17.5%, exposing a fundamental gap between current agent capabilities and genuine synthesis competence. The benchmark reveals that the primary bottleneck is not reasoning ability per se but access to and correct integration of information from diverse, underrepresented sources.

Problem

Existing agentic benchmarks focus on shallow fact retrieval, single-source queries (e.g., Wikipedia-only), or artificially constructed English-only questions that can be answered via direct lookup. No benchmark adequately tests the ability to gather information from multiple heterogeneous web sources, combine structured and unstructured data, and perform analytic synthesis (correlation, trend detection, anomaly detection) to produce non-retrievable, verifiable answers—the kind of task that occupies expert analysts for hours.

Method

DEEPSYNTH tasks are constructed via a four-stage expert annotation pipeline: (1) data source identification — 16 experts curate 223 official data sources across 7 domains (socio-economic, finance, environment, science, education, transportation, political); (2) hypothesis generation — annotators formulate plausible, non-surface-level hypotheses for each source; (3) hypothesis validation — a second pass verifies that the data supports the hypothesis and that a verifiable answer exists; (4) task formulation — annotators write the question, gold-answer JSON, intermediate reasoning chain (avg. 7.54 steps), supporting URLs, and any required code snippets or formulas. A cross-validation stage retains only tasks where two independent annotators produce identical answers, yielding 120 final tasks. Answers are structured as JSON key-value dictionaries to enable automatic exact-match and partial (F1) evaluation, supplemented by an LLM-as-judge soft metric (GPT-4.1) that tolerates small numerical deviations (1–5.5%).

Key Contributions

• DEEPSYNTH benchmark: 120 expert-annotated, multi-source information synthesis tasks spanning 7 domains and 67 countries, each requiring an average of 4.2 web pages and 7.54 reasoning steps (avg. 5.5 hours to annotate per task).
• Evaluation of 11 SOTA LLMs (o4-mini, GPT-4.1, GPT-5.2-Pro, Gemini-2.5-Pro, DeepSeek-R1, etc.) and 3 deep research agentic frameworks (o3-deep-research, smolagents, OWL), revealing universal near-zero performance.
• Ablation studies isolating the contribution of individual tool capabilities (web search, browsing, document processing, code execution) and ground-truth planning (intermediate steps).
• Multi-regional performance breakdown exposing systematic failure on African and South American tasks due to underrepresented data sources.

Results

• Best LLM baseline: GPT-5.2-Pro — F1 = 8.70, Exact Match = 6.25, LLM-judge = 6.67%.
• Best agentic framework: o3-deep-research — F1 = 8.97, EM = 2.50, LLM-judge = 17.5%.
• Under strict exact match, nearly all models score 0, meaning no model solves even one task perfectly in the standard setting.
• o3-deep-research outperforms base o3 by +5.68 F1 and +2.50 EM, confirming tool augmentation helps but remains insufficient.
• Best@5 with Smolagents (GPT-4.1) reaches 25.0% LLM-judge vs. 7.5% at Best@1; self-consistency@5 collapses to 5%, showing high output variance and low reliability.
• Providing ground-truth intermediate steps (planning oracle) substantially boosts performance: GPT-4.1 jumps from F1 = 3.46 to 9.36; smolagents + GPT-4.1 from 3.75 to 10.50 — implicating planning failure as a root cause.
• Removing web search causes the largest single-tool drop (−1.81 F1); all tools contribute meaningfully.
• Performance on African-region tasks: 0.0 F1 for all models except o3-deep-research (0.0 as well), confirming severe geographic bias.

Limitations

• Only 120 tasks — small enough that per-task variance significantly affects aggregate scores.
• Geographic skew: 38.3% of tasks are Europe-centric; Africa and South America are severely underrepresented (8.3% and 5%), limiting generalizability claims about global coverage.
• Annotation by 16 domain experts from a single research network introduces potential selection bias in hypothesis choice and domain framing.
• Verifiable answers are constrained to JSON key-value format, which may underrepresent the open-ended reasoning that real synthesis tasks require.
• Benchmark snapshots live URLs; web content drift over time may invalidate gold answers or require periodic re-validation.
• The annotation process is extremely expensive (5.5 hours/task), limiting future scaling.

Relevance to Agentic AI / LLM Agents

DEEPSYNTH directly targets the frontier capability gap in agentic AI: the ability to plan multi-step information gathering, integrate heterogeneous sources (tables, documents, live web pages), and perform analytic synthesis rather than retrieval. The near-zero scores across all SOTA models and dedicated research agents (o3-deep-research, smolagents, OWL) provide strong empirical evidence that current ReAct-loop and tool-augmented architectures are insufficient for real-world knowledge-work tasks. The planning oracle ablation (intermediate steps) quantifies how much agents are bottlenecked by planning versus execution, directly informing architectural choices for next-generation agents. For researchers developing LLM agents with tool use, multi-step reasoning, or deep research capabilities, DEEPSYNTH offers a concrete, high-ceiling evaluation target aligned with practical analyst workflows.