Skip to content

InfoMosaic-Bench: Evaluating Multi-Source Information Seeking in Tool-Augmented Agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Multi-source information seeking benchmark beyond open-web search

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

InfoMosaic-Bench is the first benchmark targeting multi-source information seeking in tool-augmented agents, requiring agents to combine general-purpose web search with domain-specific MCP tools across six domains. Evaluating 14 state-of-the-art LLMs, the benchmark reveals that even GPT-5 achieves only 38.2% accuracy, and 22.4% of failures stem from incorrect tool usage or selection rather than reasoning failures.

Problem

Existing LLM agent benchmarks either evaluate single-channel web search (BrowseComp, WebWalkerQA) or isolated tool-call correctness (Ï„-Bench, MCP-Bench), leaving unanswered whether agents can reliably integrate evidence from heterogeneous domain-specific tools alongside general web search. Real-world tasks in medicine, finance, and geospatial domains require precise, verifiable knowledge unavailable from open-web sources, yet no benchmark systematically tests this multi-source integration capability.

Method

InfoMosaic-Flow is a two-stage agentic synthesis pipeline built on an organizer–workers architecture:

  • Stage 1 (Information Seeking): A synthesizer (organizer) proposes scenarios from seed data (Wikipedia, NCI IDs, etc.), issues high-level subtasks to domain-specific executor workers, collects verified tool outputs, and integrates them into coherent multi-condition QA pairs. The synthesizer remains tool-agnostic while each executor freely selects among domain tools.
  • Stage 2 (Iterative Refinement): A Refiner decomposes each problem into individual conditions; a Verifier (web-search-only) attempts to solve them independently. If any condition yields the answer via web lookup, the Refiner applies condition fuzzing—rewriting, augmenting, or combining conditions—until no single condition or web-only query suffices.

Quality control layers: automated tool-call threshold filtering, answer–evidence consistency checks (answers must be exactly derivable from tool outputs), coherence filtering, and human review. The final benchmark contains 621 tasks across 6 domains (medical/biology, finance, maps, video, web, multi-domain) with 77 MCP tools across 7 servers. Tasks are bilingual (63.1% English, 36.9% Chinese).

Key Contributions

  • First benchmark explicitly targeting multi-source information seeking with contemporary MCP-based domain tools, rather than single-source web search or isolated API correctness.
  • InfoMosaic-Flow: a scalable organizer–workers agentic pipeline that grounds task conditions in verified tool outputs and enforces cross-source dependencies, producing non-trivial tasks by construction.
  • Empirical evaluation of 14 LLMs (7 closed, 7 open-source) revealing tool-usage failure modes quantitatively: 22.4% of failures are tool selection or usage errors rather than reasoning failures.
  • Two complementary metrics: strict end-to-end Accuracy and fine-grained Pass Rate (condition-level subgoal success).

Results

  • Best web-only performance (GPT-5): 38.2% accuracy, 67.5% pass rate across all domains.
  • Closed vs. open-source gap: closed-source models exceed open-source by 15–20% on accuracy.
  • Pass rate > accuracy consistently, indicating agents satisfy some conditions but fail to synthesize a correct final answer.
  • Domain tool effects (GPT-5, Table 3):
  • Map: +7.41pp; Video: +10.0pp (structured/exclusive signals benefit from domain tools)
  • Medical/Biology: −9.73pp; Finance: −9.0pp (domain tools degrade performance)
  • Multi-domain: −1.94pp (cross-source orchestration overhead)
  • Overall GPT-5 gain from domain tools: +0.43pp (marginal)
  • Web-only domain variance: Medical/Biology 53.1%, Finance 41.0%, Map 40.7%, Video 36.0% (GPT-5)
  • Tool failure breakdown: 22.4% of all failures attributed to wrong tool usage (usage errors) or wrong tool selection (selection errors).
  • Open-source SOTA (GLM-4.5): 20.61% accuracy web-only; +0.90pp with domain tools.

Limitations

  • Benchmark size is relatively small (621 tasks), potentially limiting statistical reliability within individual domains (83–135 samples each).
  • The iterative refinement stage uses web-search-only as the shortcut-detection proxy; a more capable adversary might over-prune valid tasks.
  • Domain coverage (6 domains, 77 tools) may not generalize to other high-stakes fields (law, code, scientific literature).
  • Evaluation relies on LLM-as-judge for semantic answer matching, introducing potential judge bias.
  • The organizer–workers synthesis architecture itself may introduce systematic biases toward the question types that organizer LLMs naturally generate.
  • Results are a snapshot in time; rapidly improving frontier models may saturate some domains quickly.

Relevance to Agentic AI / LLM Agents

InfoMosaic-Bench directly benchmarks the MCP-era capability gap: whether agents can orchestrate heterogeneous tool ecosystems rather than relying on single-channel retrieval. The finding that domain tools yield only marginal average gains (+0.43pp for GPT-5) despite being the correct information source is a critical empirical result—tool availability is not the bottleneck; tool planning, selection, and parameterization are. The 22.4% tool-error failure rate quantifies a concrete, actionable deficit in current agent architectures, distinguishing it from reasoning failures. This work directly informs research on tool-use policies, multi-step planning, and the design of agentic frameworks that must compose web search with MCP-enabled domain APIs.