MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

MCP-based benchmark for realistic multi-step tool use and coordination

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

MCP-Bench is a benchmark for evaluating LLM agents on realistic multi-step tool-use tasks via Model Context Protocol (MCP) servers. It connects agents to 28 live MCP servers exposing 250 tools across 11 domains, using fuzzy task descriptions that omit explicit tool references to stress-test planning. Evaluation of 20 frontier models reveals persistent weaknesses in multi-hop planning and cross-domain orchestration.

Problem¶

Existing tool-use benchmarks (ToolBench, BFCL v3, τ-Bench, MCP-RADAR, MCPEval) suffer from: isolated APIs with poor natural I/O coupling, shallow few-step workflows, limited domain coverage, and tasks with explicit tool names/steps that fail to test authentic planning under underspecified instructions. No prior benchmark combines fuzzy instruction handling, information grounding, multi-goal objectives, and cross-domain orchestration at scale.

Method¶

MCP-Bench is formalized as a POMDP \((S, A, O, T, R, U, \Sigma)\) with a multi-turn execution loop (up to \(T_{max}=20\) rounds) implementing \(\pi_{plan} \to \pi_{exec} \to \pi_{compress} \to \text{state update}\). Tasks are constructed via a three-stage synthesis pipeline: (1) dependency chain discovery — analyzing tool I/O signatures to find natural and scenario-based chains (linear, parallel, hybrid); (2) automatic quality filtering — LLM scoring discards tasks below solvability 9.0/10 or utility 5.0/10; (3) task description fuzzing — rewrites structured instructions into high-level natural language omitting tool names and execution steps while preserving numerical parameters. Each task appends 10 distractor servers (~100 extra tools) to test tool-selection specificity. Evaluation is two-tier: rule-based metrics (tool name validity rate \(R_{valid}\), schema compliance rate \(C_{schema}\), execution success rate \(R_{success}\)) plus LLM-as-a-Judge (o4-mini) scoring four axes — task fulfillment, information grounding, tool appropriateness, parameter accuracy, dependency awareness, parallelism/efficiency — with 5× prompt-shuffled rubrics averaged to reduce ordering bias.

Key Contributions¶

28 live production-grade MCP servers with 250 tools across 11 domains, enabling both intra-server dependency chains and cross-server orchestration
Structured LLM-driven task synthesis pipeline producing 104 tasks (56 single-server, 30 two-server, 18 three-server) with fuzzy instruction variants
Two-tier evaluation framework combining rule-based execution checks with rubric-driven LLM-as-a-Judge scoring and prompt-shuffling for stability
Large-scale empirical study of 20 SOTA LLMs exposing systematic weaknesses in multi-hop planning, information grounding, and parallel tool exploitation

Results¶

Top overall scores (averaged across single- and multi-server tasks): gpt-5 (0.749), o3 (0.715), gpt-oss-120b (0.692), gemini-2.5-pro (0.690), claude-sonnet-4 (0.681)
Bottom performers: llama-3-1-8b-instruct (0.428), llama-3-2-90b-vision (0.495), nova-micro-v1 (0.508)
Schema understanding is near-saturated across models (valid tool name rate: 96–100%; schema compliance: 85–100%; execution success: 87–99%), indicating basic API grounding is solved
Parallelism and efficiency is the weakest axis universally — even gpt-5 scores only 0.339; o3 scores 0.359; most models fall below 0.35, indicating failure to exploit parallel execution opportunities
Information grounding shows the widest spread: llama-3-1-8b at 0.295 vs. gpt-5 at 0.828, revealing a strong capability gap in grounding answers in intermediate tool outputs
Task fulfillment gap: llama-3-1-8b (0.261) vs. gpt-5 (0.677), with mid-tier models (gpt-4o: 0.394, gemini-2.5-flash-lite: 0.412) substantially behind frontier models

Limitations¶

104 tasks is a relatively small evaluation set, limiting statistical power for fine-grained model differentiation
Task synthesis and quality filtering use o4-mini, potentially biasing the benchmark toward tasks that o4-class models handle well
Distractor servers are fixed at 10; generalization to different distractor counts or adversarially crafted distractors is untested
LLM-as-a-Judge (o4-mini) for scoring introduces model-dependent evaluation bias, even with prompt shuffling
Multi-server task combinations are limited (Table 11); 3-server tasks number only 18, restricting coverage of highly complex cross-domain scenarios
Live MCP servers may change over time, threatening benchmark reproducibility

Relevance to Agentic AI / LLM Agents¶

MCP-Bench directly addresses the agentic capability evaluation gap by providing the first large-scale benchmark grounded in the emerging MCP ecosystem standard, which is rapidly becoming the de facto protocol for LLM-tool integration. The fuzzy instruction design specifically tests the planning and tool-discovery abilities that distinguish true agents from glorified API callers — a distinction central to the agentic AI research agenda. The finding that parallelism and efficiency remain universally weak (even at frontier scale) points to a concrete open problem: agents do not yet exploit concurrency in multi-tool workflows, which is critical for real-world latency and cost. Results also quantify how far even top models fall short of reliable information grounding (~0.83 for gpt-5 vs. a perfect 1.0), directly informing research on retrieval-augmented and evidence-driven agentic pipelines.