Skip to content

MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR; stress-tests realistic MCP use with full read/write/deep interaction coverage

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MCPMark is a benchmark of 127 expert-curated tasks across five real MCP servers (Notion, GitHub, Filesystem, PostgreSQL, Playwright) designed to stress-test LLM agents on multi-step, CRUD-diverse workflows. The best model (gpt-5-medium) achieves only 52.56% pass@1 and 33.86% pass^4, exposing deep reliability gaps in current frontier models. A minimal evaluation agent harness (MCPMark-Agent) ensures model-agnostic, reproducible assessment via programmatic verification.

Problem

Existing MCP benchmarks focus on read-heavy or shallow-interaction tasks and use either LLM-as-judge evaluation or synthetic environments, failing to capture real-world multi-step workflow complexity. This makes it impossible to reliably probe the planning, reasoning, long-context, and tool-use capabilities required for authentic agentic use of MCP in production systems.

Method

Each of the 127 tasks is grounded in a curated initial environment state (38 unique states across 5 official MCP servers) paired with a natural-language instruction and a programmatic verification script (~209.8 lines of code on average). Tasks are constructed via a four-stage human–AI collaborative pipeline: (I) Exploration — human expert and a task-creation agent jointly survey the environment; (II) Evolvement — the agent proposes or refines task instructions to increase complexity; (III) Verification — the agent drafts a verification script, which a task-execution agent runs and iteratively refines; (IV) Iteration — steps II–III repeat to progressively raise difficulty while preserving verifiability. Quality is enforced by cross-review and a month-long community check.

Evaluation runs through MCPMark-Agent, a minimal tool-calling loop built on LiteLLM and the MCP Python SDK. It converts MCP tool schemas to OpenAI function-call format and routes to each model's native API. The loop terminates when the model issues a final response without further tool calls (max 100 turns, 3600-second timeout). Three metrics are reported: pass@1 (single-run success), pass@4 (success in at least one of four runs), and pass^4 (success in all four runs — a consistency indicator).

Key Contributions

  • A benchmark of 127 high-quality, CRUD-diverse tasks spanning five production MCP environments, each with a carefully designed initial state and a programmatic verifier.
  • A human–AI collaborative pipeline for task construction, averaging 3–5 expert-hours per task with 10 domain-expert contributors.
  • MCPMark-Agent: a lightweight, model-agnostic evaluation harness providing standardized, reproducible MCP agent assessment.
  • Systematic evaluation of 20+ frontier models (proprietary and open-source), revealing that the best achieves only 52.56% pass@1 and 33.86% pass^4.
  • Identification of a local-vs-remote environment gap and a robustness gap (pass^4 \(\ll\) pass@1) as key failure modes.

Results

  • Best model: gpt-5-medium — 52.56% pass@1, 68.50% pass@4, 33.86% pass^4.
  • claude-sonnet-4: 28.15% pass@1, 44.88% pass@4, 12.60% pass^4.
  • o3: below 30% pass@1, below 15% pass^4.
  • Best open-source: qwen3-coder-plus at 24.80% pass@1.
  • Several open-source models: below 10% pass@1.
  • Local vs. remote gap: gpt-5-medium achieves 76.19% on PostgreSQL, 57.50% on Filesystem vs. below 25% on Notion/GitHub for most models.
  • Interaction cost: average 16.2 turns and 17.4 tool calls per task; kimi-k2-instruct averages 26.95 turns and 26.22 tool calls.
  • Comparison with prior MCP benchmarks: MCPEval (Synthetic/Hybrid, N/A turns), LiveMCPBench (3.2 turns), MCP-Universe (6.8 turns, read-heavy), LiveMCP-101 (5.4 turns, LLM-as-judge) vs. MCPMark (16.2 turns, CRUD-diverse, programmatic).

Limitations

  • Covers only five MCP environments; generalization to other MCP servers (e.g., email, calendar, code execution) is untested.
  • 127 tasks is a modest scale; tail-distribution coverage may be incomplete.
  • MCPMark-Agent is deliberately minimal, omitting production-oriented optimizations (planning, memory, reflection), so absolute numbers underestimate what optimized harnesses could achieve.
  • Remote-service tasks (Notion, GitHub) depend on live APIs, introducing potential non-determinism from external state changes.
  • No small open-source models (≤100B) are evaluated, leaving the open-source landscape partially characterized.

Relevance to Harnesses / Meta-Harnesses

MCPMark-Agent is itself a concrete example of a minimal evaluation harness: a tool-calling loop built on LiteLLM + MCP Python SDK that wraps arbitrary LLMs in a standardized execution environment without task-specific heuristics, directly relevant to harness design principles. The benchmark's four-stage human–AI task-construction pipeline is a meta-harness pattern — an orchestrated pipeline of creation, execution, and verification agents that produces other harnesses' inputs. The programmatic verifier design (state initialization → agent execution → state-diff check → environment reset) mirrors the lifecycle management concerns central to meta-harness architecture. Practitioners building evaluation or orchestration harnesses over MCP can use MCPMark both as a stress-test for their harness frameworks and as a reference design for sandboxed, reproducible multi-step agent evaluation.