Skip to content

MCPEvol-Bench: Benchmarking LLM Agent Performance Across Dynamic Evolutions of MCP Servers

🕒 Published (v1): 2026-07-16 07:09 UTC · Source: Arxiv · link

Why this paper was selected

MCP is now core agent infrastructure; first benchmark tracking agent perf across server version drift

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MCPEvol-Bench is a benchmark that evaluates LLM agent robustness when MCP server tool interfaces evolve over time. Using 11 mutation operators derived from empirical study of 515 real MCP servers, it reveals that even frontier models like GPT-5.4 and Claude-Sonnet-4-6 degrade by ~14% in task fulfillment across evolution stages. The benchmark exposes a critical blind spot in current static-environment agent evaluations.

Problem

Existing tool-use benchmarks (ToolBench, MCPEval, MCP-Bench) assume a static toolset, but real-world MCP servers undergo continuous evolution: 20.7% of remotely hosted servers become unavailable within three months, and 54.6% of tools in versioned repositories are modified or deprecated. No prior benchmark evaluates whether LLM agents can adapt their workflows when tool interfaces, parameters, or descriptions change between versions.

Method

Empirical Study: Monitored 1,869 remote MCP servers over 3 months (weekly) and analyzed 9,273 historical NPM versions of 515 repositories to identify dominant evolution patterns (P1–P6 are platform-independent).

Mutation Operators: 11 operators at three hierarchical levels — TOOL (Tool Addition, Replacement, Deletion, Integration), PARAM (Flexible Expansion, Constraint Mutation, Parameter Pruning, Interface Refactoring), and DESC (Tool/Parameter/Joint Description Update) — encapsulated as LLM prompt templates.

LLM-Driven Evolution: Claude-Opus-4-5 performs AST-based code anchoring via server.tool registration syntax, selects mutation operators, and modifies source code iteratively. DeepSeek-Chat validates mutated tools via multi-case coverage testing. This generates multi-version MCP servers without broken-dependency failures.

Benchmark: 123 MCP servers (1,272 tools, 9 domains), 201 tasks requiring on average 4.37 tool calls across 3.76 servers. Each task is evaluated at three evolution stages: Early (original), Middle (3 rounds), Late (5 rounds). Tasks are formalized as a POMDP \((U, S, A, O, T, R, \Sigma)\). The Evolutionary Competency Score (ECS) is:

\[\text{ECS} = \frac{1}{|U|} \sum_{u \in U} \left( \mu\!\left(\{S_{u,i}^{TF}\}_{i=1}^{N}\right) - \sigma\!\left(\{S_{u,i}^{TF}\}_{i=1}^{N}\right) \right)\]

penalizing both low mean performance and high cross-version variance.

Key Contributions

  • Large-scale empirical study of 515 MCP servers identifying platform-independent evolution patterns and quantifying their prevalence (54.6% tool churn)
  • 11 mutation operators covering TOOL/PARAM/DESC levels that fully cover observed evolution patterns and compose to model complex scenarios
  • MCPEvol-Bench: 123 multi-version MCP servers + 201 tasks with three-stage evaluation pipeline; first benchmark to assess dynamic tool adaptability
  • Evaluation of 12 LLMs revealing universal performance degradation and identifying cognitive modules (reflection, planning, memory) as the key mitigants
  • ECS metric that jointly captures mean performance and cross-version stability, separating genuine adaptability from invariant low performance

Results

  • GPT-5.4: Task Fulfillment drops from 7.23 (Early) → 6.24 (Late), −13.7%; Planning Effectiveness drops from 5.71 → 3.87
  • Claude-Sonnet-4-6: 7.22 → 6.18, −14.4%
  • Claude-Opus-4-6: Highest ECS of 6.09, most stable across evolution rounds
  • Gemma-4-31B-it achieves ECS 4.45 vs. GPT-5.1's 4.32 despite lower baseline task fulfillment (5.05 vs. 6.28)
  • Weak models (Qwen3.5-9B, Llama-3.3-70B) show task fulfillment <4.0 and are insensitive to evolution (low variance but low performance)
  • Planning errors increase by 34.1% and reasoning errors by 35.6% from Early → Late stage; tool misalignment and syntax violations do not increase significantly
  • TOOL-level mutations (additions/modifications) severely impair performance; removing redundant tools/parameters has negligible impact

Limitations

  • Evolution simulation relies on LLMs (Claude-Opus-4-5 for mutation, DeepSeek-Chat for validation), introducing potential artifacts not present in human-authored updates
  • Fixed MCP server sets per task bypass dynamic tool retrieval, isolating evolution impact but not capturing compounded retrieval failures
  • Benchmark covers 9 functional domains but only 123 servers and 201 tasks — may not generalize to all MCP ecosystems
  • The paper text is truncated; full results on reflection/planning/memory module ablations and cross-operator analysis are not fully available here
  • Evaluation relies on LLM-as-judge (DeepSeek-Chat), which may introduce evaluation bias despite rubric grounding

Relevance to Agentic AI / LLM Agents

This work directly addresses a gap in agentic system robustness: production MCP-based agents will inevitably encounter tool interface drift, yet no prior benchmark measured this failure mode. The finding that degradation is concentrated in planning and reasoning errors (not tool selection or syntax) suggests that future agent architectures should prioritize context re-interpretation over static tool routing. The ECS metric and mutation operators provide a reusable evaluation and stress-testing framework applicable to any tool-augmented LLM system beyond the MCP ecosystem.