Skip to content

Scalable LLM Agent Tool Access in the Cloud

🕒 Published (v1): 2026-07-17 03:40 UTC · Source: Arxiv · link

Why this paper was selected

MCP at cloud scale; concrete infra for tool-calling agents — immediately buildable

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper presents a production cloud-scale gateway for the Model Context Protocol (MCP) deployed at Alibaba Cloud, solving the core infrastructure bottlenecks that prevent LLM agents from accessing thousands of tools reliably. The gateway interposes between MCP clients and heterogeneous backends to handle legacy API bridging, protocol incompatibilities, context-efficient tool recommendation, and stateful session routing — reducing tool selection tokens by \(23.8\times\) and time by \(8.9\times\).

Problem

Four compounding limitations prevent MCP from scaling to real cloud environments: 1. \(O(10^6)\) legacy OpenAPI services are not natively callable via MCP, but MCP servers number only \(O(10^4)\). 2. Four MCP specification revisions in 13 months produce transport-layer (HTTP+SSE vs. Streamable HTTP) and authentication incompatibilities across deployed clients and servers. 3. The "expose-all" approach of inlining all tool schemas into the LLM prompt fails at scale: 120 VPC tools consume >166k tokens for Qwen-Plus/Qwen3-MAX; latency exceeds 100 s for Qwen3-MAX at 120 tools; the no-call rate rises from 8.0% to 19.0% as the mounted tool count grows (analogous to the lost-in-the-middle phenomenon). 4. Stateful MCP server replicas require session affinity routing, which the default direct-connect model forces onto every client, adding per-client engineering overhead.

Method

A Layer-7 gateway is interposed between MCP clients and all backends, breaking the direct-connect model with four building blocks:

Protocol Adaptation. Each OpenAPI operation is compiled into exactly one MCP tool (one-to-one mapping) via a two-layer translation: data-layer (MCP primitives ↔ OpenAPI calls) and transport-layer (HTTP+SSE ↔ Streamable HTTP bridging). Adaptation layers are decoupled so transport churn does not require data-layer rewrites.

Function Offloading. The gateway centralizes ingress authentication (unified interface), egress credential mapping (identity → backend-specific token/API key/OAuth/JWT), and fine-grained identity-to-tool access control, removing this logic from individual backends.

Hybrid Tool Recommendation. Instead of exposing the full catalog, the gateway supports iterative, query-based retrieval. A dual-path index combines dense semantic embeddings with lexical (BM25-style) matching on tool names, which often encode functionality explicitly. A domain-aware query-rewrite step extracts domain knowledge from the query to close the gap when generic embeddings miss domain-specific terminology.

Session-Aware Load Balancing. A centralized store maintains the session-to-backend mapping shared across all gateway replicas. A Pub/Sub mechanism synchronizes streaming responses across gateway nodes to preserve long-lived SSE connections for stateful backends.

Key Contributions

  • Production-deployed cloud-scale MCP gateway supporting \(3{,}000+\) tools with sub-millisecond protocol conversion overhead (\(\text{P50} \le 143\,\mu\text{s}\)).
  • Hybrid semantic–lexical tool recommendation achieving 98% Top-15 recall at \(O(1\text{k})\) mounted tools with latency \(< 250\,\text{ms}\), reducing agent token cost \(23.8\times\) and tool-selection time \(8.9\times\).
  • Session-aware routing with centralized-store synchronization and Pub/Sub cross-gateway response forwarding, transparent to both clients and backends.
  • Production lessons: persistent gateway–backend connections cut mean/P90/P99 latency by 51.7%/55.7%/56.5%; LRU semantic caching cuts recommendation latency by up to 60%; tool-dependency-aware knowledge caching achieves 100% prerequisite tool-chain completeness; domain-aware query rewriting improves per-tool recommendation accuracy from 80.8% to 99.7%.

Results

  • Protocol conversion overhead: \(\text{P50} \le 143\,\mu\text{s}\) per call.
  • Tool recommendation recall: 98% Top-15 at \(O(1\text{k})\) tools, latency \(< 250\,\text{ms}\).
  • Token reduction: \(23.8\times\) fewer tokens consumed by the agent compared to expose-all.
  • Latency reduction for tool selection: \(8.9\times\) faster than direct LLM-based tool selection.
  • Connection persistence: mean/P90/P99 inference latency reduced by 51.7%/55.7%/56.5%.
  • Knowledge caching: 100% completeness on prerequisite tool chains.
  • LRU semantic cache: up to 60% reduction in average recommendation latency.
  • Query rewriting: accuracy improved from 80.8% to 99.7% with domain-aware rewriting.
  • Baseline (no gateway): no-call rate escalates from 8.0% to 19.0% as mounted tools scale from 15 to 120+ (Qwen3-MAX, VPC service).

Limitations

  • Implementation is tightly coupled to Alibaba Cloud's Layer-7 load balancing infrastructure; portability to other cloud stacks is undemonstrated.
  • The paper is truncated before the full implementation and evaluation sections, so detailed ablations and failure-mode analyses are not visible.
  • Tool recommendation recall (98% Top-15) means ~2% of relevant tools are missed per query, which can cause task failures in tightly constrained multi-step agent workflows.
  • The one-to-one OpenAPI-to-MCP mapping is explicitly chosen to minimize ambiguity but may produce very large tool catalogs with fine-grained, overlapping tools — a potential tradeoff not fully analyzed.
  • Session synchronization via a centralized store introduces a consistency/latency tradeoff under high cross-gateway traffic that is acknowledged but not fully characterized in the visible text.

Relevance to Agentic AI / LLM Agents

This work directly attacks the production infrastructure gap that prevents tool-using LLM agents from scaling beyond toy settings: the context window is a hard constraint that collapses under realistic enterprise tool catalogs, and this gateway's hybrid retrieval + iterative discovery pattern is a concrete, deployable answer. The 98% Top-15 recall with \(23.8\times\) token reduction demonstrates that effective tool grounding does not require full schema exposure, which has direct implications for designing agent memory and tool-use pipelines. The session-aware routing work is particularly relevant for multi-turn agentic workflows where backends maintain state across ReAct-style select–execute loops. More broadly, the MCP gateway architecture represents an emerging infrastructure pattern — analogous to API gateways in the web era — that will likely become a standard component in production agentic systems.