Optimizing Agentic Language Model Inference via Speculative Tool Calls¶

🕒 Published (v1): 2025-12-17 18:22 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper identifies tool-call interruptions as the dominant inference bottleneck for LM-based agents—each call evicts the sequence, stalls generation, and forces re-prefill. It introduces speculative tool calling in two variants (client-side and engine-side) that pre-execute tools using a smaller model in parallel with main-model generation, achieving up to 196 tok/s throughput improvement and up to 21% per-agent time savings.

Problem¶

Standard tool-calling creates a strict sequential dependency: generation halts at every tool call, the sequence is evicted from the inference engine's batch, and the KV-cache must be rebuilt on re-entry. In multi-tenant settings with many concurrent agents, these eviction/rescheduling overheads compound and dominate end-to-end latency. Existing optimizations (parallel tool calls, prefix caching) reduce but do not eliminate these costs.

Method¶

Client-side speculative tool calling (Algorithm 2): When a prompt is submitted to the main model M, λ parallel requests are also sent asynchronously to a smaller speculative model S. When S predicts a tool call, the tool is launched immediately and its result stored in a future-keyed cache. When M eventually returns a tool call, the client checks the cache: on a hit it awaits the already-running future rather than re-launching the tool. This requires no engine modifications and is implemented over the OpenAI async API with Python asyncio.

Engine-side speculative tool calling (Algorithm 4/5): A fork of vLLM is modified to accept a /cache-tool-output/{response_id} endpoint. The client submits speculated tool results to the engine before M finishes generating. Inside the engine: - O1: Tools are pre-executed by S and results submitted to the engine's tool cache. - O2: At tool-call-start tokens, the engine injects the cached tool-call argument tokens as draft tokens and validates them via speculative sampling, potentially cutting decode steps from tᵢ to 1. - O3: At tool-call-end, if the full key matches, the tool output is appended in-place and the KV-cache updated, keeping the sequence resident and eliminating eviction/re-prefill overhead entirely.

Speculation models used: xLAM-2-{1B, 3B, 8B} (Llama fine-tuned on tool-calling datasets, ~80% accuracy for 8B). Workload: BFCL benchmark prompts run as 32 concurrent async agents against gpt-oss-120b on 4×A100.

Key Contributions¶

Two speculative tool-calling algorithms: client-side (zero engine changes) and engine-side (vLLM fork with tool cache)
Theoretical performance models for both variants with closed-form speedup bounds
Proof that client-side speedup is strictly < 2×, maximized when tool latency T ≈ main-model generation time G
A proposed POST /cache-tool-output API specification for inference providers to natively support this optimization
Empirical validation across sweep of tool latencies, speculative model sizes, and concurrency levels

Results¶

Engine-side, 32 agents: up to +196.4 tok/sec throughput over vanilla vLLM (from ~900 to ~1,100 tok/sec, Figure 1)
Client-side, 32 gpt-oss-120b agents, tool latency ~2.0–2.5s: up to 21% per-agent time saved (Figure 6); gains start appearing even at 0.5s average tool latency (~6% saved)
Client-side with commercial APIs (gpt-5 + gpt-5-nano, 1 speculation): ~10% time savings at ~1/10 the cost of a main API call; 9 speculations yields marginally more savings at ~25–30% of main call cost
xLAM-2-8B achieves ~80% tool-prediction hit rate; xLAM-2-1B with 9× samples approaches xLAM-2-8B performance
Speedup upper bound proven at < 2× for client-side; engine-side breaks this ceiling by also eliminating prefill overheads

Limitations¶

Speculation is restricted to stateless, cheap tools; stateful tools require rollback mechanisms (out of scope)
Client-side maximum speedup is theoretically bounded at < 2×; engine-side offers more headroom but requires modifying the serving stack
Wasted compute and cost from incorrect speculative model calls; cost scales with λ (number of speculative samples)
Engine-side modifications require provider adoption of the proposed /cache-tool-output API; not available in stock vLLM or commercial endpoints today
Experiments use pre-computed tool outputs with manually configured latencies (controlled simulation, not fully end-to-end live tools)
Evaluation limited to a single hardware configuration (4×A100) and one primary agent model (gpt-oss-120b)

Relevance to Harnesses / Meta-Harnesses¶

This work is directly relevant to any harness that orchestrates multiple LLM agent calls with tool-augmented execution pipelines. The proposed engine-side /cache-tool-output API represents a systems-level primitive that meta-harnesses could exploit by pre-running likely tools (e.g., file reads, searches) before the main model requests them—turning the harness's knowledge of likely next steps into inference efficiency gains. The client-side algorithm is immediately deployable in any harness using the OpenAI-compatible async API without infrastructure changes, making it a drop-in optimization for multi-agent harness orchestrators. The theoretical analysis (speedup peaks when T ≈ G) provides actionable guidance for harness designers deciding when speculation is worth the added complexity and speculative API cost.