A Policy-Driven Runtime Layer for Agentic LLM Serving

🕒 Published (v1): 2026-05-26 22:38 UTC · Source: Arxiv · link

Why this paper was selected

Policy-driven runtime layer separating agent framework identity from serving infrastructure

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper argues that multi-agent LLM serving stacks have a structural gap — the agent framework knows agent identity/dispatch but not engine events, while the serving engine sees all events but knows nothing about agents — and proposes inserting a third "agent runtime layer" between them. The layer exposes four primitives (observe, score, predict, act) into which any cross-cutting agent-aware policy can plug. A KV-cache instantiation called CacheSage, which learns a per-workload agent transition matrix online, demonstrates +13 to +37 pp cache hit-rate gains over unmodified vLLM.

Problem

Production multi-agent LLM systems run on a two-tier stack (framework + serving engine) that cannot cleanly host cross-cutting policies requiring both agent-level metadata and engine-level events. Policies like KV eviction, batch shaping, speculative agent execution, and fairness each currently receive ad-hoc, per-policy patches into whichever neighbor is more accessible, leading to brittle, non-composable one-off fixes. Reactive prefix caches also fail cross-session reuse: the recurring system+tools anchor (\(\varphi \in [0.34, 0.52]\) of first-turn tokens) is dropped before the next session arrives because the eviction logic has no notion of agent identity.

Method

Architectural proposal: Insert an agent runtime layer between the framework and the engine. It owns agent-identity-tagged metadata and exposes four typed primitives to hosted policies:

• observe(event) — ingests policy-relevant events from either neighbor, keyed by agent identity
• score(item) → per-item priority for a neighbor's ranking decision (e.g., eviction order)
• predict(horizon) → probabilistic forecast of future agent activity (next agent, next tool, etc.)
• act(side_effect) — issues off-critical-path mutations to a neighbor (prefetch, hedge, throttle)

CacheSage (KV-cache instantiation): Three components realize the four primitives:

1. Transition Learner (observe): maintains sliding-window pairwise counts \(n(a', a)\) to estimate the first-order Markov transition matrix \(\hat{P}(b|a) = n(a,b)/n(a)\) in \(O(1)\); state \(\leq 20\) KB/workload. Agent identity is a content hash of the first 4 block hashes after the chat-template prefix.

2. Survival-Probability Scorer (score): constructs a thresholded DAG \(G = (\mathcal{A}, E)\) with edge \((a,b) \in E\) iff \(\hat{P}(b|a) \geq \tau\) (\(\tau = 0.01\)), runs BFS from the current agent \(a_0\) to obtain hop count \(E[a]\), and computes: $\(\tilde{p}_{\text{surv}}(a) = 1 - \min(E[a], E_{\max})/E_{\max}\)$ Block eviction score is \(\text{score}(b) = w_{\text{pred}} \cdot \tilde{p}_{\text{surv}}(a_b) + \rho_b\) where \(\rho_b\) is the engine's existing recency clock. Falls back to LRU when no structure is learned.

3. Cross-Session Prefetch Hook (predict+act): between scheduler steps, queries \(a^* = \arg\max_b \hat{P}(b|a_t)\) and issues a lightweight warmup request (one-character user message, one generated token) to populate \(a^*\)'s identity blocks in cache without injecting session-specific content.

The transition matrix is learned online because workload routing varies significantly across task sources (27–47 pp swings on individual edges for the same framework fed different inputs).

Key Contributions

• A formal architectural argument for a three-tier serving stack and a four-primitive interface (observe, score, predict, act) that any agent-aware policy can implement
• A taxonomy of nine cross-cutting policies (Table 1) — spanning KV caching, tool-result memoization, batch shaping, disaggregated P/D placement, speculative execution, adapter/drafter selection, fairness, safety, and failure hedging — all mapped onto the same four primitives
• CacheSage: a concrete instantiation for KV caching using online Markov transition learning, survival-probability eviction scoring, and between-step identity-block prefetch
• Empirical characterization of five real multi-agent workloads showing \(\varphi \in [0.34, 0.52]\) cross-session reusable surface and conditional entropy reduction \(R \in [0.40, 0.48]\), justifying the first-order Markov assumption

Results

Evaluated on NVIDIA H100 80 GB with Llama-3.1-8B-Instruct, five workloads (MMLU, MT-Bench, GAIA, GSM8K, 12-agent Synthetic), baselines vLLM and Continuum:

• Cache hit rate: CacheSage wins 5/5 workloads; +13 to +37 pp over vLLM; Continuum degrades hit rate by 9–23 pp on three of four real-task workloads due to TTL mismatch
• Per-turn TTFT / end-to-end latency: 6% to 26% reduction vs. vLLM
• Throughput: 6% to 14% higher turns/s vs. vLLM
• Runtime overhead: observe ~1 µs/block-touch; BFS scorer in single-digit µs; coordinator state ≤ 25 KB at 24 agents

Limitations

• CacheSage is the only policy validated in depth; the other eight rows of Table 1 remain open challenges with no implementation or results
• Experiments use a single model (Llama-3.1-8B-Instruct) and one framework (AutoGen SelectorGroupChat); generalization to other model families and dispatch strategies is not demonstrated
• Full microbenchmark accounting (memory, latency sensitivity to cache budget and concurrency, runtime-disabled ablations) is deferred to an extended version not yet available
• The Markov assumption (first-order) may degrade for workflows with longer-range dependencies not captured by \(R \in [0.40, 0.48]\)
• Prefetch warmup requests consume GPU compute and memory bandwidth; no cost analysis of prefetch overhead under high concurrency is provided

Relevance to Harnesses / Meta-Harnesses

The agent runtime layer is itself a meta-harness primitive: a framework-agnostic, engine-portable substrate that intercepts and coordinates cross-cutting behaviors across multiple agents without modifying either the orchestration framework or the serving engine. The four-primitive interface (observe/score/predict/act) is a generalizable harness API — analogous to how meta-harnesses intercept tool calls, route sub-agents, and enforce policies without coupling to any specific agent implementation. The nine-policy taxonomy in Table 1 maps directly to concerns meta-harnesses must handle (scheduling, fairness, memoization, safety guardrails, retry/hedging), suggesting this layer architecture could inform how production-grade multi-agent harnesses instrument and govern their inner loops. The online Markov transition learning approach is particularly relevant for meta-harnesses that need to adapt routing and resource allocation policies dynamically based on observed agent behavior rather than statically declared graphs.