Skip to content

KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

KVCOMM is a training-free framework that enables KV-cache reuse across agents in multi-agent LLM systems by estimating and correcting the context-dependent offset that diverging per-agent prefixes introduce. It maintains an online anchor pool of previously observed cache deviations and interpolates offsets at runtime, achieving up to 7.82× TTFT speedup with <2.5% accuracy drop on standard benchmarks.

Problem

In multi-agent LLM pipelines, agents frequently share large context segments (retrieved passages, upstream agent outputs), but existing KV-cache reuse techniques require identical prefixes. Each agent's role-specific system prompt shifts token positions via RoPE and alters attention context, producing an "offset-variance" problem: the same shared text yields meaningfully different KV vectors under different prefix contexts. Naive cache reuse either causes large accuracy degradation or forces full recomputation, and no prior method handles this dynamically without model retraining or profiling.

Method

KVCOMM models a multi-agent system as a directed graph where each node is an identical RoPE-based LLM. It decomposes each agent's prompt into fixed prefix segments and runtime placeholder segments.

Core mechanism: 1. Positional alignment: Keys are de-rotated (R⁻¹) from their source position before comparison and re-rotated to the target position, isolating contextual deviation from positional shift. 2. Anchor pool: For each placeholder type, a pool stores base KV-caches (computed without external context) and per-agent offset vectors (base → in-context cache) observed in prior requests. An entry is {placeholder_name: base_KV, agent_id_ph: placeholder_offset, agent_id_pf: prefix_offset}. 3. Anchor prediction: A new input's shareability is decided by length compatibility and embedding-entropy criterion (Eq. 5); if no sufficiently close anchor exists, full prefilling runs and the result becomes a new anchor. 4. Offset approximation: At inference, the framework retrieves the nearest anchors by embedding similarity and computes weighted interpolation of stored offsets (Eqs. 6–7) to shift both placeholder and neighboring prefix KV caches into the agent's context. 5. Online update: After decoding, newly produced caches are evaluated for anchor addition; least-frequently-accessed anchors are evicted when the pool hits capacity V.

Theoretical grounding: Propositions 1–2 bound KV distance and offset deviation by the token embedding gap (via Lipschitz continuity of Attn/FFN), justifying that embedding-similar tokens have transferable cache offsets across prefix contexts.

Key Contributions

  • Formal identification of the multi-context redundancy and offset-variance problem as the core blocker for KV-cache sharing in multi-agent inference.
  • KVCOMM: the first training-free, prompt-adaptive cache-sharing framework for multi-agent LLMs; no model modification, no offline profiling, no fine-tuned connectors.
  • Theoretical analysis (Propositions 1–2) bounding KV deviation by token embedding distance, providing principled motivation for anchor-based interpolation.
  • Anchor pool design with online update and LFU-based eviction that adapts to shifting request distributions.

Results

  • TTFT speedup (5-agent, Llama-3.1-8B, H100, 1K input / 512 prefix / 512 output): up to 7.82× for Agent 5; ~6.7× average over 3-agent setting.
  • TTFT range (3-agent, varying context): 2.24× (64-token prefix, 128-token output) to 6.72× (1K prefix, 1K output).
  • GSM8K (math, 4 agents): KVCOMM 80.6% vs. original 82.1% (−1.5%); CacheBlend drops to 65.1% (−17%).
  • HumanEval (coding, 4 agents): KVCOMM 83.2% Pass@1 vs. original 84.5%; CacheBlend 30.4% (−54%).
  • MMLU (RAG, 4 agents): KVCOMM 68.0% vs. original 68.0%; CacheBlend 65.4%.
  • Reuse rate: 70–87.6% across workloads (vs. CacheBlend's fixed 80%); degrades gracefully with more agents.
  • Hyperparameter sensitivity: γ=0.3, V=20 balances accuracy and reuse; raising γ to 0.9 reaches 98.2% reuse at −3.3% accuracy cost on GSM8K.

Limitations

  • Speedup for the first agent is minimal (1.11×) because it lacks upstream cached context to reuse.
  • Performance is sensitive to request ordering: descending-length order slightly degrades accuracy (66.0% vs. 68.0% baseline on MMLU) due to anchor prediction criterion design.
  • Requires all agents to run identical model checkpoints (same RoPE-based decoder), precluding heterogeneous agent architectures.
  • Anchor pool introduces GPU memory overhead; pool size V must be tuned per workload.
  • Evaluated only on a single H100 GPU; distributed multi-GPU or multi-node scenarios are not addressed.
  • Minor accuracy drops accumulate as agent count grows; not analyzed beyond 5 agents.

Relevance to Agentic AI / LLM Agents

Prefilling latency is a fundamental bottleneck in multi-agent LLM pipelines: as agent graphs scale, reprocessing shared context quadratically multiplies compute. KVCOMM directly attacks this infrastructure-level inefficiency, cutting TTFT by up to 7.8× without retraining or quality loss—a practical enabler for real-time, many-agent collaboration. The anchor-pool mechanism is also an interesting architectural primitive for stateful agent communication, where cross-agent KV sharing functions as a lightweight shared memory. For researchers building or studying multi-agent systems (RAG pipelines, code-generation collectives, reasoning chains), KVCOMM represents a drop-in efficiency layer that decouples system scale from inference cost.