KVComm: Enabling Efficient LLM Communication through Selective KV Sharing¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

KV-sharing cuts inter-LLM communication cost in multi-agent pipeline inference

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

KVComm is a communication framework for multi-agent LLM systems that replaces natural-language or hidden-state messaging with selective sharing of key-value (KV) pairs across transformer layers. A layer selection strategy combining per-layer attention importance scores with a Gaussian prior identifies the most informative subset of layers to transmit. With only 30–70% of layers' KV pairs shared, KVComm matches or exceeds the "Skyline" upper bound of directly concatenating both inputs, while cutting computation by 2.5–6×.

Problem¶

Existing inter-LLM communication protocols are inadequate: natural language debate (NLD) requires expensive multi-step decoding and loses information through token sampling; hidden-state methods (CIPHER, AC) either still need decoding steps or suffer from information concentration bias — the last token's hidden state dominates later layers, so replacing or averaging it corrupts the receiver's ($M_r$) own representation. No prior method enables non-contiguous, data-driven selection of which layers to share.

Method¶

Given a sender $M_s$ holding context $C$ and a receiver $M_r$ holding query $Q$:

Prefill-only sender: $M_s$ runs a single forward pass over $C$, generating KV pairs $\{(k_l^s, v_l^s)\}_{l=1}^{L}$ — no decoding required.
KV injection: Selected layers' KV pairs are concatenated into $M_r$'s attention at the matching layer indices: $k_l^r \leftarrow [k_{l_i}^s;\, k_l^r]$, $v_l^r \leftarrow [v_{l_i}^s;\, v_l^r]$. This leaves $M_r$'s hidden states untouched.
Layer selection: The attention importance score for layer $l$ is $$\hat{S}_a^l = \frac{1}{H|Q|}\sum_{h=1}^{H}\sum_{q=1}^{|Q|}\sum_{c=1}^{|C|} a_{h,q,c}^l$$ normalized to $[0,1]$. A Gaussian prior $P^l = \exp\!\left(-\frac{(l-\mu)^2}{2\sigma^2}\right)$ centered on intermediate layers is combined as $S^l = \alpha S_a^l + (1-\alpha)P^l$. The top-$M$ scoring layers form the transmitted subset $S$.
Calibration: Layer selection runs once on a single calibration sample per model pair, then is fixed for the entire test set.

The two design hypotheses — (H1) intermediate layers carry the most transferable semantic knowledge; (H2) layers with stronger attention concentration are more communication-valuable — are both validated experimentally.

Key Contributions¶

Empirical characterization of why hidden states fail: information concentration bias at the last token's position grows in later layers, and prepending all-layer hidden states is either computation-equivalent to full concatenation (if taken from early layers) or degrades performance (if injected late).
KVComm: first protocol enabling non-contiguous KV layer selection for inter-LLM communication, driven by attention importance scores and a Gaussian depth prior.
Demonstration that a calibration set of a single sample suffices to generalize layer selection to the full test set.
Reduction of $M_s$ computation by 2.5–6× vs. Skyline and up to 3× less data transmitted vs. full KV sharing, while matching Skyline performance at 70% layer selection ratio.

Results¶

vs. Skyline (upper bound — full input concatenation): KVComm at 70% layers achieves comparable F1/ROUGE across 8 datasets and 9 model pairs; at 30% layers it still outperforms all communication baselines (NLD, CIPHER, AC variants) on most datasets.
Selected numbers (Table 1, Llama-3.1-8B pair): HotpotQA F1 — Baseline 0.23, NLD 0.43, AC(mean) 0.25, KVComm(0.7) 0.65, Skyline 0.73; MuSiQuest F1 — Baseline 0.02, NLD 0.18, KVComm(0.7) 0.36, Skyline 0.51.
vs. random layer selection (Table 2): KVComm(0.3) significantly outperforms Random(0.3); gap narrows at 0.7 ratio as expected.
Non-contiguous vs. contiguous chunks: KVComm matches or exceeds the best contiguous-chunk configuration without requiring hyperparameter search over layer ranges.
KVComm occasionally surpasses Skyline (e.g., Countries, Tipsheets on Falcon pair), attributed to complementary model capabilities and a regularization effect from selective attention.
Computation savings over NLD scale as $O(L(T_s+T_r+|Q|)^2 d)$; savings over Skyline scale as $O(|C|d(L(2|Q|+T_r)-M(|Q|+T_r)))$.

Limitations¶

Restricted to model pairs that share the same base architecture (two instances of the same model, or two fine-tuned variants); cross-family or heterogeneous-architecture communication is not addressed.
Layer indices are matched 1-to-1 between $M_s$ and $M_r$, requiring equal depth.
Performance gains on mathematical reasoning (TMATH) are marginal, suggesting KV-pair sharing is less effective when the task relies on internal parametric knowledge rather than contextual information.
The Gaussian prior center $\mu$ and bandwidth $\sigma$, as well as $\alpha$, are hyperparameters requiring tuning per model family.
Only two-agent (one sender, one receiver) communication is studied; scaling to larger agent networks is left to future work.

Relevance to Agentic AI / LLM Agents¶

KVComm directly addresses a fundamental bottleneck in multi-agent LLM pipelines: the cost and fidelity of inter-agent information transfer. By bypassing natural-language serialization — the de facto standard in frameworks like AutoGen, CAMEL, and ChatDev — it offers a path toward low-latency, high-bandwidth "neural messaging" between specialized agents. The finding that 30% of KV layers can be transmitted while preserving most task performance is practically significant for scaling multi-agent systems where communication overhead compounds across many hops. This work complements agent-architecture research (role assignment, tool use, orchestration) by providing a more efficient substrate-level communication primitive, potentially enabling denser agent collaboration at lower inference cost.