ICaRus: Identical Cache Reuse for Efficient Multi-Model Inference

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; ICaRus: identical KV-cache reuse across multi-model agentic inference

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ICaRus enables multiple task-specialized LLMs to share a single KV cache by decomposing each decoder-only Transformer into a frozen "logical encoder" (KV generator) and a fine-tuned "logical decoder" (token predictor). This eliminates per-model KV cache duplication in multi-model agentic workflows, achieving up to 11.1× lower P95 latency and 3.8× higher throughput compared to conventional multi-model serving.

Problem

In multi-model agentic pipelines (planner → executor → summarizer, etc.), every specialized model must independently compute and store KV caches for identical shared prefixes. This causes KV memory to grow as \(O(M + N L_t)\) with \(N\) models, triggering cache evictions and redundant prefill recomputation that inflate latency and energy use. Cross-model prefix caching is structurally infeasible in standard architectures because each fine-tuned model produces distinct KV representations even for identical inputs.

Method

ICaRus decomposes a decoder-only Transformer \(F\) into: - Logical encoder \(E_{\text{base}}\): the frozen pretrained base model, which maps input tokens \(x_{1:i}\) to key-value pairs \(K_{1:i}, V_{1:i} = E_{\text{base}}(x_{1:i})\). - Logical decoder \(D_{\text{task}}\): a task-specific module trained to predict \(x_{i+1} = D_{\text{task}}(x_i, K_{1:i}, V_{1:i})\) using the shared KV cache.

All task variants (math, coding, instruction-following) share the same \(E_{\text{base}}\), so their KV caches are identical and can be stored once. Fine-tuning uses LoRA adapters applied only to the logical decoder, so the decoder adds lightweight parameters atop the shared base weights. During decoding, the logical encoder and decoder run in parallel by concatenating their queries along the head dimension and performing a single fused attention pass over the shared KV cache—avoiding the naive 2× latency overhead of sequential execution. Integration is done within vLLM. This reduces memory to \(O(M + L_t)\) and prefill complexity to \(O(M L_t + L_t^2)\), both independent of \(N\).

Key Contributions

• First architecture enabling full KV cache sharing across multiple independently task-specialized decoder-only Transformers (all layers, not just non-sensitive ones).
• A training procedure that explicitly conditions logical decoders on shared-encoder KV caches, ensuring robustness at inference time without post-hoc adaptation.
• Parallel logical encoder/decoder execution via head-dimension query concatenation, recovering single-model decode latency.
• Demonstrated accuracy parity (and occasional improvement) with conventional task-specific LoRA fine-tuning across math, coding, and knowledge tasks on LLaMA-3.1-8B and Qwen3-1.7B/8B/14B.
• vLLM integration and empirical serving evaluation in multi-agent ReAct and Reflexion workflows.

Results

• Latency: Up to 11.1× reduction in P95 latency vs. conventional multi-model system (8-agent workflow on HotPotQA).
• Throughput: Up to 3.8× higher throughput vs. baseline.
• Accuracy (LLaMA-3.1-8B): ICaRus vs. Multi Model — GSM8K 67.9 vs. 69.7; GSM+ 45.8 vs. 48.5; HumanEval 48.2 vs. 48.2; HumanEval+ 43.9 vs. 41.5; GPQA-Diamond 28.8 vs. 27.3.
• Accuracy (Qwen3-8B-Base): ICaRus outperforms conventional fine-tuning on GSM8K (87.3 vs. 85.4), GSM+ (67.5 vs. 66.1), HumanEval (86.6 vs. 81.7), HumanEval+ (79.9 vs. 75.6).
• Scaling: ICaRus improvement grows with model size; +2% on Qwen3-14B-Base GSM8K (88.8 vs. 85.6).
• Training loss curves of ICaRus overlap with conventional LoRA fine-tuning, indicating no optimization degradation from freezing the encoder.

Limitations

• Freezing the logical encoder means task adaptation must be expressible entirely through the decoder; tasks requiring fundamentally different token representations may be disadvantaged.
• The decode phase still performs two forward passes (encoder + decoder), adding some compute overhead even if memory traffic is parallelized; this becomes non-negligible in compute-bound (short-sequence) regimes.
• Evaluated only on LoRA-based adaptation; full-parameter fine-tuning of the decoder is discussed but not benchmarked for serving performance.
• All models in a workflow must be derived from the same base model; heterogeneous-backbone multi-model systems are not addressed.
• Accuracy evaluations use zero-shot settings; performance on few-shot or chain-of-thought prompting is not reported.

Relevance to Agentic AI / LLM Agents

Multi-model inference—where specialized agents (planner, coder, critic) collaborate within a single pipeline—is the dominant architecture in advanced agentic systems (ReAct, Reflexion, LLMCompiler, LATS), and KV cache explosion is the central serving bottleneck as these pipelines scale. ICaRus directly attacks this bottleneck with a training-time architectural fix rather than a runtime scheduling heuristic, making cross-agent prefix caching possible for the first time across model boundaries. For researchers building or deploying multi-agent systems, ICaRus's vLLM integration and demonstrated 11× latency reduction at 8-agent scale offer a practical path to efficient orchestration without sacrificing per-agent specialization. The approach also connects to the broader question of how to share representations across agents—here the frozen encoder functions as a shared world model, while each decoder captures agent-role-specific behavior.