Skip to content

VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VL-Cache is a KV cache compression method for VLMs that exploits the modality-specific attention sparsity unique to vision-language models. By using post-vision attention (attention from language tokens that follow visual tokens) to both allocate layer-wise cache budgets and score token importance, VL-Cache retains only 10% of the KV cache while achieving accuracy comparable to full cache. This yields up to 7.08× decoding speedup and 90% GPU memory reduction.

Problem

Existing KV cache sparsification methods (H2O, PyramidKV, StreamingLLM, ZipCache) were designed for LLMs and are modality-unaware: they conflate attention scores from visual and language query tokens, which leads to suboptimal token eviction decisions in VLMs. Additionally, all prior methods either assign equal cache budgets to every layer (ignoring per-layer sparsity variation of 70–99%) or apply a monotonically decreasing pyramid schedule that does not match the non-monotonic sparsity pattern observed across transformer layers. Both flaws are compounded by the fact that VLM visual tokens can dominate prompt length (e.g., 110 GB KV cache for LLaVA-1.6-34B with 5 images at 2K tokens each, batch size 4).

Method

VL-Cache introduces two coordinated components, both built on post-vision attention — the sub-matrix of attention scores computed from only the language tokens that appear after the visual tokens in the prompt (indices Q_{m-τ:m} where τ is the post-vision language token count):

  1. Sparsity-aware layer-adaptive cache budget allocation: During prefill, per-layer sparsity γ⁽ˡ⁾ is computed from the post-vision attention sub-matrix using a 1% relative threshold filter. Each layer's cache budget β⁽ˡ⁾ is then set proportional to its non-sparse ratio (1−γ⁽ˡ⁾), normalized across all layers and scaled by the global budget α (e.g., α=0.10 for 10%). This is prompt-adaptive, unlike PyramidKV's fixed schedule, and runs in O(τm) rather than O(m²) since τ≪m.

  2. Modality-aware token scoring (post-vision attention policy): Tokens are ranked by their accumulated attention scores computed only over the post-vision query rows. This avoids the signal dilution caused by visual-to-visual attention (which is nearly uniform) that biases accumulated or normalized full-attention scoring. Top-k tokens per layer are retained according to the per-layer budget β⁽ˡ⁾. A custom Triton kernel handles the sparse attention forward pass during decoding.

Key Contributions

  • First empirical characterization of attention sparsity patterns specific to VLMs, showing a clear modality boundary along the query dimension absent in LLMs.
  • Layer-adaptive, prompt-specific KV cache budget allocation driven by per-layer post-vision sparsity ratios, overcoming the limitations of uniform and pyramid allocations.
  • Post-vision attention scoring policy that uses a dynamic, prompt-dependent window (length = τ) instead of a fixed window, yielding higher CacheHitRate across all layers compared to accumulated, normalized, and sliding-window policies.
  • Triton-based sparse attention kernel enabling practical decoding speedups at reduced KV cache sizes.

Results

  • Accuracy (10% KV cache budget): Achieves ≥98% of full-cache performance on Coco-Caption (CIDEr), DocVQA (ANLS), and MathVista (ACC) for both LLaVA-Mistral-7B and LLaVA-1.6-34B; outperforms H2O, StreamingLLM, ZipCache, and PyramidKV at all budgets ≤40%.
  • DocVQA (LLaVA-1.6-34B): VL-Cache 84 vs. H2O 75, PyramidKV 83, StreamingLLM 34, ZipCache 74 (full cache: 85)
  • Coco-Caption (LLaVA-1.6-34B): VL-Cache 137.35 vs. H2O 58.14, PyramidKV 116.91, StreamingLLM 8.23, ZipCache 131.61 (full cache: 135.07)
  • Decoding speedup: Up to 7.08× (batch=1, prompt=128K tokens, 10% cache); 5.23× at batch=64, prompt=2K.
  • End-to-end speedup: Up to 2.33× (batch=64, prompt=2K, 100 output tokens).
  • Memory: 90% reduction in KV cache GPU memory footprint.
  • Prefill overhead: Only 1–4% added latency for sparsity statistics computation.
  • Concurrency: Up to 10× higher server concurrency when KV cache size is the bottleneck.

Limitations

  • Prefill latency is essentially unchanged; end-to-end speedup is bounded by prefill time (e.g., at 128K prompt length the 7.08× decoding speedup reduces to 1.66× end-to-end because prefill accounts for 53% of total latency).
  • Only prefill KV cache is compressed; decoded tokens accumulate in the cache without periodic eviction, limiting gains for tasks with very long outputs.
  • Experiments are confined to the LLaVA family; generalization to other VLM architectures (e.g., cross-attention-based models) is not demonstrated.
  • Maximum batch size in the benchmark is still bounded by peak prefill memory, not KV cache size; continuous batching / chunked prefill needed to fully exploit the memory savings for throughput.
  • Video models with extreme token counts are identified as future work, not yet evaluated.

Relevance to Vision-Language Models

VL-Cache directly addresses the inference efficiency bottleneck that arises as VLMs process longer visual contexts (high-resolution images, multi-frame video), a central challenge for practical VLM deployment. The core insight — that the modality boundary in VLM attention makes LLM-derived compression heuristics systematically wrong — motivates a new design principle for VLM-specific systems work. For researchers tracking VLMs, this paper complements token-reduction methods (FastV, HiRED) with a KV-level approach that is architecture-agnostic and can be stacked with quantization. The post-vision attention concept also has implications for understanding which parts of a visual prompt actually influence generation, relevant to interpretability and grounding research.