Skip to content

VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Caching

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLA-Cache is a training-free inference acceleration method for Vision-Language-Action (VLA) models that exploits temporal redundancy in robotic manipulation by selectively caching and reusing static visual token KV representations across frames. It combines pixel-level cosine similarity for static token selection, cross-modal attention filtering to protect semantically critical tokens, and a layer-adaptive reuse ratio based on attention entropy. On LIBERO and SIMPLER benchmarks it achieves up to 1.7× CUDA latency speedup with <1% drop in task success rate.

Problem

VLA models (e.g., OpenVLA, CogAct) built on large LLM decoders (LLaMA, Gemma) incur prohibitive inference latency for real-time robotic control (~50–80 ms/step). Existing VLM acceleration methods (FastV, SparseVLM, ToMe) operate within a single frame and ignore temporal redundancy intrinsic to closed-loop manipulation, while VLA-specific methods (TinyVLA, DeeR-VLA, QAIL) require architectural changes or retraining and do not directly attack the LLM decoder bottleneck.

Method

VLA-Cache operates at inference time without modifying model weights, in three stages:

  1. Static token selection: Each input frame is divided into N×N patches. For consecutive frames, cosine similarity is computed per patch; patches exceeding threshold τ and surviving a Top-k filter form the candidate static pool P_static.

  2. Task-relevance filtering: Cross-modal text-to-vision attention scores are extracted from the LLM decoder layers. Vision tokens whose mean attention score (averaged across heads and selected layers) exceeds Ï„_task are designated task-relevant (P_task-relevant) and removed from the reuse pool: P_final = P_static \ P_task-relevant. These tokens are recomputed fresh each step regardless of visual similarity.

  3. Layer-adaptive caching: Attention entropy is computed per decoder layer. The cumulative entropy reduction ratio R_cum determines a per-layer reuse fraction αl = min(k·Σ R_j, 1). Deeper layers with more concentrated attention reuse a higher fraction of tokens from P_final; early layers with diffuse attention reuse fewer. KV entries for reused tokens are carried over; remaining tokens recompute K and V via the projection matrices.

Key Contributions

  • Training-free, plug-and-play cross-frame KV caching tailored to VLA temporal structure, requiring no retraining or architectural change.
  • Attention-guided task-relevance filter that recovers 8.4 pp success rate lost by naive static reuse (74.2% → 82.6% on LIBERO-Spatial with OpenVLA).
  • Layer-adaptive reuse ratio driven by per-layer attention entropy, adding a further +1.2 pp (82.6% → 83.8%) with minimal latency overhead.
  • Demonstrated compatibility with heterogeneous action heads (token-based OpenVLA; diffusion-based CogAct) and high-frequency chunked architectures (OpenVLA-OFT).
  • Real-robot validation on Kinova Jaco2 with robustness to dynamic backgrounds.

Results

  • LIBERO (OpenVLA): 74.7% avg success rate vs. 75.0% baseline; FLOPs reduced 27.3% (1.864T → 1.355T); CUDA latency 31.83 ms vs. 51.91 ms (1.63×); control frequency 4.59 Hz vs. 4.23 Hz. FastV achieves only 73.3% avg; SparseVLM drops to 64.7%.
  • LIBERO (OpenVLA-OFT): 97.4% avg vs. 96.8% baseline; FLOPs 3.097T vs. 4.013T; latency 62.59 ms vs. 79.05 ms; control frequency boosted ~14 Hz (65.1 → 79.0 Hz).
  • SIMPLER (CogAct): 74.4%/62.3% avg (Matching/Aggregation) vs. 74.8%/61.3% baseline; ~19% fewer FLOPs; 1.37× latency reduction; control frequency ~14.5 Hz vs. 12.4 Hz.
  • Real robot (Kinova Jaco2): 84.6% avg vs. 82.1% baseline across four tasks; 28.2% fewer FLOPs; latency 51.85 ms vs. 64.16 ms; control frequency 4.21 Hz vs. 4.02 Hz.
  • Dynamic background (PickPot): Baseline drops 95%→80%; VLA-Cache holds 90% while reducing FLOPs by 42% and latency by 35%.

Limitations

  • Limitations are relegated to Appendix A (not included in the provided text), so full scope is not assessable from the main paper.
  • Hyperparameters (Ï„, Ï„_task, k) must be tuned per model/environment; sensitivity analysis is partial (ablation covers token counts but not threshold sweeps).
  • The speedup (1.7× CUDA latency, 15% control-frequency gain) is measured on an RTX 4090; gains on edge/embedded hardware are untested.
  • Evaluation is limited to tabletop pick-and-place manipulation; generalization to contact-rich or high-dexterity tasks with rapidly changing scenes is unclear.
  • One-step temporal look-back (frame t vs. t−1 only); multi-step caching or longer temporal horizons are not explored.
  • FastV and SparseVLM comparisons are applied only to OpenVLA, not to OpenVLA-OFT or CogAct.

Relevance to Vision-Language Models

VLA-Cache directly addresses the inference-efficiency bottleneck that constrains deployment of LLM-backbone VLA models, which inherit the same transformer decoder architecture as general VLMs. The task-relevance filtering mechanism—using cross-modal text-to-vision attention as a dynamic semantic signal—is a generalizable technique applicable to any VLM operating over temporal streams (e.g., video QA, embodied navigation). The layer-adaptive entropy-guided reuse strategy extends prior intra-frame token reduction work (FastV, SparseVLM) into the cross-frame temporal domain, establishing a new efficiency axis specifically for closed-loop multimodal reasoning. For researchers tracking VLMs, this paper demonstrates that domain-specific structure (temporal redundancy, short action-token outputs) enables substantially better efficiency trade-offs than generic token pruning.