EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EfficientVLA is a training-free inference acceleration framework for diffusion-based Vision-Language-Action (VLA) models that simultaneously targets three distinct redundancies: LLM layer depth, visual token count, and temporal redundancy in the diffusion action head. Applied to CogACT, it achieves 1.93× speedup and reduces FLOPs to 28.9% of baseline with only a 0.6% success rate drop on the SIMPLER robotic manipulation benchmark.

Problem

Diffusion-based VLA models (e.g., CogACT, π0) face prohibitive computational and memory overhead at inference, blocking deployment on resource-constrained robotic hardware. Existing acceleration approaches address isolated bottlenecks—visual token pruning alone hits a memory-bound wall from the LLM; LLM layer pruning alone (e.g., Mole-VLA) requires retraining; token caching (VLA-Cache) yields only 1.38× speedup because it ignores LLM memory and action head compute. No prior work holistically targets all three pipeline components in a training-free manner.

Method

EfficientVLA applies three synergistic, training-free strategies:

1. LLM Layer Pruning: Each transformer layer's importance score \(I^{(\ell)}\) is defined as 1 minus the average cosine similarity between its input and output hidden states across a calibration dataset. Layers with the lowest scores (minimal representational transformation) are removed non-contiguously. Additionally, 25% MLP weight sparsity (via PruneNet) is applied to remaining layers.

2. Task-Relevance + Diversity Visual Token Pruning: From \(N_\text{total}\) visual tokens, a compact set of \(K_\text{final}\) is selected in three stages: (a) \(K_\text{key}\) tokens with highest cross-attention scores toward language instruction tokens are unconditionally retained; (b) \(K_\text{task} = \lfloor \alpha \cdot K_\text{aug} \rfloor\) additional tokens are selected by task relevance score; (c) \(K_\text{div}\) tokens are selected to maximize cosine distance from the already-selected key set, ensuring feature diversity. Empirically: \(K_\text{key}=4\), \(\alpha=50\%\), starting from the 2nd transformer layer.

3. Static N-Step Diffusion Caching: Intermediate attention and MLP outputs (\(h_t^\text{attn}\), \(h_t^\text{mlp}\)) in DiT blocks are computed once every \(N\) denoising timesteps and cached; intervening steps reuse cached values, bypassing the self-attention and MLP computations entirely. Cache interval \(N=5\) is used in main experiments.

Key Contributions

• Systematic redundancy analysis across all three VLA components (LLM depth, visual tokens, diffusion temporal) identifying why isolated optimizations underperform.
• Training-free, non-contiguous LLM layer pruning guided by cosine-similarity importance scoring—no gradient updates required.
• Task-aware visual token selection that explicitly balances task-criticality (cross-attention to language) with feature diversity (cosine distance), outperforming random and attention-only baselines.
• Static N-step caching for diffusion action head to eliminate redundant self-attention/MLP recomputation across denoising steps.
• Holistic framework demonstration: 71.1% FLOPs reduction, 1.93× speedup, −0.6% average success rate on SIMPLER; scales to CogACT-Large with 2.0× speedup.

Results

• Overall (CogACT, L=22, T=56 vs. CogACT baseline on SIMPLER Visual Matching): 74.2% vs. 74.8% average success rate; FLOPs reduced to 28.9%; 1.93× speedup; parameter count 4.86B vs. 7.63B.
• vs. VLA-Cache: EfficientVLA achieves 1.93× vs. 1.38× speedup; 28.9% vs. 80.1% relative FLOPs.
• vs. FastV (T=56 only): FastV yields only 1.21× speedup (memory-bound by LLM); EfficientVLA reaches 1.93×.
• Random token dropping to 112 tokens: collapses average success to 20.9%; EfficientVLA at T=56 retains 74.2%, confirming guided selection is essential.
• Paradoxical improvement: Pruning 36% of parameters (L=22, T=112) raises PickCan success from 91.3% → 94.0%, indicating parameter redundancy exceeds task requirements.
• Scalability: On CogACT-Large (308M action params), 2.0× speedup with 76.1% vs. 76.7% success rate.
• Ablation: Visual-token-only pruning gives 1.23×; layer+MLP compression only gives 1.43×; all three combined yields 1.93×.

Limitations

• Evaluated solely on CogACT in the SIMPLER simulation environment; no real-robot experiments reported in the main paper.
• Layer importance calibration requires a representative sample of VLA training data; not entirely zero-data.
• Static cache interval \(N\) is a fixed hyperparameter; dynamic scheduling (adapting \(N\) per timestep based on feature drift) is not explored.
• Visual token pruning ratio and cache interval interact with LLM memory-bound vs. compute-bound regimes in a non-trivial way; Table 4 shows saturation effects that complicate principled joint tuning.
• MLP weight sparsity (PruneNet) is bundled into the reported numbers but is a separate compression axis not systematically ablated against the other three components.

Relevance to Vision-Language Models

EfficientVLA directly addresses a key deployment bottleneck for VLMs embedded in action-generation pipelines: the LLM backbone (Llama2-7B) dominates both memory and latency, and standard VLM acceleration techniques (visual token pruning, KV caching) are insufficient in isolation when coupled with a diffusion decoder. The task-relevance + diversity token selection strategy is a transferable contribution applicable to any VLM where cross-attention scores to text tokens can be used to prioritize visual inputs. The layer-importance metric based on inter-layer cosine similarity of hidden states connects to a broader line of structured pruning work for LLMs, now contextualized in the embodied AI setting. For researchers tracking VLMs, this paper establishes that holistic, multi-axis compression is necessary once a VLM is embedded in a multi-component inference pipeline, as bottlenecks shift dynamically across modules.

🔗 Related in this collection

• VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Caching