Skip to content

Action-aware Dynamic Pruning for Efficient Vision-Language-Action Manipulation

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Action-aware dynamic pruning enables efficient VLA inference; ICLR 2026

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Vision-Language-Action (VLA) models for robotic manipulation waste computation by treating all visual tokens uniformly across manipulation phases. This paper introduces Action-aware Dynamic Pruning (ADP), which gates visual token pruning on the robot's end-effector trajectory magnitude — pruning aggressively during coarse motion and preserving full vision during fine-grained operations. On LIBERO benchmarks, ADP achieves up to 1.35× speedup on OpenVLA-OFT with less than 1% success rate degradation.

Problem

Existing VLA efficiency methods (e.g., FastV, VLA-Cache, EfficientVLA) reduce visual token counts via static or attention-heuristic rules, ignoring that visual redundancy is not uniform across manipulation stages. During coarse phases (e.g., object transport), most visual tokens are redundant; during fine-grained phases (e.g., grasping, alignment), full visual context is critical. Treating all timesteps identically leads to either insufficient savings or accuracy loss from over-pruning.

Method

ADP combines two complementary mechanisms applied before the LLM backbone:

Text-driven Anticipatory Pruning: At layer \(l=0\), cross-modal attention scores between text queries and visual keys are computed as: $\(A^{(l)} = \frac{Q^{(l)}(K^{(l)})^\top}{\sqrt{d}} \in \mathbb{R}^{N_h \times L^{\text{txt}} \times L^{\text{vis}}}\)$ Per-token importance \(\Phi^{(l)}(v)\) is obtained by averaging across heads and text positions; the top-\(k = \lfloor\rho \cdot L^{\text{vis}}\rfloor\) tokens are retained. In multi-view settings, per-view quotas are weighted by \(\alpha \in \mathbb{R}^C\).

Action-aware Dynamic Gating: Each decoded action chunk defines a temporal window; end-effector translation is composed via body-frame SE(3) integration (Definition 4.1) to yield a scalar windowed trajectory distance \(\delta_i = \sum_{t=b_i}^{e_i-1}\|p_{t+1}-p_t\|_2\). A binary gate \(s_{i+1} \in \{0,1\}\) switches pruning on when \(\delta_i\) exceeds a running mean (global rule) or the local max/min extrema of the past \(\tau\) windows (adjacent-extrema rule), and switches it off otherwise. The expected episode cost is: $\(\mathbb{E}[F_{\text{episode}}] = T\bigl[\gamma F_{\text{ADP}} + (1-\gamma)F_{\text{base}}\bigr]\)$ where \(\gamma\) is the fraction of forwards in the pruned state. A cold-start (first two windows use full vision) and a forced full-vision reset after three consecutive pruned windows limit error accumulation.

Key Contributions

  • Empirical identification that visual token redundancy in VLAs correlates with end-effector motion magnitude, varying systematically across manipulation phases.
  • Text-driven anticipatory pruning using early-layer (layer 0) cross-modal attention for token ranking, with per-view quota allocation in multi-camera settings.
  • Action-aware binary gating conditioned on windowed trajectory distance, with two variants (running-mean and adjacent-extrema); plug-and-play, training-free.
  • Theoretical FLOPs analysis showing uniform savings across all \(H\) LLM layers when pruning is applied at the embedding stage.
  • Evaluation on LIBERO (4 suites, simulation) and 4 real-world Jaco2 tasks.

Results

  • LIBERO (simulation, OpenVLA-OFT 7B baseline: 97.1% avg SR, 7.91 TFLOPs):
  • VLA-ADP (50% keep ratio): 96.3% avg SR, 6.43 TFLOPs, 1.23× speedup
  • VLA-ADP (30% keep ratio): 94.4% avg SR, 5.85 TFLOPs, 1.35× speedup
  • Spatial task at 50% keep: 99.4% SR (vs. 98.6% baseline)
  • FastV on OFT: 86.8% avg SR (−10.3 pp vs. baseline) at comparable compute reduction
  • Random Dropping (50%): 1.29× speedup but 73.0% Object / 76.2% Long SR (severe degradation)
  • Real world (Jaco2, 4 tasks):
  • OpenVLA-OFT baseline: 85.8% avg SR, 76.9 ms latency
  • VLA-ADP: 88.3% avg SR, 51.8 ms latency, 1.49× speedup
  • Ablation:
  • Removing action-aware dynamics (w/o D): −2.85 pp avg SR with only −0.20 TFLOPs savings
  • Periodic switching (w/o D + PS): 89.9% avg SR, −16.6 pp on Object vs. full ADP
  • Layer 0 importance scoring dominates: 96.3% SR / 6.43 FLOPs vs. layer 4 (95.8% / 6.89 FLOPs)

Limitations

  • The gating signal depends on end-effector trajectory from decoded action chunks; early-episode windows lack history, requiring a cold-start heuristic that always uses full vision for the first two windows.
  • The forced full-vision reset after three consecutive pruned windows is a hand-tuned safeguard against error accumulation, not a principled solution.
  • Multi-view token quota weighting \(\alpha\) (e.g., 4:6 main/wrist split) is set heuristically rather than learned.
  • Evaluated on a single VLA backbone family (OpenVLA / OpenVLA-OFT); generalization to diffusion-based VLAs (e.g., \(\pi_0\)) is not demonstrated.
  • Real-world experiments cover only 4 tasks on a single robot platform (Jaco2).

Relevance to Vision-Language Models

ADP is directly relevant to the growing line of work on efficient VLMs: it extends token-pruning techniques (originally studied for image-language models like FastV) to the temporally sequential, action-conditioned setting of VLA inference. The key insight — that token redundancy is not static but governed by the model's own output trajectory — generalizes the principle that pruning should be conditioned on dynamic context rather than fixed attention patterns. For VLM researchers, the text-driven cross-modal ranking at layer 0 (before deep fusion) offers a computationally cheap alternative to multi-layer attention aggregation, and the training-free, plug-and-play design makes ADP readily applicable to other large multimodal architectures. The work also highlights multi-view token importance imbalance as an underexplored challenge in embodied VLMs.