Skip to content

AVA-VLA: Improving Vision-Language-Action models with Active Visual Attention

🕒 Published (v1): 2025-11-24 10:22 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AVA-VLA reformulates Vision-Language-Action (VLA) policy learning under a Partially Observable MDP (POMDP) framework instead of the standard MDP assumption. It introduces a recurrent state (neural belief approximation) derived from previous hidden states, and an Active Visual Attention (AVA) module that uses this recurrent state to dynamically reweight visual tokens at each timestep. The method achieves state-of-the-art on LIBERO and CALVIN and transfers to real-world dual-arm manipulation.

Problem

Existing VLA models process each visual observation independently, implicitly treating robot manipulation as a Markov Decision Process. This is mismatched to reality: robot manipulation is partially observable, prior actions alter the visual scene, and relevant objects may be occluded or only transiently visible. Without historical context, visual attention is guided solely by static language instructions and cannot suppress temporally redundant tokens or focus on regions made salient by past actions.

Method

POMDP reformulation: Policy is conditioned on both current observation \(x_t\) and a recurrent state \(r_{t-1}\) (a neural approximation to the POMDP belief \(b_{t-1}\)), instead of \(x_t\) alone.

Recurrent state: Extracted via an MLP \(B\) from the action-related hidden states of the final decoder layer at \(t{-}1\): \(r^{t-1} = B(h^{t-1}_M) \in \mathbb{R}^{L_A \times d}\). It is also used to initialize action placeholder embeddings (replacing the zero vector in OpenVLA-OFT).

AVA module: Takes current visual tokens \(z^t_I\) and \(r^{t-1}\) as input. FiLM conditions visual features on language. Cross-attention is computed with visual tokens as queries and the recurrent state as keys/values, followed by self-attention and an FFN+linear+Softmax head that produces per-token soft importance scores \(\omega^t \in \mathbb{R}^{L_I \times 2}\) (enhance/weaken logits). These scores modulate all LLM attention matrices by scaling attention weights on visual token columns, enabling history-aware visual focus throughout the backbone.

Training: Truncated BPTT with horizon \(T{=}4\); MAE action loss plus an L2 regularizer on the mean of \(\omega^t\) to prevent degenerate attention spreading. Built on OpenVLA-OFT (DINOv2+SigLIP vision encoder, LLaMA2-7B backbone).

Key Contributions

  • First VLA framework to explicitly address the MDP/POMDP mismatch via a recurrent belief-state approximation.
  • AVA module: cross-attention-based mechanism that computes history-conditioned soft importance scores and injects them multiplicatively into every LLM self-attention layer.
  • State-based initialization: recurrent state replaces the zero action-placeholder embedding, preserving temporal context for action decoding.
  • Demonstration that importance scores enable aggressive visual token pruning (70% pruning with near-zero performance loss; 90% pruning still beats most baselines).

Results

  • LIBERO (multi-task, one policy): 98.0% avg. SR vs. OpenVLA-OFT 96.8%; best on LIBERO-Long (97.6% vs. 95.3%).
  • LIBERO (single-task, one policy per suite): 99.2% avg. SR vs. OpenVLA-OFT 97.1%; LIBERO-Long 96.2% vs. 94.5%.
  • CALVIN ABC→D: Avg. length 4.65 vs. FLOWER 4.53 (prior SOTA); 5-task chain SR 84.1% vs. FLOWER 77.8%.
  • Mobile ALOHA (real-world): Highest average success across 4 tasks (Pick & Place, Tower of Hanoi, towel folding, dexterous scooping) vs. UniVLA and OpenVLA-OFT.
  • Backbone ablation (LIBERO-Long): +1.7% over OpenVLA-OFT on OpenVLA-7B, +2.6% on LLaMA2-7B, +1.4% on Qwen2.5-0.5B.
  • Token pruning: 70% pruning ratio retains competitive performance; 90% still outperforms many full-token baselines.

Limitations

  • Recurrent state introduces sequential dependency at inference—each step requires the previous forward pass output, precluding trivial parallelism across timesteps.
  • Full backpropagation through time is computationally intractable; truncated BPTT (\(T{=}4\)) may underfit long-horizon temporal dependencies.
  • Framework is built on and evaluated relative to OpenVLA-OFT; generality to non-parallel-decoding or non-LLaMA-backbone VLAs is untested.
  • Real-world evaluation is limited to a single robot platform (dual-arm cobot magic) with small demonstration sets (30–450 per task), leaving broader sim-to-real generalization unverified.
  • Token pruning analysis is exploratory (greedy ranking at inference, not trained end-to-end), so efficiency gains are not fully realized.

Relevance to Agentic AI / LLM Agents

AVA-VLA directly addresses a core challenge in embodied agentic systems: how to maintain and exploit task history when the agent's world model is partial and observations are non-Markovian. The POMDP-grounded recurrent belief state is conceptually analogous to working memory or agent state in LLM-based agentic loops, offering a principled mechanism for grounding sequential decision-making in prior context. The AVA module's approach—using cross-attention between a compressed historical state and current perceptual tokens to produce dynamic attention masks—is a transferable pattern for any multimodal agent that must dynamically allocate perceptual bandwidth based on task history. This work is part of a broader trajectory (alongside CogACT, UniVLA, FLOWER) of moving VLA models from single-step reactive policies toward temporally coherent, history-aware agents capable of long-horizon manipulation.