Skip to content

CogVLA: Cognition-Aligned Vision-Language-Action Models via Instruction-Driven Routing & Sparsification

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CogVLA is a Vision-Language-Action framework that addresses the computational bottleneck of VLM-based robot control via a 3-stage instruction-driven routing and sparsification pipeline. It achieves state-of-the-art 97.4% success rate on LIBERO and 70.0% on real-world tasks while reducing inference latency 2.8× and training cost 2.5× relative to OpenVLA.

Problem

Adapting pre-trained VLMs for robot action prediction is computationally expensive, yet existing sparsification techniques (Mixture-of-Depths, layer skipping, early exit) optimize intra-LLM computation in isolation, ignoring semantic coupling across vision, language, and action modalities. This modular approach causes: (i) loss of task-relevant visual features during encoder compression, (ii) disrupted contextual coherence from token skipping in the LLM, and (iii) incoherent action generation lacking cross-modal causal reasoning.

Method

CogVLA introduces a 3-stage progressive design biomimetically mapped to human neural coordination (VAS → SMA → PMC):

  1. EFA-Routing (Encoder-FiLM based Aggregation Routing, Stage 1): FiLM conditioning injects the language instruction into the vision encoder (dual-stream SigLIP + DINOv2), modulating scale/shift parameters to aggregate all image tokens into a single aggregation token per encoder. An instruction-conditioned sigmoid gate dynamically weights the two encoder outputs. Net result: visual tokens reduced to 25% of original size before the LLM.

  2. LFP-Routing (LLM-FiLM based Pruning Routing, Stage 2): Inside the LLM, a lightweight MLP computes per-token routing weights conditioned on the instruction via FiLM modulation. Tokens below the β-th percentile relevance threshold are skipped, pruning 50% of remaining tokens and skipping their attention computation.

  3. CAtten (V-L-A Coupled Attention, Stage 3): A unified hybrid attention mask combines (i) causal lower-triangular attention over V-L tokens (preserving instruction-conditioned reasoning), (ii) unidirectional attention within V-L for semantic consistency, and (iii) bidirectional attention within action tokens enabling parallel decoding of the full action chunk in a single forward pass.

Key Contributions

  • Cognition-aligned 3-stage architecture (EFA-Routing → LFP-Routing → CAtten) achieving end-to-end instruction-driven sparsification across the full perception-to-control pipeline
  • EFA-Routing: instruction-conditioned intra- and cross-encoder visual aggregation compressing to 25% scale
  • LFP-Routing: FiLM-guided token pruning within the LLM targeting task-irrelevant visual tokens
  • CAtten: hybrid causal + bidirectional attention enabling parallel action chunk decoding on doubly-compressed representations
  • 8× total visual input reduction relative to baseline VLA models

Results

  • LIBERO overall: 97.4% SR (rank 1), vs. OpenVLA-OFT 97.1%, Ï€0.5-KI 96.0%, Ï€0 95.8%, PD-VLA 94.7%
  • LIBERO-Long (hardest suite): 95.4% SR (rank 1), vs. OpenVLA-OFT 94.5%, Ï€0.5-KI 85.8%
  • Real-world ALOHA average: 70.0%, vs. OpenVLA-OFT 56.7%, STAR 45.0%, PD-VLA 50.0%
  • Inference time: 0.091 s vs. OpenVLA 0.254 s (2.79× faster); throughput 87.9 Hz vs. 3.9 Hz (22.5×)
  • FLOPs: 2.72 T vs. OpenVLA 8.48 T (3.12× reduction)
  • Training cost: 4.7 h/10k steps vs. OpenVLA 11.7 h/10k steps (2.49× reduction)
  • Stage 1+2 compression (98.6% Spatial SR) outperforms FastV (88.2%) and SliME (77.6%) as drop-in alternatives

Limitations

  • Ranks second on LIBERO-Goal suite (96.6% vs. OpenVLA-OFT 97.9%) due to explicit performance-efficiency tradeoff at 8× visual reduction
  • Requires dual vision encoders (SigLIP + DINOv2), increasing architectural complexity vs. single-encoder baselines
  • Real-world evaluation covers only 3 task types on a single hardware platform (Agilex ALOHA); generalization to other robot morphologies untested
  • The fixed β sparsity ratio is a hyperparameter that must be tuned per deployment; performance degrades notably at extreme ratios (1× Stage 1 / 8× Stage 2 drops to 92.0% Spatial SR)

Relevance to Vision-Language Models

CogVLA directly demonstrates how VLMs can be efficiently repurposed for closed-loop robot control without sacrificing cross-modal semantic integrity, a key challenge for the VLM community as these models scale to embodied applications. The FiLM-based instruction-conditioned token routing approach is a technically principled alternative to generic token pruning methods (FastV, SliME) that lack awareness of downstream task semantics. The CAtten mechanism generalizes the question of how to structure attention across heterogeneous modalities—relevant to any multimodal architecture requiring joint vision-language-action reasoning. Results show that instruction-aware sparsification substantially outperforms instruction-agnostic compression even at the same compression ratio, which is a transferable finding for efficient VLM inference broadly.