Skip to content

Pelican-VLA 0.5: Attending Before Acting Benefits Generalization

🕒 Published (v1): 2026-07-07 17:50 UTC · Source: Arxiv · link

Why this paper was selected

Unified VLA with attention-before-action; integrates future-frame gen + action in one model

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Pelican-VLA 0.5 is a unified Vision-Language-Action model that inserts a compact set of learnable Bottleneck Tokens (BotTokens) between the perception and action pathways of a shared Qwen3-VL 4B backbone, inducing manipulation-centric attention during pre-training without any object annotations or attention supervision. After fine-tuning on RoboTwin, it achieves 91.4%/91.0% success on Clean/Randomized settings—best among open-source VLA baselines—while its attention patterns remain nearly identical to the pre-trained model. The work identifies a clear "attention-to-action gap": the model already learns what to attend to during pre-training, but reliable zero-shot execution still requires stronger action-level generalization.

Problem

Current VLA models generalize poorly to unseen objects, scenes, tasks, and embodiments because their action decoders attend diffusely over all visual tokens—robot arms, background clutter, task-irrelevant objects—rather than consistently focusing on manipulation-relevant regions. This diffuse attention encourages reliance on environment-specific visual shortcuts rather than transferable, task-centric representations, making cross-domain transfer dependent on additional data collection and fine-tuning.

Method

Pelican-VLA 0.5 shares a single Qwen3-VL 4B Transformer backbone across three functions: vision-language understanding, future-frame prediction, and action generation. The input sequence is organized as four segments:

\[Z = [\underbrace{P}_{\text{vision-language prefix}};\; \underbrace{M}_{\text{Cosmos latents}};\; \underbrace{S}_{\text{K BotTokens}};\; \underbrace{U}_{\text{state + noisy action}}]\]

Bottleneck Tokens (BotTokens): \(K=32\) learnable tokens \(S_0 \in \mathbb{R}^{1 \times K \times D}\) (initialized from \(\mathcal{N}(0, 0.02^2)\)) are inserted between perception and action. A hard bottleneck attention mask blocks all direct edges from the action suffix to upstream perception tokens (\(A_{i \to j} = 0\) for \(i \in \text{suffix},\, j \in \{P, M\}\)), forcing all perception-to-action communication through \(S\). This constraint is applied via a curriculum schedule: at step \(t\), the hard mask is applied with probability \(p_t = \min(1,\, t / T_\text{warm})\) with \(T_\text{warm}=10\text{k}\), allowing dense attention early in training for stability.

Three auxiliary mechanisms enforce bottleneck quality: 1. Orthogonality regularizer: penalizes the BotTokens Gram matrix deviating from identity (\(L_\text{reg} = \frac{1}{BK^2}\sum_b \|\hat{Z}^\top_b \hat{Z}_b - I_K\|_F^2\)) to encourage each token to encode distinct information. 2. BotTokens-gated future-frame generation: a channel-wise gate \(g = \sigma(W_g \bar{z})\) (where \(\bar{z}\) is the mean-pooled BotTokens output) modulates Cosmos latent features, tying BotTokens to forward-predictive structure. 3. BotTokens-language contrastive alignment: symmetric InfoNCE loss between trajectory embeddings (from BotTokens) and language instruction embeddings.

Training combines: flow-matching action loss (\(L_\text{action}\)), future-frame prediction loss (\(L_\text{gen}\), \(\lambda=0.01\)), contrastive alignment (\(L_\text{task}\), \(\lambda=0.1\)), and orthogonality regularization (\(L_\text{reg}\), \(\lambda=0.01\)). At inference, only the \(K\) BotTokens KV pairs are cached for the denoising loop (10 Euler steps), discarding the much larger prefix/middle cache.

Key Contributions

  • Introduces Pelican-VLA 0.5, a unified architecture unifying vision-language understanding, future-frame prediction, and action generation in a single Qwen3-VL 4B Transformer via a four-segment token sequence.
  • Identifies attention-level generalization emerging from pre-training: without task-specific fine-tuning, object annotations, or attention supervision, the action pathway already focuses on manipulation-relevant objects and contact regions across unseen scenes and embodiments.
  • Verifies that this property originates from the BotTokens bottleneck via a curriculum attention mask + orthogonality regularizer + generation gating; removing BotTokens after training partially degrades but does not eliminate this property.
  • Articulates the attention-to-action gap: attention-level generalization precedes action-level generalization; fine-tuning mainly improves the mapping from pre-formed attention to executable actions rather than creating attention structure from scratch.
  • Achieves best average open-source VLA result on RoboTwin (91.2% average) and shows non-zero zero-shot success on fully held-out RoboTwin 2.0 tasks, embodiments, and scenes.

Results

  • RoboTwin Clean: 91.4% (vs. next best Hy-VLA 90.9%, JoyAI-RA 90.5%)
  • RoboTwin Randomized: 91.0% (vs. Hy-VLA 90.1%, JoyAI-RA 89.3%)
  • RoboTwin Average: 91.2% (vs. Hy-VLA 90.5%, JoyAI-RA 89.9%)
  • Clean–Randomized gap is only 0.4 points, indicating robustness to visual randomization
  • Zero-shot (no fine-tuning) on held-out RoboTwin 2.0: non-zero success on tasks including picking bottles, placing toy cars, and switch activation; failures concentrated at fine-grained grasp/placement stages
  • Baselines compared: \(\pi_0\), \(\pi_{0.5}\), X-VLA, StarVLA-OFT, ABot-M0, LingBot-VLA, Qwen-VLA, JoyAI-RA, Hy-VLA
  • Pre-training used only 0.4 epoch (~2,400 hours) over a 6,000+ hour mixture, so results are obtained in an under-trained regime

Limitations

  • Joint-position action representation is embodiment-specific; end-effector representations would improve cross-embodiment transfer.
  • Only 0.4 epoch of pre-training (~2,400 of 6,000+ available hours); results are explicitly under-trained.
  • Zero-shot success rates on RoboTwin 2.0 remain low; failures occur at grasp stability and precise placement rather than target selection.
  • A representation-to-action gap persists: the model learns what to attend to before learning to reliably act on it.
  • No evaluation beyond simulation (RoboTwin); real-robot zero-shot results shown only qualitatively in figures.
  • Task-prompt overlap between InternData-A1 and RoboTwin 2.0 required manual auditing, and some residual leakage cannot be fully excluded.

Relevance to Vision-Language Models

This paper directly addresses how vision-language representations in VLA models are routed to action generation—a core concern for VLM-to-embodiment grounding research. The BotTokens mechanism is a novel architectural intervention that constrains the information bottleneck between a frozen/shared VLM backbone (Qwen3-VL) and a downstream task head, with implications for how VLMs can be adapted to action-producing tasks without supervision on intermediate representations. The finding that manipulation-centric attention emerges from pre-training alone—preceding action competence—suggests that VLM backbone choice and attention routing are as critical as action supervision for generalization. For VLM researchers, this work provides empirical evidence that grounding quality (as measurable through attention patterns) and behavioral competence dissociate and can be separately optimized.