Skip to content

InternVLA-A1.5: Unifying Understanding, Latent Foresight, and Action for Compositional Generalization

🕒 Published (v1): 2026-07-06 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

InternVLA series (Shanghai AI Lab); unifies semantic priors and future prediction in one VLA

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

InternVLA-A1.5 is a Vision-Language-Action model that preserves VLM semantics during robot policy learning by continuously co-training the backbone on VQA and subtask prediction, while offloading future prediction to a compact set of learnable "foresight tokens" supervised by a frozen video generation model (WAN2.2-5B). At inference the video branch is dropped entirely, retaining real-time speed. Pretrained on 1.2M robot episodes and 3M multimodal samples, it achieves state-of-the-art results on all six simulation benchmarks and the strongest compositional generalization on held-out instruction bindings in real-world tasks.

Problem

Existing unified VLA+world-model designs suffer three compounding failures: (1) joint generation and action objectives cause the pretrained VLM backbone's semantic and instruction-following ability to drift; (2) heterogeneous loss forms (pixel regression, flow matching, language modeling) interfere during joint training; and (3) the visual prediction module is learned from scratch in pixel space, ignoring spatiotemporal dynamics priors already encoded in large pretrained video generators.

Method

InternVLA-A1.5 uses a Mixture-of-Transformers (MoT) architecture coupling a 2B-parameter Qwen-3.5 VLM backbone with a lightweight 460M unified expert that shares full-attention layers with the VLM but keeps separate Gated DeltaNet linear-attention layers.

Stage 1 (VLM Transferring): The VLM is co-trained on robot manipulation data and VQA under a single cross-entropy objective. Each robot sample predicts a subtask description followed by a FAST-tokenized discrete action chunk; proprioceptive state is uniformly binned to 256 levels per DOF and appended as tokens. The joint distribution is factored as \(\pi_\theta(a_{t:t+H}, \hat\ell \mid o_t, \ell) = \pi_\theta(a_{t:t+H} \mid o_t, \hat\ell)\,\pi_\theta(\hat\ell \mid o_t, \ell)\).

Stage 2 (Foresight + Continuous Action): \(M=50\) learnable foresight tokens \(Q_f \in \mathbb{R}^{M\times d}\) attend to the shared VLM hidden states via the unified expert to produce contextualized embeddings \(Z_t^f\). These are projected and injected as conditioning into the frozen WAN2.2-5B denoising transformer (replacing its T5 text encoder) via cross-attention. A flow-matching video prediction loss \(\mathcal{L}_\text{video} = \mathbb{E}\|u(x_s, C_t^f, s) - v_s\|^2\) (where \(x_s = (1-s)x_0 + s x_1\)) back-propagates only through the conditioning pathway, updating foresight tokens without training the pixel generator. Continuous actions are predicted by the unified expert with a separate flow-matching head: \(\mathcal{L}_\text{action} = \mathbb{E}\|v_\theta^\text{act}(a^\tau_{t:t+H}, H_t, Q_f) - (a_{t:t+H} - \epsilon)\|^2\), using \(\tau \sim \text{Beta}(1.5, 1.0)\). The combined objective is \(\mathcal{L}_\text{stage2} = \mathcal{L}_\text{stage1} + \alpha\mathcal{L}_\text{video} + \beta\mathcal{L}_\text{action}\) with \(\alpha=1, \beta=10\).

At inference the entire video branch is discarded; action denoising runs via Euler integration over the foresight-conditioned flow.

Key Contributions

  • Demonstrates that continuous VQA + subtask co-training during policy learning preserves (and strengthens) VLM semantics rather than allowing semantic drift from heavy generation objectives.
  • Recasts future prediction as latent querying: foresight tokens distill dynamics priors from a frozen video generator into a compact code rather than learning pixel-level generation from scratch.
  • MoT architecture with shared full-attention and separate linear-attention layers decouples modality-specific processing while enabling cross-component context flow without full parameter coupling.
  • Training-only video branch: foresight knowledge is absorbed into the token embeddings and the expensive generator is discarded at inference, adding zero latency overhead vs. non-world-model baselines.
  • Unified action space across embodiments via padded morphology-specific slots and a single flow-matching action head.

Results

  • LIBERO simulation benchmark: 98.9% success (vs. \(\pi_{0.5}\) 93.2%, Motus baseline implied lower).
  • RoboTwin 2.0 (bimanual): Sort Tubes 78%, Insert Tubes 73%, Move Tubes 73% — consistently above \(\pi_{0.5}\) (65%, 52%, 56%) and Motus (76%, 44%, 81% on move tubes).
  • EBench, SimplerEnv, LIBERO-Plus, DOMINO: best overall across all six simulation benchmarks; leads on both zero-shot generalization tests (LIBERO-Plus, DOMINO).
  • Real-world compositional generalization (held-out instruction bindings): strongest performance among InternVLA-A1.5, \(\pi_{0.5}\), and Motus.
  • Long-horizon chemistry task (MOF Reaction): InternVLA-A1.5 84.8% vs. \(\pi_{0.5}\) 76%; clearly leads in long-horizon execution.

Limitations

  • Foresight supervision requires a 5B-parameter frozen video generator (WAN2.2-5B) during training, substantially increasing training compute and memory cost.
  • The video branch is discarded at inference, so the quality of future prediction cannot be directly inspected or corrected during deployment.
  • Only \(N=4\) future frames are sampled per action chunk; the temporal horizon of the foresight signal is therefore limited relative to long-horizon task length.
  • Evaluation is on a single robot platform and limited set of real-world tasks; broader multi-embodiment real-world generalization is not demonstrated.
  • No ablation quantifying the isolated contribution of foresight tokens vs. VQA co-training is fully presented in the provided text.

Relevance to Vision-Language Models

InternVLA-A1.5 directly addresses one of the central tensions in applying VLMs to embodied control: preserving the pretrained backbone's semantic and instruction-following capacity while injecting physical dynamics knowledge. Its continuous VQA co-training strategy is a concrete, replicable recipe for preventing catastrophic forgetting of VLM priors under action-generation objectives — relevant to any setting that fine-tunes VLMs for new modalities. The latent-querying foresight mechanism generalizes the broader paradigm of using frozen generative models as structured supervisors (rather than trainable components), connecting to techniques like knowledge distillation from video foundation models and query-based cross-modal adapters. Researchers tracking VLMs will find the MoT architecture and attention masking design informative for multi-objective joint training with heterogeneous loss scales.