Skip to content

Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Training VLA models by naively adding a continuous action expert (e.g., flow matching) to a pretrained VLM backbone degrades both language following and training speed due to gradient interference from randomly initialized expert weights. This paper proposes knowledge insulation: simultaneously training the VLM backbone with discretized (FAST-tokenized) actions via next-token prediction while training the action expert with flow matching, with a stop-gradient blocking expert gradients from reaching the backbone. The result is a VLA that trains ~7.5× faster than π0, achieves fast continuous-action inference, and retains stronger VLM knowledge.

Problem

When a continuous action expert (diffusion/flow matching head) is grafted onto a pretrained VLM to form a VLA, the gradients from the randomly initialized expert corrupt the pretrained backbone's representations, causing degraded language following and slow, unstable training convergence. Simply freezing the backbone fails because pretrained VLMs lack sufficient robotic representations for high-performance policies.

Method

Knowledge insulation combines three mechanisms applied simultaneously in a single training stage:

  1. Joint discrete/continuous action prediction: The VLM backbone is trained with next-token prediction on FAST-tokenized discrete actions (cross-entropy loss), while the action expert is trained with flow matching on continuous action chunks. Only the action expert is used at inference time.
  2. Stop-gradient in cross-attention: The attention mechanism is modified so that gradient flow from action expert tokens back into backbone parameters is blocked via a stop_gradient operator on the key/value projections where backbone features are read by the action expert (Eq. 5–6). This isolates backbone weights from the flow matching loss.
  3. VLM data co-training: General vision-language data (VQA, image captioning, bounding box prediction, robot planning) is mixed into training using a combined loss mask, preserving and leveraging pretrained semantic knowledge.

The backbone is initialized from PaliGemma (3B); the action expert is a smaller separate transformer (300M) with its own Q/K/V projections that cross-attend to backbone features.

Key Contributions

  • Formal analysis of why naive continuous-action VLA training harms VLM knowledge (gradient interference from randomly initialized action experts).
  • A single-stage training recipe combining joint AR + flow-matching losses with stop-gradient insulation, enabling simultaneous fast convergence and fast inference.
  • Demonstration that FAST tokenization as a representation learning signal (training only, not inference) outperforms naive discretization for representation quality.
  • State-of-the-art results on LIBERO-90 and LIBERO-Spatial; open-source model weights released on DROID.
  • Extensive ablation over gradient flow, co-training data sources, state representations, and tokenization choices.

Results

  • DROID benchmark: Ours 0.55 ± 0.09 vs. Ï€0 0.49 ± 0.09 vs. Ï€0-FAST 0.45 ± 0.09.
  • LIBERO-90: Ours (from generalist) 96.0% vs. OpenVLA-OFT –, Ï€0 –, Ï€0-FAST –. Best published result on LIBERO-90.
  • LIBERO-Spatial: Ours (from generalist) 97.8% vs. Ï€0 98.8%; competitive with the best.
  • Training convergence: Reaches comparable table-bussing performance in ~7.5× fewer training steps than Ï€0.
  • Inference speed: Ï€0-FAST requires ~750 ms per action chunk; ours (continuous action expert) achieves 10 Hz control vs. Ï€0-FAST's ~1.3 Hz effective rate, taking roughly half the wall-clock time to solve table-bussing tasks.
  • Language following: Stop-gradient model follows language instructions significantly better than Ï€0 and joint-training without stop-gradient, approaching Ï€0-FAST's language adherence while matching or exceeding its task performance.
  • Items-in-drawer (held-out environments): Ours outperforms all baselines (Ï€0, Ï€0-FAST, HybridVLA, OpenVLA-OFT, frozen backbone) with large margins.

Limitations

  • Joint discrete + continuous training increases per-step compute cost by ~20%; net wall-clock is still faster due to faster convergence.
  • Language following remains imperfect; dataset correlations can still cause models to ignore language instructions.
  • LIBERO-10 (long-horizon) performance (85.8%) is below OpenVLA-OFT (94.5%) and MoDE (95.0%).
  • Evaluated primarily within the Ï€0/PaliGemma architecture family; generalization of the recipe to other backbone architectures is unverified.

Relevance to Vision-Language Models

This paper directly addresses the core tension in VLA research: how to adapt large pretrained VLMs to robot control without destroying the semantic representations that make web-scale pretraining valuable. The stop-gradient insulation and co-training recipe it proposes is a principled solution to catastrophic forgetting/interference in multimodal fine-tuning, with lessons directly applicable to any domain where a new continuous-output modality is grafted onto a discrete-token VLM. For researchers tracking VLMs, it establishes that the training objective and gradient routing—not just architecture—critically determine knowledge retention, and that hybrid discrete/continuous training is a powerful strategy for bridging representation learning and precise output generation.