ChatVLA-2: Vision-Language-Action Model with Open-World Reasoning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ChatVLA-2 is a Vision-Language-Action (VLA) model that addresses catastrophic forgetting of VLM capabilities during robotic fine-tuning. It combines a dynamic Mixture-of-Experts (MoE) backbone with a two-stage training pipeline to preserve open-world reasoning (math, OCR, spatial) while learning robot control. The model demonstrates zero-shot generalization to out-of-distribution equations and unseen objects, surpassing prior VLA baselines in open-world scenarios.

Problem

Existing VLA models fine-tuned on robot-specific data suffer from "knowledge erosion": the pretrained VLM's general capabilities (math reasoning, OCR, spatial understanding, object recognition) degrade as model parameters are repurposed for manipulation. This limits robots to tasks explicitly covered in training data, undermining the rationale for using large pretrained VLMs at all.

Method

ChatVLA-2 uses Qwen2-VL as its VLM backbone with DexVLA as the base architecture and a pretrained 1B ScaleDP action expert. Two core architectural changes are made:

1. Dynamic MoE in the LLM backbone: The dense MLP layers are replaced with a dynamic mixture-of-experts (8 experts, 2 activated per token). This disentangles feature spaces for multimodal understanding vs. robotic control while sharing common representations (e.g., spatial reasoning). Static/shared experts are avoided because they disrupt Qwen2-VL's native architecture and degrade pretrained knowledge.

2. Reasoning-following enhancement module: In the action expert's latter-half layers, the original observation embedding is replaced with MLP-projected reasoning tokens from the VLM's upper layers. These are combined with timestep embeddings to generate FiLM-style scale/shift parameters, injecting reasoning context into action generation without disrupting lower-layer dynamics.

Two-stage training: Stage 1 co-trains on image-text data (COCO, TextVQA, GQA, robot-scene pairs) and robot trajectories (600 math-game + 300 toy-placement) at a 1:3 image-text-to-robot ratio for 50k steps, preserving VLM knowledge while connecting it to actions. Stage 2 freezes the VLM entirely and trains only the action expert, forcing actions to follow the frozen VLM's reasoning outputs.

Key Contributions

• Dynamic MoE architecture that disentangles conflicting feature spaces between multimodal understanding and robotic control within a pretrained VLM backbone (Qwen2-VL)
• Reasoning-following enhancement module that injects VLM reasoning tokens into the action expert's latter layers via learned scale/shift conditioning
• Two-stage training strategy separating knowledge preservation (Stage 1) from reasoning-to-action alignment (Stage 2)
• Demonstrated zero-shot open-world generalization: math equations and spatial instructions entirely absent from training data
• Comprehensive VQA benchmark evaluation showing VLM capability retention alongside robotic control

Results

Math Matching Game (open-world): - ChatVLA-2: 43/52 success rate, OCR score 3.58/4, math reasoning score 1.73/2 - All baselines (Octo, Diffusion Policy, OpenVLA, GR00T N1, DexVLA, ChatVLA, π0): 0/52 success rate in open-world

Toy Placement (open-world): - ChatVLA-2: 127/156 success rate (81.4%), object recognition 0.94, spatial affordance 0.88 - DexVLA: 36/156 (ChatVLA-2 is 3.52× better); π0: 25/156; ChatVLA: 22/156

VQA benchmarks (vs. ChatVLA predecessor): - TextVQA: 78.2 (+7.0), ChartQA: 71.6 (+11.7), InfoVQA: 59.7 (+6.4), OCRBench: 792 (+63), MME: 1464.5 (+29.3); improvements on 11/12 benchmarks

Ablation (MoE): - Dynamic MoE: 43/52; Static MoE+Dynamic: 11/52; 3B Dense: 2/52; 7B Dense: 8/52

Ablation (training stages): - Stage 1 only (no reasoning-following): 12/52; Stage 2 only (no co-training): 3/52; Both: 43/52

Limitations

• Evaluated exclusively on two custom real-robot tasks; no standard simulation benchmarks used (authors claim current benchmarks cannot assess these capabilities)
• Training data scale is modest (600 + 300 robot trajectories), limiting assessment of scalability
• Dynamic MoE adds inference overhead; 8 experts with top-2 routing increases compute vs. dense baseline
• No ablation on the number of experts or which layers benefit most from MoE
• Open-world reasoning capability depends entirely on the quality of the pretrained VLM backbone; the method does not learn new knowledge, only preserves it
• Stage 2 freezes the VLM, so the action expert must generalize to any reasoning the frozen VLM produces — failure modes for truly novel reasoning chains are not characterized

Relevance to Vision-Language Models

ChatVLA-2 directly addresses a critical failure mode when deploying VLMs in embodied settings: fine-tuning for downstream tasks erases the general-purpose representations that motivated using VLMs in the first place. The dynamic MoE approach offers a principled architectural solution to the VLM-control feature conflict that is broadly applicable to any dense-backbone VLM being adapted for action generation. The reasoning-following module also connects to the wider VLM alignment question of whether model outputs faithfully reflect internal reasoning, a concern documented in chain-of-thought work on LLMs. For VLM researchers, this paper establishes that preserving pretrained multimodal knowledge during task-specific fine-tuning requires both architectural disentanglement and staged training protocols, not just data mixing.