Skip to content

VideoVLA: Video Generators Can Be Generalizable Robot Manipulators

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VideoVLA adapts a large pre-trained video diffusion transformer (CogVideoX-5B) into a vision-language-action model by adding action as a jointly denoised output modality alongside future video frames. The dual-prediction objective—forecasting both 7-DoF actions and future visual outcomes—creates an internal consistency signal that substantially improves generalization to novel objects and cross-embodiment skill transfer compared to VLAs built on understanding-oriented backbones.

Problem

Existing VLA models (OpenVLA, π0, CogACT) inherit pre-trained vision-language understanding models, which limits their generalization to unseen tasks, novel objects, and unfamiliar embodiments. Video generation models, by contrast, learn rich physical dynamics from massive real-world video data, but have not been directly adapted into end-to-end robot manipulators.

Method

VideoVLA builds on CogVideoX-5B, a multi-modal Diffusion Transformer (DiT). The architecture takes T5-encoded language tokens and the latent of the current observation (from a 3D-causal VAE) as conditioning; it then jointly denoises two output streams via DDPM: (1) future frame latents {V²…Vⁿ} representing anticipated visual outcomes, and (2) a chunk of K=6 future 7-D actions. All modalities are projected into a shared embedding dimension and processed by the same self-attention stack—no separate action head. During inference, DDIM sampling (50 steps) produces both action predictions and imagined video; only the first 3 of 6 predicted actions are executed before re-observation. Pre-training uses the OXE dataset (22.5M frames, 22 embodiments); real-world experiments fine-tune on 5,824 Realman-robot trajectories.

Key Contributions

  • First end-to-end VLA model built by directly adapting a large-scale pre-trained video generator (CogVideoX-5B) rather than a vision-language understanding model.
  • Dual-prediction strategy that jointly denoises video latents and actions within a single DiT, enforcing visual-action consistency.
  • Empirical demonstration that motion similarity between imagined futures and actual executions correlates with task success rate—providing a proxy metric for action reliability.
  • State-of-the-art generalization: best average on novel-object pick-up (65.2% vs. CogACT's 42.4%) and cross-embodiment skill transfer (48.6% vs. CogACT's 20.4%) in simulation; best on all real-world generalization evaluations.

Results

  • In-domain simulation (SIMPLER, 12 tasks overall): VideoVLA achieves 63.0% overall average, highest among all baselines (CogACT: 62.6%, π0: 50.0%).
  • Novel objects (Google robot, 10 YCB/GSO objects): VideoVLA 65.2% vs. CogACT 42.4%, SpatialVLA 50.8%, π0 28.8%.
  • Skill generalization (WidowX→Google, 7 skills): VideoVLA 48.6% vs. CogACT 20.4% (+28.2 pp).
  • Real-world in-domain (Realman robot): VideoVLA 64.6% vs. CogACT 58.4%.
  • Real-world novel objects (12 objects): VideoVLA 50.6% vs. CogACT 26.9%; VideoVLA covers all 12 objects while OpenVLA/SpatialVLA score 0% on roughly half.
  • Real-world skill transfer: VideoVLA 58.0% vs. CogACT 35.1%.
  • Ablation—backbone: CogVideoX-5B pretrained → 80.4% avg; trained from scratch → 12.6%; OpenSora-1.1 → 50.2%.
  • Ablation—dual prediction: removing video loss drops average from 80.4% to 27.0%; action-only drops to 25.5%.
  • Ablation—temporal horizon: 49/25/13 frames → 80.4/77.4/75.2% avg; more frames consistently helps.
  • Imagination-execution correlation: visual imagination success 84.0% (novel objects) / 63.4% (new skills) vs. actual execution 65.2% / 48.6%.

Limitations

  • Inference requires 50 DDIM denoising steps over a large 5B-parameter DiT, making real-time deployment computationally expensive.
  • For real-world efficiency, future frame prediction is reduced to 13 frames (vs. 49 in simulation), suggesting a speed-accuracy trade-off that is not fully characterized.
  • Visual imagination success is evaluated by human judges rather than an automated metric, limiting scalability of analysis.
  • Cross-embodiment skill transfer still relies on shared training data from source embodiments (WidowX in OXE); truly zero-shot transfer to unseen embodiments is not demonstrated.
  • Generalization is evaluated on manipulation-centric benchmarks; performance on longer-horizon, multi-stage tasks is not assessed.

Relevance to Vision-Language Models

VideoVLA challenges the dominant paradigm in embodied VLMs—using frozen vision-language understanding encoders—by demonstrating that generative video priors transfer more effectively to open-world manipulation than discriminative VLM priors. This is directly relevant to researchers tracking VLMs because it reframes the backbone selection question: generative world-model pre-training may encode physical grounding that perception-focused VLMs (CLIP, LLaVA, InternVL) lack. The dual-prediction design also echoes recent trends in VLM alignment where auxiliary generative objectives improve downstream task performance. For the embodied-AI branch of VLM research, VideoVLA establishes a strong baseline showing that video diffusion models are viable and superior alternatives to LLaVA-style backbones for grounded action generation.