Skip to content

DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DreamVLA replaces full future-frame prediction in Vision-Language-Action models with compact world knowledge forecasting—dynamic regions, depth maps, and semantic features—as an explicit intermediate reasoning step before action generation. A block-wise structured attention mechanism keeps the three knowledge streams disentangled, and a diffusion transformer decodes latent embeddings into action sequences. The framework achieves 4.44 average task length on CALVIN ABC-D and 76.7% success on real-world Franka robot tasks.

Problem

Prior VLA methods that incorporate future-state prediction either reconstruct full future frames (redundant pixel information, no 3D understanding, no semantic context) or rely on a separate copilot generation model (extra latency, two-stage training). Directly predicting raw images conflates background pixels with task-relevant regions, injects noisy gradients, and fails to supply the spatial and semantic look-ahead that action planning requires.

Method

World knowledge prediction. At each step the model predicts three complementary future signals rather than a full frame: 1. Dynamic regions: CoTracker extracts binary motion masks of pixels co-moving with the end-effector or manipulated objects; the model reconstructs only these masked tokens via a dVAE ELBO objective (Eq. 4–5). 2. Depth: a dedicated query regresses the future depth map using Depth-Anything pseudo-labels with scale-normalized MSE (Eqs. 6–7) to remove global scale ambiguity. 3. Semantics: future DINOv2/SAM features are predicted with an InfoNCE contrastive loss (Eq. 8); spatially shifted features serve as negatives.

Architecture. A GPT-2-based LLM processes CLIP text tokens, MAE visual patches, and proprioceptive state tokens together with learnable <dream> and <action> queries. Three lightweight convolutional decoders project the world embedding into the three prediction targets (training only; skipped at inference). A denoising diffusion transformer (DiT) conditioned on the latent action embedding generates an n-step action sequence from Gaussian noise.

Block-wise structured attention. Each of the three dream sub-queries attends to shared vision/language/state tokens but is masked from all other sub-queries, preventing cross-modal leakage (dynamic flow details contaminating depth reasoning, etc.). The <action> query likewise uses causal attention restricted to past context.

Key Contributions

  • Reframes VLA as a perception–prediction–action loop using compact, multi-modal world knowledge rather than raw future-frame synthesis.
  • Dynamic-region-guided forecasting via binary optical flow masks (CoTracker), avoiding the complexity of full dense flow prediction.
  • Block-wise structured attention with per-modality sub-queries to maintain disentangled representations across dynamic, depth, and semantic streams.
  • Diffusion transformer action decoder that separates action representation from the shared world-embedding latent space.

Results

  • CALVIN ABC-D (avg. task length, 1000 rollouts): DreamVLA 4.44 vs. VPP 4.29, Seer 4.28, Robovlm 4.25, UP-VLA 4.08, Ď€0 3.84, UNIVLA 3.80, RoboDual 3.66, OpenVLA 3.27.
  • LIBERO (average across Spatial/Object/Goal/Long): DreamVLA 92.6% vs. CoT-VLA 81.1%, SpatialVLA 78.1%, OpenVLA 76.5%, Octo 75.1%, Diffusion Policy 72.4%.
  • Real-world Franka (pick, place, drawer): DreamVLA 76.7% overall vs. Diffusion Policy 50.8%, Octo-Base 45.0%, OpenVLA 35.0%.
  • Ablation (CALVIN): dynamic region alone → 4.32; +depth → 4.40; +all → 4.44. Depth or semantics alone degrades below vanilla VLA baseline (3.64), confirming they require the dynamic region as anchor.
  • Structured attention vs. vanilla causal: 4.44 vs. 4.23. Separated vs. shared queries: 4.44 vs. 4.17. Future prediction vs. auxiliary reconstruction: 4.44 vs. 4.14.

Limitations

  • Dynamic regions provide the dominant gain; depth and semantic streams offer marginal incremental benefit (~0.08 and ~0.04 avg length) and individually harm performance, suggesting the design is sensitive to loss weighting.
  • DINOv2 features were found unstable/unhelpful; only SAM semantics are retained in the final model, reducing claimed "comprehensive" semantics in practice.
  • Relies on off-the-shelf CoTracker (optical flow) and Depth-Anything at data-generation time; deployment in environments with unusual motion patterns or poor depth estimation may degrade supervision quality.
  • LLM backbone is GPT-2—a relatively small, task-specific transformer rather than a modern large VLM—limiting the language generalization promised by the broader VLA paradigm.

Relevance to Vision-Language Models

DreamVLA directly extends the VLA paradigm by inserting structured world-model reasoning between perception and action, mirroring chain-of-thought prompting in language models. The block-wise attention design offers a practical template for integrating heterogeneous auxiliary supervision signals (optical flow, depth, foundation-model features) into a shared VLM backbone without representation collapse. For researchers tracking VLMs, the key insight is that replacing pixel-level generation with semantically structured intermediate predictions (dynamic masks, depth, SAM features) is a more token-efficient and gradient-stable approach to grounding language instructions in spatial-temporal context. The contrastive semantic forecasting with DINOv2/SAM also demonstrates a general recipe for bootstrapping VLM world models from off-the-shelf vision foundation models.