Skip to content

Unified Diffusion VLA: Vision-Language-Action Model via Joint Discrete Denosing Diffusion Process

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; unified discrete diffusion VLA with future image understanding

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

UD-VLA proposes a unified Vision-Language-Action model where future image generation and action prediction are coupled through a single Joint Discrete Denoising Diffusion Process (JD3P), enabling actions to be iteratively refined under continuous visual guidance. Unlike prior unified VLAs that decouple these decoding streams, JD3P operates over a shared masked-token trajectory so each action token attends causally to all future image tokens at every denoising step. The result is SOTA performance on CALVIN, LIBERO, and SimplerEnv with 4× faster inference than autoregressive methods.

Problem

Existing unified VLAs fall into two failure modes: (1) modular architectures that rely on external expert encoders/decoders for vision and action, introducing cross-modal misalignment and weak coupling; (2) unified token-space models that nonetheless decode future images and actions in separate processes, preventing the rich mutual reinforcement that explicit joint generation could provide. Autoregressive interleaving is also insufficient because each action token integrates image information via only a single forward pass, rather than through iterative refinement.

Method

UD-VLA operates on a unified discrete token sequence: \([\text{text}; \text{current image}; \text{future image}; \text{action}]\). Images are tokenized with a VQ tokenizer (vocabulary size \(V_v\)) and actions with FAST (vocabulary size \(V_a\)).

JD3P: A joint BERT-style masked diffusion process over the concatenated future-image and action tokens \((\mathbf{v}_0, \mathbf{a}_0)\). A single augmented vocabulary of size \(V_m = V_v + V_a + 1\) (adding a \(\langle\text{MASK}\rangle\) token) is used. At noise level \(t\), each token is independently masked with probability \(\beta_t\): $\(Q_t \mathbf{e}_{t,r} = (1-\beta_t)\,\mathbf{e}_{t,r} + \beta_t\,\mathbf{e}_M.\)$ Denoising factorizes as \(p_\theta(\mathbf{v}_{t-1}, \mathbf{a}_{t-1} | \mathbf{v}_t, \mathbf{a}_t, c) = p_\theta(\mathbf{v}_{t-1}|\mathbf{v}_t, c)\,p_\theta(\mathbf{a}_{t-1}|\mathbf{v}_t, \mathbf{a}_t, c)\), so action denoising causally conditions on the current visual prediction at every step. Training collapses the multi-step chain to a single-step mask-predict cross-entropy loss with visual token down-weighting \(\omega\): $\(\mathcal{L}_{CE}(\theta) = -\omega\sum_j \log p_\theta^{(v)}(v_{0,j}|\mathbf{v}_t, c)\cdot\mathbf{1}[v_{t,j}=M] - \sum_i \log p_\theta^{(a)}(a_{0,i}|\mathbf{v}_t, \mathbf{a}_t, c)\cdot\mathbf{1}[a_{t,i}=M].\)$

Hybrid attention: Text and current image use causal and bidirectional attention respectively; future image tokens (generation block) and action tokens (acting block) use intra-block bidirectional + inter-block causal attention, enforcing the foresight → inverse-kinematics factorization without action-to-vision feedback.

Two-stage training: Stage (i) post-trains a pretrained VLM on large-scale video data for future image prediction only; stage (ii) fine-tunes jointly on robot action datasets under JD3P.

Inference optimizations: prefix KV-cache, pre-filled special boundary tokens, cosine mask schedule \(\rho_t = \cos\!\left(\frac{\pi}{2}\cdot\frac{T+1-t}{T+1}\right)\), confidence-guided top-K unmasking via Gumbel-max sampling, and modality-restricted decoding space mapping.

Key Contributions

  • JD3P: first joint discrete denoising diffusion process that couples future image generation and action prediction in a single synchronous denoising trajectory over a pretrained VLM backbone.
  • Hybrid attention mechanism that enforces causal cross-block flow (generation → acting) while allowing intra-block bidirectional attention, eliminating action-to-vision leakage.
  • Two-stage training pipeline to sequentially inject world-model capability and then robot-action capability into a pretrained VLM.
  • Inference-time techniques: confidence-guided decoding, prefix KV-cache, pre-filled boundary tokens, and decoding space mapping for 4× speedup over autoregressive baselines.

Results

  • CALVIN ABCD→D: avg. len 4.64 vs. MDT (4.52), UP-VLA (4.42), UniVLA* (4.26), MODE (4.39), GR-1 (4.21).
  • LIBERO: average success rate 96.1%; Object suite 98.8%, Goal suite 95.2%; outperforms CoT-VLA (avg ~87.5%), FlowVLA (93.2%), DreamVLA (97.5% Spatial only shown), UniVLA, F1.
  • SimplerEnv: SOTA on WidowX tasks (Put Spoon on Towel, Put Carrot on Plate, Stack Green on Yellow Block, Put Eggplant in Yellow Basket); specific per-task numbers not fully reproduced in the provided excerpt.
  • Inference speed: 4× faster than autoregressive methods due to parallel diffusion decoding.
  • Real-world robotic evaluations demonstrate practical generalization (qualitative; numeric results not shown in excerpt).

Limitations

  • Discrete tokenization introduces quantization error for both images and actions; VQ codebook fidelity bounds the ceiling of visual generation quality.
  • JD3P requires selecting a fixed number of denoising steps \(T\) at inference; the trade-off between speed and quality is not fully characterized across benchmarks.
  • Stage (i) video post-training requires large-scale video data; dependence on the quality and domain coverage of that data is not analyzed.
  • The hybrid attention design prohibits action-to-vision feedback, which may limit bidirectional refinement in settings where actions and images are highly co-dependent.
  • Real-world experiments are qualitative/demonstration-level; quantitative generalization metrics on physical hardware are absent from the excerpt.

Relevance to Vision-Language Models

UD-VLA demonstrates how discrete diffusion—a technique gaining traction in language modeling—can be extended to multimodal generation in the VLM-for-robotics setting, directly coupling visual generation and action prediction within a single pretrained VLM backbone. The JD3P framework is a concrete instantiation of intrinsic cross-modal synergy that contrasts with the dominant paradigm of separate diffusion heads conditioned on VLM embeddings, showing that joint denoising yields measurable task performance gains. The hybrid attention and unified tokenization design are directly relevant to broader efforts in native multimodal generation (understanding, generation, and acting in one model), a key open problem for next-generation VLMs. The 4× inference speedup from parallel discrete diffusion over autoregressive decoding is a practically important result for deploying large VLMs in latency-sensitive embodied settings.