Skip to content

LaViDa: A Large Diffusion Language Model for Multimodal Understanding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LaViDa introduces the first family of Vision-Language Models (VLMs) built on discrete diffusion language models (DMs) rather than autoregressive (AR) generation. By equipping a pretrained DM backbone with a vision encoder and three novel training/inference techniques, LaViDa matches AR VLMs on standard benchmarks while enabling parallel decoding, flexible speed-quality tradeoffs, and strong constrained generation via text infilling.

Problem

Existing AR VLMs (LLaVA, Qwen-VL, InternVL, GPT-4) generate tokens sequentially, making inference slow and inherently unable to leverage bidirectional context. This makes them poor at controllable generation tasks (structured output formatting, text infilling, constrained poem completion) that require global sequence coordination. Discrete diffusion models address both issues in language-only settings but their application to multimodal tasks was unexplored.

Method

LaViDa follows the LLaVA architectural blueprint but replaces the AR LLM with a diffusion language model (LLaDA-8B or Dream-7B). A SigLIP-400M vision encoder processes the image as five 384² views (one resized original + four crops of a 768² upscale), producing 980 pooled embeddings passed through an MLP projector into the DM backbone. The DM uses a non-causal bidirectional attention mask and the masked-diffusion objective.

Three novel techniques address key challenges: - Complementary masking: For each training sample, two complementary masked versions with disjoint corrupted spans are generated, ensuring all tokens (including sparse, critical answer tokens) contribute to the loss. Adds ~8% training overhead. - Prefix-DLM: A specialized attention mask where image and prompt tokens attend only to each other while answer tokens attend to all tokens, enabling KV caching of the multimodal prefix. This converts DMs' inherent caching limitation into near-AR-level efficiency. - Timestep shifting (schedule shift with α=1/3): A convex monotonic remap of the linear unmasking schedule that front-loads token unmasking, dramatically improving quality at low NFE (fewer diffusion steps). Unlike continuous diffusion (SD3), the optimal schedule is convex (α<1) rather than concave.

Training is two-stage: stage-1 trains only the MLP projector on 558K image-text pairs; stage-2 jointly fine-tunes all components on 1M instruction-following examples. Optional stage-3 variants (LaViDa-Reason, LaViDa-FIM) specialize for CoT reasoning and fill-in-the-middle respectively.

Key Contributions

  • First diffusion-based VLM family (LaViDa), demonstrating DMs are viable for multimodal instruction following at scale.
  • Complementary masking scheme for data-efficient DM training on vision-language tasks.
  • Prefix-DLM inference with KV caching for multimodal prompts, achieving up to 3.9× speedup over full-attention DM decoding.
  • Timestep shifting schedule (α=1/3 convex map) enabling graceful quality degradation at reduced NFE.
  • Systematic design study of DM adaptation choices (backbone, resolution, pooling) for vision-language tasks.

Results

  • MMMU: LaViDa-L 43.3 vs. LLaVA-1.6-7B 37.4, Open-LLaVA-Next-8B 35.1 (best among comparable-data models).
  • ScienceQA: LaViDa-D 81.4 (best), LaViDa-L 80.2 vs. LLaVA-1.6-7B 78.3.
  • MathVista: LaViDa-L 44.8 vs. LLaVA-1.6-7B 41.8, Open-LLaVA-Next-8B 34.4.
  • COCO Captioning (CIDEr) at NFE=100%: LaViDa 117.3 vs. Open-LLaVA-Next-Llama3-8B 111.8 (+4.1 CIDEr) at 1.92× speedup; at NFE=50%, still outperforms AR baselines at lower latency.
  • Constrained Poem Completion: LaViDa and LaViDa-FIM achieve 100% sentence- and sample-level constraint satisfaction vs. LLaVA-1.6-7B 41%/37% and Qwen2.5-VL-7B 45%/16%.
  • Prefix-DLM KV cache: up to 3.9× latency reduction vs. full-attention DM at NFE=100%.
  • Complementary masking ablation: +67% relative improvement on ScienceQA (28.4→33.4 MathVista, 38.56→41.78 MMMU).
  • Timestep shifting at NFE=25%: CIDEr 101.1 (α=1/3) vs. 84.9 (linear) vs. 87.7 (cosine); on MathVision at NFE=50%, 21.05 vs. 16.12 (linear).
  • OCR (ChartQA, TextVQA, DocVQA): LaViDa competitive but lags behind frontier AR models (e.g., ChartQA: 64.6 vs. Qwen2.5-VL 87.3), attributed to average-pooling-induced loss of fine-grained spatial detail.

Limitations

  • Average pooling of vision tokens (to manage sequence length under limited compute) degrades fine-grained spatial reasoning and OCR performance relative to frontier AR VLMs.
  • At NFE=25%, generation quality drops noticeably below AR baselines despite the improved schedule.
  • Prefix-DLM introduces a train/inference attention mask mismatch (stage-2 trained with full attention, Prefix-DLM applied at inference); training with Prefix-DLM mask or custom kernels was explored but found to add overhead with minimal benefit.
  • DMs still require specifying a maximum generation length L at inference, unlike AR models which naturally terminate.
  • Evaluated only at 8B-scale with limited training data (1M SFT examples) compared to frontier models (Qwen2.5-VL: >7M pretrain images, 2M SFT).

Relevance to Vision-Language Models

LaViDa directly challenges the AR paradigm that underlies virtually all competitive VLMs (LLaVA, Qwen-VL, InternVL) by showing that a diffusion-based backbone can match AR performance on standard benchmarks (MMMU, MathVista, ScienceQA) while providing structurally superior controllability and inference flexibility. The work is particularly relevant as VLM deployment increasingly demands structured output (JSON schemas, form filling) and low-latency serving, both areas where DMs have inherent architectural advantages. The three technical contributions (complementary masking, Prefix-DLM, timestep shifting) are general recipes applicable to any multimodal DM extension, making this a foundational paper for the emerging diffusion-VLM research direction. It opens questions about scaling DM-based VLMs and whether bidirectional context yields consistent gains on more complex reasoning tasks.