Skip to content

Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding

🕒 Published (v1): 2026-07-07 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Nemotron-Labs-Diffusion is a tri-mode language model family (3B/8B/14B) that jointly optimizes autoregressive (AR) and block-wise diffusion objectives within a single architecture, enabling three inference modes: standard AR, parallel diffusion decoding, and self-speculation (diffusion drafts, AR verifies). The joint objective yields complementary gains: AR anchors left-to-right linguistic priors, while diffusion enhances lookahead planning. The family includes vision-language variants and outperforms SOTA open-source AR and diffusion LMs in both accuracy and throughput.

Problem

Autoregressive LMs decode sequentially, limiting parallelism and throughput at low concurrency. Existing diffusion LMs parallelize decoding but trail AR models in accuracy due to ignoring left-to-right language structure, and they fail to clearly outperform multi-token prediction (MTP) methods in practical efficiency–accuracy trade-offs. No prior work unified AR and diffusion objectives within a single model that adapts across concurrency regimes.

Method

Joint AR-diffusion training. The model is trained on a weighted combination \(\mathcal{L}(\theta) = \mathcal{L}_\text{AR}(\theta) + \alpha\,\mathcal{L}_\text{diff}(\theta)\) with \(\alpha=0.3\). The AR loss is the standard left-to-right cross-entropy. The diffusion loss uses block-wise denoising: the sequence is partitioned into \(B\) contiguous blocks; at noise level \(t \sim \mathcal{U}[0,1]\) only the current block is corrupted while the prefix remains clean: $\(\mathcal{L}_\text{diff}(\theta) = \mathbb{E}\!\left[-\frac{1}{B}\sum_{b=1}^B \log p_\theta\!\left(x^b \mid \tilde{x}^b_t, x^{<b}\right)\right]\)$ Attention is bidirectional within each block and causal across blocks; the clean stream uses a strictly causal mask to enable AR and diffusion objectives in a single forward-backward pass without label leakage.

Two-stage training. Stage 1 trains on the pure AR objective (\(\alpha=0\)) to establish left-to-right priors; Stage 2 enables joint training (\(\alpha=0.3\)).

Global loss averaging treats all tokens across the batch equally (\(\mathcal{L}_\text{global} = \frac{1}{NL}\sum_{n,i}\ell_{n,i}\)) rather than per-sequence, reducing gradient variance from variable masking ratios at different noise levels.

Tri-mode inference. - AR mode: standard causal left-to-right decoding; preferred at high concurrency. - Diffusion mode: iterative block-wise parallel denoising using confidence-based or trained-sampler selection. - Self-speculation mode: diffusion generates \(k\) draft tokens in one forward pass; AR verifies them causally in a second pass, accepting the longest matching prefix (yielding 1 to \(k+1\) tokens per round). A LoRA adapter (\(\text{rank}=128\), \(\sim\)36M parameters on \(o_\text{proj}\)) fine-tunes the diffusion draft pathway using an LK-hybrid distribution-matching loss to align drafts with the AR verifier.

Key Contributions

  • Unified single-model architecture supporting AR, diffusion, and self-speculation decoding, switchable at inference time.
  • Demonstration that AR and diffusion objectives are complementary: both modes peak at \(\alpha=0.3\) with no zero-sum trade-off.
  • Self-speculation mode that outperforms MTP methods (e.g., Qwen3-Eagle3) in acceptance rate and per-user throughput at low concurrency.
  • Speed-of-light (SOL) analysis establishing that diffusion with an optimal sampler can correctly predict 76.5% more tokens per forward pass than self-speculation, quantifying remaining headroom.
  • LoRA-enhanced drafter with LK-hybrid loss for improved draft–verifier alignment.
  • Global loss averaging and DP-rank varying masking ratios as training stabilization techniques.
  • Full model family (base, instruct, VLM) at 3B/8B/14B released on HuggingFace.

Results

  • NLD-8B (diffusion mode) vs. Qwen3-8B: 6× tokens per forward pass; 4× system throughput on SPEED-Bench with SGLang on NVIDIA GB200.
  • NLD-8B vs. Qwen3-8B at batch size 1 on H100: 3.3× throughput in self-speculation mode; 1.4× in diffusion mode, with comparable accuracy across 10 benchmarks (HumanEval, MBPP, LiveCodeBench-CPP, GSM8K, Math500, AIME24, AIME25, GPQA, IFEval, MMLU).
  • Ablation (25B-token continuous pretraining from Ministral3-8B):
  • Block-wise attention baseline: 54.23% avg.
  • +Global loss averaging: +2.12% → 56.35%.
  • +DP-rank varying masking: +0.71% → 57.06%.
  • +Two-stage training: +5.74% → 62.80%.
  • +AR loss: +7.48% → 70.28%.
  • Cumulative gain: +16.05%.
  • AR-mode accuracy with diffusion loss (\(\alpha=0.3\)): Base model avg +0.14%; instruct model avg +0.43%, with gains on coding/math (e.g., +4.24% LCB-CPP, +1.59% MBPP) and minor drops on IFEval (−3.01%) and HumanEval (−2.44%).
  • SOL analysis: Optimal diffusion sampler predicts 76.5% more tokens per forward pass than self-speculation.
  • NLD-8B outperforms SDAR-8B, Dream-7B, LLaDA-8B (diffusion baselines) and Ministral3-8B, Qwen2.5-7B, Qwen3-4B (AR baselines) on the 10-task average.

Limitations

  • Sampling tokens to approach the SOL upper bound remains an open problem; the trained sampler closes only part of the gap.
  • Joint training slightly degrades strict instruction-following (IFEval −3.01% for instruct model).
  • Self-speculation advantage over MTP (Eagle3) diminishes at high concurrency, where AR mode is preferred.
  • The paper is truncated; full VLM-specific results, architecture details, and the quadratic self-speculation variant are not fully described in the provided text.
  • LoRA drafter training requires additional data collection of decoding trajectories, adding pipeline complexity.

Relevance to Vision-Language Models

The Nemotron-Labs-Diffusion family explicitly includes vision-language model variants at 3B/8B/14B scales, making its tri-mode decoding framework directly applicable to VLMs where inference throughput is a practical bottleneck. The finding that joint AR-diffusion training is complementary rather than competitive is significant for VLM researchers, as it suggests multimodal models can inherit parallel decoding capabilities without sacrificing the autoregressive accuracy that VLMs depend on for grounded language generation. The self-speculation mechanism — using a single model's diffusion mode to draft and AR mode to verify — offers a parameter-efficient alternative to auxiliary draft heads (as in MTP/Eagle methods), which are increasingly explored for VLM acceleration. Researchers tracking VLMs should watch whether the tri-mode approach transfers to vision-conditioned generation tasks, particularly in low-concurrency personal inference settings.