HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Model¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Renrui Zhang (CUHK); ICLR 2026; hybrid diffusion+autoregressive VLA model

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

HybridVLA unifies autoregressive and diffusion-based action generation within a single LLM backbone for robotic manipulation, rather than appending a separate diffusion head after the VLM. A collaborative training recipe embeds DDIM denoising steps into the next-token prediction process, and a confidence-guided ensemble mechanism adaptively fuses both action types at inference. The model achieves +17% and +19% mean success rate over prior SOTA on simulation and real-world tasks, respectively.

Problem¶

Autoregressive VLAs (e.g., OpenVLA, RT2) discretize continuous actions into bins, sacrificing precision. Diffusion-based VLAs (e.g., π0, CogACT) attach an independent diffusion head after the VLM, treating the LLM purely as a feature extractor—the head lacks internet-scale pretraining and cannot use the LLM backbone as an iterative action expert. Neither approach captures both the continuity of diffusion and the pretrained reasoning of autoregression simultaneously.

Method¶

HybridVLA is built on a Prismatic VLM (LLaMA-2 7B or Phi-2 2.7B backbone) with DINOv2+SigLIP vision encoders. The core innovation is a token sequence formulation that organizes, in order: vision/language tokens → robot state token (continuous MLP projection) → <BOD> → diffusion noisy-action tokens → <EOD> → autoregressive discrete-action tokens. This ordering prevents GT leakage (diffusion precedes AR tokens) and avoids diffusion tokens predicting masked discrete tokens.

Collaborative training optimizes a joint loss: $$\mathcal{L}_{\text{hybrid}} = \mathcal{L}_{\text{dif}} + \mathcal{L}_{\text{ar}}$$ where $\mathcal{L}_{\text{dif}} = \mathbb{E}_{a,i,c}\|\epsilon - \epsilon_\pi(a_t^i, i, c)\|^2$ (DDIM noise prediction) and $\mathcal{L}_{\text{ar}}$ is cross-entropy over discretized action bins. Each DDIM denoising step is cast as a next-token prediction iteration inside the LLM; KV caching is reused across denoising steps to reduce latency.

Collaborative action ensemble at inference: if the mean confidence of autoregressive action tokens $c^{ar}_{t+1} > \theta$ ($\theta = 0.96$, satisfied in >80% of successful samples), the final action is $a_{t+1} = (a^d_{t+1} + a^{ar}_{t+1})/2$; otherwise only the diffusion action is used. Training is two-stage: large-scale pretraining on 760K cross-embodiment trajectories (Open X-Embodiment, DROID, ROBOMIND), then fine-tuning on self-collected data.

Key Contributions¶

Unified LLM backbone performing both DDIM denoising and autoregressive next-token prediction within a single token sequence—no separate diffusion head.
Token sequence formulation with <BOD>/<EOD> markers that prevents GT leakage, prevents diffusion tokens predicting masked discrete tokens, and provides continuous latent conditioning for downstream AR generation.
Collaborative training recipe (hybrid $\mathcal{L}_{\text{dif}} + \mathcal{L}_{\text{ar}}$) that produces mutual reinforcement between paradigms (validated by ablation and PCA).
Confidence-guided action ensemble that selects between pure diffusion and averaged diffusion+AR based on AR token confidence.
SOTA results on RLBench (10 tasks), real-world single-arm and dual-arm setups, with demonstrated generalization to unseen objects, backgrounds, spatial layouts, and lighting.

Results¶

RLBench simulation (10 tasks, mean success rate):
HybridVLA (7B): 0.78 ±0.04 vs. CogACT (7B): 0.60 ±0.04 (+17 pp) and π0 (2.6B): 0.61 ±0.03
HybridVLA-dif (7B): 0.72; HybridVLA-ar (7B): 0.65 — ensemble outperforms either alone
HybridVLA (2.7B): 0.67 at 12.3 Hz inference
ARP (four views): 0.68; OpenVLA-OFT: 0.45; OpenVLA: 0.41
Real-world (single-arm and dual-arm): +19% mean success rate over prior SOTA (CogACT 7B baseline shown at 55–61% vs. HybridVLA at 71–83% in bar charts)
Inference speed: HybridVLA (7B) at 6.1 Hz; HybridVLA (2.7B) at 12.3 Hz; CogACT at 9.8 Hz; OpenVLA-OFT at 13.4 Hz
Token sequence Type 4 (proposed) achieves 0.72 diffusion / 0.65 AR vs. Type 1 baseline at 0.67 / 0.59

Limitations¶

Inference speed (6.1 Hz for 7B) is lower than purely autoregressive methods (OpenVLA-OFT at 13.4 Hz) due to iterative DDIM steps inside the LLM.
Pretraining uses only single 2D observations; multi-view is introduced only at fine-tuning, limiting cross-embodiment diversity.
The confidence threshold $\theta = 0.96$ is empirically tuned and may require re-calibration for new task distributions.
Evaluation is on relatively short-horizon tabletop tasks (RLBench); long-horizon generalization beyond the ablated unseen conditions is not demonstrated.
The 760K trajectory pretraining requires >10K A800 GPU hours, limiting accessibility.

Relevance to Vision-Language Models¶

HybridVLA directly addresses how to convert a pretrained VLM into an action generator without degrading its pretrained knowledge—a central tension in the VLA literature. By embedding diffusion denoising as reasoning iterations within the LLM's next-token prediction loop, it reframes continuous action generation as a first-class VLM capability rather than a bolt-on module, offering a concrete architectural blueprint for future embodied VLMs. The confidence-based ensemble connects to emerging work on uncertainty-aware decoding in VLMs. The finding that AR and diffusion predictions are complementary—AR excels at instruction following and novel objects, diffusion at precise fine-grained control—provides empirical grounding for hybrid generation paradigms that are increasingly relevant across multimodal generation tasks beyond robotics.