Skip to content

Uni-World VLA: Interleaved World Modeling and Planning for Autonomous Driving

🕒 Published (v1): 2026-03-28 14:39 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; interleaved world modeling and planning unifies AD perception and action

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Uni-World VLA proposes an interleaved world modeling and planning framework for autonomous driving that alternates between predicting future frames and ego actions step-by-step, rather than generating a full scene rollout before planning. This closed-loop coupling prevents "frozen hallucination" of outdated future states and is augmented with monocular depth fusion for stronger geometric reasoning. On NAVSIM, it achieves state-of-the-art PDMS of 89.4 among single-camera methods.

Problem

Prior unified world-model approaches for autonomous driving fall into two flawed paradigms: "predict-and-plan" (tasks remain functionally decoupled despite joint training) and "predict-then-plan" (assumes a stationary environment, producing a multi-second "frozen hallucination" that does not reflect ego-vehicle adjustments made mid-horizon). Neither paradigm allows the planner to continuously update decisions based on evolving imagined observations, causing error accumulation in dynamic traffic scenarios.

Method

Uni-World VLA builds on Show-o (a Phi-1.5-based multimodal LLM) fine-tuned from PWM and uses MagVIT-v2 for visual tokenization into contextual tokens (256×448) and dynamic tokens (128×224). The core contribution is an interleaved autoregressive generation scheme:

\[\hat{d}_{t+k} \sim p_\theta(d_{t+k} \mid \hat{d}_{\le t+k-1}, \hat{a}_{\le t+k-1}), \quad \hat{a}_{t+k} \sim p_\theta(a_{t+k} \mid \hat{d}_{\le t+k}, \hat{a}_{\le t+k-1})\]

At each of \(N=8\) steps (0.5 s intervals, 4 s horizon), the model predicts a future frame's discrete tokens \(\hat{d}_{t+k}\), then immediately queries an action token \(\hat{a}_{t+k}\), which feeds back into the context for the next frame. A bi-directional intra-frame attention with causal cross-frame masking allows spatial coherence within a frame while preserving temporal causality. Depth integration uses Depth Anything 3 to extract monocular depth maps at two resolutions, encoded by separate Context-Depth-Encoder (CDE) and Dynamic-Depth-Encoder (DDE) modules and fused into visual token embeddings via cross-attention:

\[E_{\text{fused},c} = \text{CA}(E_{q,c}, D_{k,c}, D_{v,c}), \quad E_{\text{fused},d} = \text{CA}(E_{q,d}, D_{k,d}, D_{v,d})\]

Training uses a Dynamic Focal Loss \(\mathcal{L}_{\text{dyn}}\) that upweights temporally changing tokens (\(\alpha > \beta\)) plus an L1 trajectory loss \(\mathcal{L}_{\text{traj}}\), combined as \(\mathcal{L} = \lambda_1 \mathcal{L}_{\text{dyn}} + \lambda_2 \mathcal{L}_{\text{traj}}\). Training is two-stage: CDE/DDE pre-training (5 epochs, action-free), then joint fine-tuning (16 epochs on 32 H20 GPUs).

Key Contributions

  • Interleaved prediction-planning: step-wise alternation between future frame token generation and action token generation, forming a closed-loop feedback that continuously conditions planning on newly imagined observations.
  • Unified autoregressive architecture: single model generates mixed visual-action token sequences with intra-frame bidirectional + inter-frame causal attention.
  • Depth integration strategy: cross-attention fusion of monocular depth features (from Depth Anything 3, encoded at two resolutions via CDE/DDE) into historical visual tokens, improving geometric grounding for long-horizon prediction without requiring depth supervision at future frames.
  • Dynamic Focal Loss: spatially weighted cross-entropy that emphasizes temporally varying image regions to reduce static-region dominance in video prediction.

Results

  • NAVSIM closed-loop planning (PDMS): Uni-World VLA achieves 89.4 PDMS (single front camera), outperforming all other single-camera world model methods including PWM (88.1), WoTE (88.3), and SGDrive-IL (87.4). ResWorld (89.0) uses camera+LiDAR.
  • Sub-metrics (best among single-camera): EP = 83.2, TTC = 96.1 (best overall on TTC), NC = 98.7, DAC = 96.7, Comfort = 100.0.
  • Video generation (FVD, 4 s/2 Hz, NAVSIM front): FVD = 141.8, outperforming SVD (227.5), GenAD (184.0), and matching DrivingGPT (142.6).

Limitations

  • Uses only a single front-view camera; multi-view or LiDAR-fused competitors (e.g., ResWorld, DiffusionDrive) provide richer scene context.
  • NAVSIM provides only 2 Hz trajectory logs; 10 Hz trajectories must be supplemented from nuPlan, introducing a cross-dataset dependency.
  • Small batch size (3 per GPU) on 32 H20 GPUs suggests high compute cost relative to model scale.
  • Depth is fused only into historical frames; no explicit depth prediction for future frames, which may limit geometric accuracy of predicted scenes.
  • Evaluation is limited to NAVSIM; generalization to real-world or other benchmarks (e.g., nuScenes, Waymo Open) is not demonstrated.

Relevance to Vision-Language Models

Uni-World VLA is directly relevant to researchers tracking VLMs applied to embodied/autonomous settings, as it extends multimodal LLM architectures (Show-o / Phi-1.5) into a joint visual-generation-and-action-prediction pipeline. The interleaved token generation paradigm—mixing visual and action tokens in a single autoregressive sequence—is a transferable design pattern for VLA models in robotics and embodied AI beyond driving. The depth fusion via cross-attention adds a geometric grounding mechanism to purely RGB-driven VLMs, relevant to any VLM task requiring 3D scene understanding. The work also demonstrates how discrete visual tokenization (MagVIT-v2) enables world modeling within the same LLM backbone used for language-grounded planning.