Skip to content

Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Genesis is a unified world model that jointly generates multi-view driving videos and LiDAR point cloud sequences with spatio-temporal and cross-modal consistency. It couples a DiT-based video diffusion model with a BEV-represented LiDAR generator via shared conditioning, including structured semantic captions produced by a VLM-based DataCrafter module. Genesis achieves state-of-the-art results on nuScenes for both video (FVD 16.95, FID 4.24) and LiDAR (Chamfer 0.611 at 1s) generation.

Problem

Existing driving scene generation methods generate either RGB video or LiDAR point clouds in isolation, with no tight cross-modal alignment. Methods that attempt multimodal generation (e.g., UniScene) rely on intermediate occupancy grid representations that are costly to acquire and introduce information loss. All prior methods use coarse semantic conditioning (BEV maps, generic captions) that limits scene controllability and geometric fidelity, preventing them from constituting a true, semantically grounded world model.

Method

Genesis employs a dual-branch diffusion architecture:

Video branch: A DiT-based spatiotemporal backbone (STDiT-Block-Cam) with a 3D-VAE encoder processes multi-view video. Scene layout S_l = {lane segments, human keypoints, 3D bounding boxes} is projected per-camera via calibrated intrinsics/extrinsics into control maps M_v, injected via cross-attention at every denoising step. Caption embeddings from a T5 encoder are also injected via cross-attention.

LiDAR branch: A BEV autoencoder (Swin Transformer backbone, 8× spatial downsampling) compresses sparse point clouds into BEV latent features. Decoding uses a Swin-based decoder plus NeRF-based rendering with adaptive spatial skipping. A STDiT-Block-L diffusion model then generates these BEV latents conditioned on scene captions, 3D bounding box embeddings, and—critically—BEV features lifted from the camera branch's RGB outputs via LSS (Lift-Splat-Shoot), enforcing direct cross-modal coupling without an occupancy intermediary.

DataCrafter module: Built on Qwen2-VL 7B (LoRA fine-tuned on OmniDrive and CoDALM), it segments multi-view clips via scene boundary detection, scores clips on clarity, structure, and aesthetics, and produces hierarchical structured captions: global scene context (weather, road type, time) plus per-instance tuples (category, bounding box, grounded description). These captions condition both branches.

Training follows a three-stage curriculum: image pretraining → multi-resolution video pretraining + adapter fine-tuning → joint video+LiDAR training with shared conditioning.

Key Contributions

  • Unified dual-branch diffusion architecture jointly generating multi-view video and LiDAR sequences via shared conditioning (shared semantic and layout inputs), without occupancy grid intermediaries.
  • DataCrafter: a VLM-based (Qwen2-VL) structured captioning pipeline producing hierarchical scene-level and instance-level captions for conditioning both modalities.
  • Cross-modal consistency via LSS-lifted BEV features from the video branch injected as ControlNet input to the LiDAR branch.
  • State-of-the-art performance on nuScenes across FVD, FID, and Chamfer distance metrics, with demonstrated downstream utility for 3D detection and BEV segmentation.

Results

Video generation (nuScenes): - Without first-frame conditioning: FVD 83.10, FID 14.90 (vs. DriveDreamer-2: 105.10/25.00; MagicDrive-V2: 94.84/20.91) - With first-frame conditioning: FVD 16.95, FID 4.24 (vs. MiLA: 18.20/3.00; DriveDreamer-2: 55.70/11.20) - Noisy latent setting (6019 clips): FVD 67.87, FID 6.45 (vs. UniScene: 70.52/6.12)

LiDAR generation (Chamfer Distance, lower is better): - Chamfer@1s: 0.611 (vs. HERMES 0.78, ViDAR 1.12, 4D-Occ 1.13) — 21% improvement over prior best - Chamfer@3s: 0.633 (vs. HERMES 1.17) — 45% relative reduction - Using generated images instead of GT: Chamfer@1s 0.634 (minimal degradation)

Downstream tasks: - BEV segmentation domain gap: mIoU 38.01, mAP 27.90 (vs. Cogen: 37.80/27.88; DiVE: 35.96/24.55) - 3D detection (BEVFusion): joint cam+LiDAR generation yields +0.91 mAP / +1.48 NDS over baseline

Ablation: - Removing DataCrafter degrades video FVD from 85.91→117.49, FID from 15.20→22.32 - Adding PoseNet further improves to FVD 83.10, FID 14.90 - Removing reference frame from LiDAR branch increases Chamfer@1s from 0.634→0.668; further removing BEV latent yields 0.661

Limitations

  • High training compute requirements; no model compression explored.
  • Evaluated exclusively on nuScenes; generalization to other domains (robotics, aerial) unvalidated.
  • LiDAR branch trained from scratch (no pretrained priors), increasing data and compute demands.
  • DataCrafter quality filtering may discard valid training samples at the threshold boundary.

Relevance to Vision-Language Models

Genesis demonstrates a concrete production-scale use of VLMs (Qwen2-VL, LoRA fine-tuned) as structured annotation engines rather than as the generative backbone — the DataCrafter pipeline extracts hierarchical scene and instance-level captions that provide semantic grounding for a downstream diffusion model. This is an important design pattern for the VLM-as-world-model line of work: VLMs supply the language priors that bridge modalities and enable controllable generation. The ablation results (FVD 117→83 with DataCrafter) quantify the semantic supervision gap that VLM-derived captions close, making this a useful data point for researchers studying VLM-guided generation and multimodal grounding in embodied/autonomous settings.