Skip to content

Image as a World: Generating Interactive World from Single Image via Panoramic Video Generation

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

IaaW is a three-stage framework that generates controllable, temporally extensible 360° panoramic video from a single image. It combines a diffusion-based visual world model augmented with 3D Spherical RoPE and multi-view composition, plus a fine-tuned VLM (IaaW-VLM) that supplies global and view-specific text prompts. This is claimed to be the first system supporting joint spatial expansion, user-directed viewpoint rotation, and autoregressive scene continuation from a single-view input.

Problem

Prior panoramic video generation methods either require multi-view or full panoramic images as input, generate frames independently (no temporal coherence), or lack interactive viewpoint control and temporal extensibility beyond a fixed clip. Single-image-to-panoramic-world synthesis that is simultaneously controllable, high-fidelity, and infinitely continuable remains unsolved.

Method

The pipeline has three stages built on CogVideoX1.5-5B-I2V fine-tuned on 120K equirectangular videos at 2048×1024:

  1. World Initialization (IaaW-IM): Given a single image, IaaW-VLM generates a global video prompt plus per-view prompts. Multi-View Composition feeds these view-specific prompts with binary spatial masks into parallel MM-DiT attention paths (gated fusion), enabling region-specific semantic control during joint spatial-temporal generation.

  2. World Exploration: A closed-form spherical coordinate transformation applies user-specified pitch/yaw offsets (Δθ, Δφ) to the equirectangular video, producing a rotated view without any learned component.

  3. World Continuation (IaaW-CM): An autoregressive model conditioned on the previous rotated video chunk (not just the last frame) and a new prompt from IaaW-VLM generates the next segment, recursively extending the scene.

3D Spherical RoPE replaces standard spatial RoPE by mapping pixel coordinates to spherical angles (θ, φ) and encoding them with a factorized trigonometric basis jointly with temporal phase, preserving rotational symmetry on the equirectangular sphere.

IaaW-VLM is fine-tuned to generate global captions from full videos and view-specific captions from individual perspective crops, trained on a dataset of {V, P, {V_v, P_v}, {I_v}} tuples extracted from the 120K video corpus.

Key Contributions

  • First unified framework for controllable, temporally infinite 360° world generation from a single image.
  • 3D Spherical RoPE: geometry-aware positional encoding that encodes latitude, longitude, and time jointly to reduce equirectangular distortion.
  • Multi-View Composition: parallel gated attention path in MM-DiT that conditions distinct spatial regions on view-specific text prompts.
  • IaaW-VLM: dual-mode fine-tuned VLM producing both holistic and view-localized prompts to drive semantic alignment across all three stages.
  • Two specialized model variants (IaaW-IM, IaaW-CM) sharing architecture but trained for complementary objectives (scene reconstruction vs. temporal extension).

Results

All metrics: higher is better. Baselines: FETA (Diffusion360 + 360I2V), 360I2V, 4K4DGen.

World Initialization (IaaW-IM vs. FETA): - Subject Consistency (SC-View): 91.8 vs. 89.4 - Aesthetic Quality (AQ-View): 55.9 vs. 49.9 - Video-Text Consistency (VTC-View): 3.72 vs. 3.19

World Continuation (IaaW-CM vs. 360I2V / 4K4DGen): - SC-View: 95.8 vs. 92.5 (360I2V) / 94.1 (4K4DGen) - AQ-View: 49.7 vs. 49.5 / 46.0 - VTC-View: 3.39 vs. 3.25 (VTC not applicable to 4K4DGen)

Full Pipeline (IaaW-IM+CM vs. FETA+360I2V): - SC-View: 91.0 vs. 81.0; SC-All: 90.3 vs. 88.7 - MS-View: 99.1 vs. 98.8; AQ-All: 57.5 vs. 55.9

Ablation on IaaW-IM: - Removing 3D Spherical RoPE drops SC-View 91.8→86.3, AQ-View 55.9→48.8, VTC-View 3.72→3.17. - Removing Multi-View Composition drops VTC-View 3.72→3.24, AQ-View 55.9→49.5.

Limitations

  • IaaW-CM conditions only on the most recent video chunk, causing coherence drift during very long continuations (beyond ~10 rounds / ~1 minute of video).
  • No full video history is maintained; long-range consistency is an open problem shared with the broader video generation field.
  • Some rendered views (top/bottom: sky, floor) inherently score lower on aesthetic quality metrics.
  • Training required 4×A100 GPUs for three weeks; no open-source code released by competing panoramic baselines forced custom re-implementation.

Relevance to Vision-Language Models

IaaW-VLM directly demonstrates a practical fine-tuning pattern for VLMs in generative pipelines: the model is adapted to produce multi-granular textual descriptions (global scene vs. per-view crop) from visual inputs, and these descriptions drive both spatial composition and temporal continuation in a diffusion model. This reflects a broader trend of using VLMs not only for understanding but as semantic controllers for video/world generation. For VLM researchers, the dual-mode captioning training (video-level and frame-crop-level) and its measurable impact on Video-Text Consistency (+0.53 VTC-View over FETA) offer a concrete case study in how VLM prompt quality propagates through downstream generation quality. It also highlights the value of view-aware, spatially grounded prompting as opposed to single global captions.