One Video, One World: Turning Monocular Video into Physical 4D Scenes¶

🕒 Published (v1): 2026-06-30 09:16 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

Saining Zhang; first training-free 4D mesh scene from single video; highly novel

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

OVOW is the first training-free system that reconstructs instance-level, simulation-ready 4D mesh scenes from a single monocular video. It composes a VLM for scene decomposition, feed-forward mesh reconstruction, metric-scale/pose recovery, and physics-grounded assembly into a unified pipeline without task-specific training. It also establishes the first benchmark for structured Video-to-4D evaluation beyond visual fidelity.

Problem¶

Current 4D reconstruction produces implicit fields, Gaussian primitives, or point clouds that lack watertight topology, instance separation, and standardized physical interfaces (e.g., URDF) required by physics simulators (MuJoCo, Isaac Gym, PyBullet) and embodied AI. No existing method produces simulation-ready 4D mesh assets from video. The field also lacks paired video-to-4D datasets and evaluation benchmarks that measure geometric correctness, instance separation, and physical plausibility rather than rendering fidelity alone.

Method¶

OVOW is a four-stage, training-free pipeline:

VLM-Guided Scene Decomposition — Uniformly samples \(N_{\text{key}}=3\) keyframes; uses Qwen3-VL for open-vocabulary object discovery, unique instance naming, and motion classification (static / rigid / deformable). SAM3 then produces per-frame binary masks via text-prompted segmentation. Instances below \(\tau_{\text{area}}=200\) px are discarded.
Instance-Level Mesh Reconstruction — For static/rigid objects: selects the anchor frame (largest mask area), optionally inpaints occluded regions with FLUX.2 conditioned on the VLM description, then generates a canonical mesh via Hi3DGen. For deformable objects: zoom-normalized masked video is processed by Motion324, producing a topology-consistent mesh sequence. Metric scale is initialized by back-projecting VGGT dense depth and refined in the next stage.
Spatiotemporal Pose and Deformation Recovery — An iterative render–match–optimize loop (\(N_{\text{iter}}\!=\!3\)) jointly refines scale and orientation per object: FoundationPose estimates 6-DoF pose, RoMa v2 finds dense correspondences, and L-BFGS-B optimizes scale with trimmed residuals. Bidirectional pose tracking across all frames follows, with fallback re-registration. Static objects are pose-locked; deformable objects decouple global rigid trajectory from per-vertex displacement.
Physics-Grounded Scene Assembly — Ground plane is estimated via iterative RANSAC with a camera-up prior, then aligned to XY. Contact projection corrects penetration/floating via gravity-axis shifts. Inter-object contact uses KD-tree nearest-surface queries to handle stacking. HDR environment lighting is recovered from the input video for photorealistic rendering.

Key Contributions¶

Task & method: First training-free pipeline for structured Video-to-4D reconstruction, producing instance-level, watertight, simulation-ready mesh scenes with unified vertex-deformation motion modeling (no category priors or skeleton rigging).
Benchmark: First benchmark for structured Video-to-4D evaluation (OVOW-3D-Scene-Bench and OVOW-4D-Scene-Bench, 120 scenes each), scoring geometric correctness, instance separation, and physical plausibility beyond visual fidelity.
Data engines: Two complementary pipelines — OVOW converts real video into paired video↔4D scene data, and an asset-based Blender pipeline generates synthetic benchmark scenes — supplying the paired supervision this task previously lacked.

Results¶

OVOW-3D-Scene-Bench (120 static scenes): Best Scene-IoU OBB (0.218), Object-IoU (0.190), PL (5.70), N-CLIP (1.87); comparable runtime to feed-forward baselines (272 s/frame).
OVOW-4D-Scene-Bench (120 dynamic scenes): Leads every quality metric — 0.440 Scene-IoU OBB, 0.210 Object-IoU, 2.90 PL, 1.43 N-CLIP.
Runtime: 3.35 s/frame on the 4D benchmark, one to two orders of magnitude faster than baselines (103–788 s/frame).
Baselines compared: CAST, SAM3D, VIGA, MIDI, SceneGen, TabletopGen.
Physical stability: Downstream physics simulation confirms reconstructed scenes remain stable under gravity.
Scalability: Performance degrades gracefully with more objects; runtime grows only modestly.

Limitations¶

Quality drops on crowded scenes due to increased mutual occlusion among instances.
Fully training-free — inherits failure modes from each component model (VLM misclassification, segmentation errors, tracking failures).
Small/motion-blurred instances below 200 px mask area are discarded.
Benchmark scenes are synthetic (rendered in Blender from Objaverse-OA assets), not real-world 4D ground truth.
Pose tracking may fail and fall back to expensive re-registration on difficult frames.
No explicit handling of articulated objects with joint topology (relies entirely on vertex deformation).

Relevance to Vision-Language Models¶

OVOW places a VLM (Qwen3-VL) at the very front of its pipeline for open-vocabulary scene decomposition — discovering, labeling, and motion-classifying every object from just three keyframes. This shows how VLMs can bridge high-level semantic understanding and physics-ready 3D reconstruction, bootstrapping structured world models from casual monocular video. For the VLM tracking community, the paper demonstrates a concrete, high-impact application where language-guided instance understanding directly unlocks downstream physical simulation and embodied AI use cases.