Skip to content

ViPSim: Collaborating Visual and Parameter Spaces for Consistent Long-Horizon Embodied World Models

🕒 Published (v1): 2026-06-27 08:18 UTC · Source: Arxiv · link

Why this paper was selected

Consistent world models for evaluating VLA systems; critical embodied AI infrastructure

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ViPSim bridges the representation gap between low-dimensional actions and high-dimensional video synthesis in embodied world models by collaborating a pixel-aligned Visual Space (action maps, Plücker camera embeddings, depth, robot masks) with a numerical Parameter Space (quaternion-reparameterized actions, camera matrices). It is backbone-agnostic (UNet & DiT), achieves drift-free long-horizon rollouts, and shows emergent generalization to deformable objects, OOD scenes, and cross-embodiment settings.

Problem

Action-conditioned Embodied World Models (EWMs) suffer from accumulated trajectory drift and inconsistent robot-object interactions during long-horizon generation. The root cause is a representation gap between low-dimensional numerical action/parameter inputs and high-dimensional visual synthesis: existing architectures lack rigorous geometric correspondence between raw actions and pixel-level scene dynamics, leading to cumulative error in autoregressive rollouts, especially in high-contact regions where robot morphology and scene geometry are tightly coupled.

Method

ViPSim operates as a chunk-based autoregressive video diffusion process (chunk length \(L=16\)). It introduces a dual-space collaboration mechanism:

  • Visual Space (dense structural grounding): Projects raw EEF poses into 2D pixel-action maps via calibrated camera intrinsics/extrinsics (unit circles at projected coordinates, 50px radius). Camera pose is represented per-pixel via Plücker embeddings \(\mathbf{p}_{u_t,v_t,t} = (\mathbf{o}_t \times \mathbf{d}_{u_t,v_t,t},\; \mathbf{d}_{u_t,v_t,t}) \in \mathbb{R}^6\) for localized geometric interpretation. Depth sequences (Video Depth Anything) and robot morphological masks (detection + SAM2 with dynamic template update) provide scene geometry and embodiment grounding. These are VAE-encoded, reduced via \(1\times1\) conv, and fused into the denoising backbone's input latent.

  • Parameter Space (numerical driving): Re-parameterizes EEF states to \([x,y,z,q_x,q_y,q_z,q_w,o] \in \mathbb{R}^8\) (quaternion, gripper), concatenates with flattened camera matrices \([R_t|T_t] \in \mathbb{R}^{12}\), and encodes via a lightweight Physics Encoder (stacked linear + nonlinear layers). The resulting embedding \(\mathbf{E}_{\text{phys}}\) is combined with CLIP and text embeddings into \(\mathbf{C}_{\text{global}}\) and injected via cross-attention (spatial latent as query, \(\mathbf{C}_{\text{global}}\) as key/value).

The framework is implemented on two backbones: DynamiCrafter (UNet, noise prediction loss) and Wan2.2-TI2V-5B (DiT, flow-matching loss). All geometric priors are pre-computed offline during training; online inference runs at 1.918s/chunk.

Key Contributions

  • Dual-space collaboration that synchronizes geometric/morphological (Visual Space) and numerical (Parameter Space) priors across complementary domains, establishing rigorous action-to-pixel correspondence.
  • Backbone-agnostic integration demonstrated on both UNet (DynamiCrafter) and DiT (Wan2.2-TI2V-5B) architectures.
  • Drift-free long-horizon generation suppressing cumulative trajectory errors, validated over extended autoregressive rollouts.
  • Emergent generalization to deformable object manipulation (cloth folding), out-of-distribution scenes, and cross-embodiment transfer (Agibot-trained → Droid actions).

Results

  • Perceptual quality (AgiBotWorld-Beta): ViPSim(DiT) achieves PSNR 20.35, SSIM 0.80, LPIPS 0.19 vs. EnerVerse-AC's 17.93/0.73/0.25 — consistent improvement across all metrics.
  • EWMBench evaluation: ViPSim(DiT) scores 2.4754 (Motion Sum), 2.1790 (Semantic Sum), 0.9153 (Scene Consistency), 5.5697 (Overall) — all above EnerVerse-AC (2.0504/2.0804/0.9078/5.0386).
  • Statistical robustness (15 evaluations): ViPSim(DiT) shows low variance (e.g., PSNR \(20.3543 \pm 0.0831\), DYN \(0.7534 \pm 0.0184\)).
  • Inference speed: 1.918s per chunk (end-to-end, including feature extraction).
  • Qualitative: preserves object identity and scene detail under occlusion; maintains structural integrity with deformable objects where baselines show chromatic/textural drift.

Limitations

  • Visual Space priors (depth, masks) require pre-extraction — depth/mask for the future \(L\) frames are populated from the current frame as a surrogate, which may become stale over very long horizons.
  • Evaluated on a single dataset (AgiBotWorld-Beta, 10 tasks, 1K trajectories) — scale and diversity are modest.
  • Requires calibrated camera intrinsics/extrinsics per sequence, which may not be available in all real-world deployment settings.
  • Mask segmentation pipeline (detection + SAM2 + dynamic template update) introduces dependency on external models and may fail for unseen robot morphologies or extreme occlusions.

Relevance to Vision-Language Models

ViPSim directly addresses a critical bottleneck for VLA evaluation: the lack of reliable, high-fidelity embodied simulators for long-horizon policy assessment. By producing geometrically consistent, action-conditioned video rollouts, it enables safe, risk-free benchmarking of VLA models without hardware cost or physical hazards. The framework also opens the door to model-based predictive control for VLA systems, where accurate visual prediction of action consequences is essential.