Skip to content

GEVRM: Goal-Expressive Video Generation Model For Robust Visual Manipulation

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GEVRM applies the classical Internal Model Control (IMC) principle to vision-language-action (VLA) robot manipulation by using a text-guided video diffusion model to generate expressive future goal states and prototypical contrastive learning to align current and goal state representations. This closed-loop design allows the robot policy to implicitly infer and suppress external environmental perturbations (lighting jitter, occlusions, noise). The approach achieves state-of-the-art on both standard and perturbed CALVIN benchmarks.

Problem

Existing VLA models are brittle under real-world deployment perturbations (lighting variation, video stream noise, image shift/occlusion) because they treat visual observations at face value without any mechanism to distinguish environmental disturbance from task-relevant state change. Prior goal-conditioned video planning methods also generate temporally and spatially inconsistent future frames, limiting policy quality.

Method

GEVRM decomposes robot decision-making into two hierarchical levels: behavior planning and action prediction.

Robot Behavior Planner: A text-guided video diffusion transformer (DiT) encodes language instructions via a frozen T5 encoder and compresses robot image state sequences through cascaded 2D VAE (8×8 spatial) + 3D VAE (4× temporal) to reduce compute. The DiT is fine-tuned using Rectified Flow (rather than DDPM) for faster training. A random mask mechanism (75% weight on unmasking the first h frames, 25% on supplementary strategies) forces the model to learn physical world dynamics and temporal coherence.

State Alignment via Prototypical Contrastive Learning: ResNet-34 encoders extract features from the current state x and generated goal state x_goal. Both are L2-normalized and projected onto a unit sphere. Cluster assignment probabilities are computed via dot products with a learnable normalized prototype matrix E, and targets are sharpened with Sinkhorn-Knopp to avoid collapse. A cross-entropy objective maximizes mutual prediction accuracy between source and target assignments, aligning current-state representations toward goal-state representations and thereby encoding perturbation-distinguishing information.

Goal-Guided Diffusion Policy: A diffusion policy conditioned on the latent embeddings (z, ẑ) of the current and goal states uses DDPM-style denoising to predict 7-DoF robot actions (6D end-effector pose + gripper). The total loss is J = J_φ + λJ_ψ (policy loss + contrastive alignment loss). At test time, the behavior planner re-samples goals every L_test steps in a closed loop.

Key Contributions

  • First integration of the classical IMC principle into a VLA framework, yielding a closed-loop visuomotor control architecture.
  • Text-guided video diffusion planner with cascaded 2D+3D VAE compression and random masking for highly expressive, temporally coherent goal generation.
  • Prototypical contrastive state alignment that enables implicit perturbation estimation without requiring explicit perturbation labels.
  • State-of-the-art on CALVIN ABC→D (standard and five perturbed conditions) and demonstrated real-robot deployment.

Results

  • Goal generation (BridgeData): FID 35.70 vs. SuSIE 114.79 / AVDC 246.45; FVD 4.16 vs. AVDC 22.89; SSIM 0.89 vs. SuSIE 0.71.
  • Goal generation (CALVIN): FID 94.47 vs. GR-1 236.75 / SuSIE 214.14; FVD 3.80 vs. GR-1 12.83.
  • Action execution (standard, CALVIN ABC→D): 1-instruction success 0.92 vs. SuSIE 0.87 / GR-1 0.75; 5-instruction chain 0.26 (ties SuSIE at 0.26).
  • Action execution (perturbed, average over 5 perturbation types): Average achieved task length 1.62 vs. GR-1 1.44 / RoboFlamingo 1.31 / SuSIE 1.11.
  • Ablation: Removing state alignment or VAE fine-tuning each substantially degrades performance on CALVIN D; λ=1 optimal but results are robust across tested λ values.

Limitations

  • Evaluated only on CALVIN (simulation) and BridgeData; real-robot experiments are limited to appendix.
  • Static third-view RGB camera only; no proprioception or gripper-view input, which simplifies the setup but may not reflect full deployment complexity.
  • Goal re-sampling interval L_test and number of sampled goals M are fixed hyperparameters with unclear sensitivity analysis beyond ablations shown.
  • The cascaded 2D+3D VAE adds inference latency; inference speed vs. simpler baselines is not quantified in the main text.
  • Authors acknowledge the need for more general, high-quality video generation models for diverse real-world manipulation tasks.

Relevance to Vision-Language Models

GEVRM sits at the intersection of VLMs and embodied AI: it uses language (via T5) to condition a video diffusion model that serves as a visual "internal model," directly relevant to researchers studying how VLMs ground language to visual future states for robot control. The prototypical contrastive alignment mechanism offers a generalizable technique for making VLA systems robust to distribution shift—a standing challenge for deployed VLMs in physical environments. The work extends the line of video-as-planning-medium approaches (UniPi, SuSIE, GR-1) by explicitly tackling perturbation robustness, an underexplored axis in VLM-based robot research. For VLM researchers, the decoupled training pipeline (text-video pairs for planning, unlabeled play data for policy) is notable as it reduces dependence on expensive language-annotated robot action datasets.