Skip to content

Disentangled Robot Learning via Separate Forward and Inverse Dynamics Pretraining

🕒 Published (v1): 2026-03-27 17:20 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; disentangled forward/inverse dynamics pretraining resolves VLA misalignment

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DeFI decouples VLA policy learning into separately pretrained forward (video prediction) and inverse (action inference) dynamics modules to resolve the misalignment between 2D image forecasting and 3D action prediction in VLA models. A General Forward Dynamics Model (GFDM) is pretrained via video generation on mixed human/robot data, while a General Inverse Dynamics Model (GIDM) is pretrained self-supervised on action-free video transitions. Both modules are then coupled end-to-end during downstream fine-tuning, achieving state-of-the-art on CALVIN, SimplerEnv, and real-world manipulation.

Problem

Conventional VLA models entangle vision forecasting and action prediction in a single training objective, creating two problems: (1) competing 2D video and 3D action objectives destabilize training; (2) entanglement prevents scaling to the vastly larger pool of action-free human/web videos. Prior "video-as-policy" methods that separate forward dynamics either omit inverse dynamics entirely (VPP) or treat it as an afterthought without a scalable pretraining recipe (Vidar), leaving the inverse dynamics module as a performance bottleneck.

Method

DeFI has two pretraining stages followed by coupled fine-tuning:

GFDM pretraining builds on Stable Video Diffusion (SVD) with a CLIP text encoder. Given current observation \(o_t\) and instruction \(l\), it learns to generate \(H+1\)-frame video clips via latent diffusion: $\(\mathcal{L}_\text{diff}(\theta) = \mathbb{E}_{z^{(0)}, s, \epsilon}\left\|\epsilon - \epsilon_\theta\!\left(z^{(s)}_{t:t+H}, s, c_t\right)\right\|_2^2\)$ where \(c_t = (z_t, f_\text{text}(l))\). At inference only a single denoising step is performed, yielding efficient future latent embeddings without full pixel reconstruction.

GIDM pretraining is fully self-supervised. DINOv2 features \(e_t, e_{t+n}\) are extracted from frame pairs \((o_t, o_{t+n})\) at a ~1-second interval. A spatial-temporal Transformer with causal masks and learnable action queries \(q_a \in \mathbb{R}^{N \times d}\) encodes these alongside T5 instruction embeddings. A VQ-VAE codebook quantizes latent actions \(\tilde{a}_{t \to t+n}^L \to \hat{a}_{t \to t+n}^L\); a Spatial Transformer decoder then reconstructs \(\hat{e}_{t+n}\), trained via MSE against ground-truth \(e_{t+n}\). No action labels are required.

Coupled fine-tuning: GFDM is frozen to preserve its broad dynamics priors. A lightweight MLP aligns GFDM future embeddings to the GIDM's input manifold; a VideoFormer extracts intermediate GFDM features. Fused representations feed a 30M DiT-B diffusion-based action adapter that outputs executable robot commands, trained end-to-end with downstream robot action labels.

Key Contributions

  • Decoupled pretraining paradigm: GFDM learns 2D motion priors from unlabeled video; GIDM learns 3D action representations from video transitions without action labels, each exploiting independent large-scale data sources.
  • Architecture that integrates separately pretrained GFDM and GIDM into a single end-to-end fine-tunable policy with a diffusion action adapter.
  • State-of-the-art results on CALVIN ABC-D (4.51 avg. task length), SimplerEnv-Fractal (51.2%), and real-world deployment (81.3%), with demonstrated data efficiency (surpasses prior SOTA using only ~60% of fine-tuning data).

Results

  • CALVIN ABC-D (multi-view): avg. task length 4.51 vs. VPP 4.33, Seer 4.28, UP-VLA 4.08, GR00T N1 4.01, OpenVLA 3.27.
  • CALVIN ABC-D (third-view): avg. task length 4.05 vs. UniVLA 3.80, CLOVER 3.53.
  • SimplerEnv-Fractal visual matching: 51.2% avg. vs. TraceVLA 42.0%, OpenVLA 27.7%, Octo 16.8%.
  • SimplerEnv-Fractal variant aggregation: 45.4% avg. vs. TraceVLA 45.0%, OpenVLA 39.8%.
  • Real-world Franka: 81.3% vs. OpenVLA 43.8%.
  • Data efficiency (CALVIN): 18% relative improvement over VPP at 10% of fine-tuning data; requires only ~60% of data to exceed the previous SOTA.
  • Ablation: both GFDM and GIDM pretraining individually improve performance; removing either degrades avg. task length.

Limitations

  • Domain shift: when GFDM is pretrained on real-world data (Fractal) and frozen, it cannot generate simulation-style images, causing erroneous inverse dynamics actions in simulation-to-real mismatched scenarios (noted for certain SimplerEnv tasks).
  • GFDM pretraining via full video diffusion is computationally expensive; single-step inference is an approximation that sacrifices generation fidelity for speed.
  • Latent action codes from GIDM are not guaranteed to align with true action manifolds across embodiments; cross-embodiment generalization is not evaluated.
  • Real-world experiments appear limited in task diversity (two RealSense cameras, Franka Panda only; full task set description is truncated in the provided text).

Relevance to Vision-Language Models

DeFI directly extends the VLA paradigm—which builds on VLMs (e.g., PrismaticVLM/SigLIP backbones) by adding action heads—and addresses a core bottleneck of VLA scaling: unlocking the enormous action-free video corpus that VLMs are normally trained on but VLAs cannot exploit due to missing action labels. The self-supervised GIDM pretraining recipe translates VLM-style representation learning (contrastive/reconstruction objectives on internet-scale data) into the action domain, a direction directly relevant to anyone studying how VLMs can be efficiently grounded in physical-world control. The disentangled pretraining strategy also suggests a broader principle for VLM-based agents: perception priors and decision-making priors can be independently scaled and then composed, rather than jointly supervised from scarce labeled demonstrations.