Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Tsung-Yi Lin + Yen-Chen Lin; ICLR 2026; fine-tuning video models for visuomotor control

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Cosmos Policy fine-tunes the NVIDIA Cosmos-Predict2-2B video diffusion model into a robot manipulation policy through a single post-training stage with no architectural changes. It encodes robot actions, proprioception, future states, and value estimates as latent frames injected directly into the video model's diffusion sequence, enabling both direct imitation and model-based planning. It achieves state-of-the-art results on LIBERO (98.5%), RoboCasa (67.1%), and real-world bimanual ALOHA tasks (93.6% average).

Problem

Prior video-model-based robot policies either require multi-stage training pipelines with separately designed action modules (e.g., inverse dynamics models, separate action diffusers), or train unified video-action models from scratch that forgo pretrained spatiotemporal priors. VLA models based on vision-language backbones similarly miss the temporal causality and implicit physics captured by video generation models. Neither paradigm cleanly unifies policy, world model, and value function without added architectural complexity.

Method

Latent Frame Injection. Cosmos Policy adapts Cosmos-Predict2-2B (a latent video diffusion transformer operating over Wan2.1 VAE tokens of shape \((1+T') \times H' \times W' \times 16\)) by inserting new modality latent frames directly into the video diffusion sequence. Robot proprioception, action chunks, future proprioception, and scalar value estimates are encoded into \(H' \times W' \times 16\) latent volumes by normalizing and tiling the values, then interleaved with camera image frames. A typical 11-frame sequence encodes \((s, a, s', V(s'))\) covering current state, action, future state, and Monte Carlo return.

Joint Training Objective. A 50/50 split of demonstration vs. rollout data drives three training modes via conditioning masks: policy learning \(p(a, s', V(s') \mid s)\), world model \(p(s', V(s') \mid s, a)\), and value function \(p(V(s') \mid s, a, s')\). Auxiliary targets (predicting future state and value alongside actions) are shown empirically to improve policy performance.

Model-Based Planning. After collecting on-policy rollouts, a separate "planning model" checkpoint is fine-tuned with 90% weight on world model/value function objectives. At test time, best-of-\(N\) sampling draws \(N\) action proposals from the original "policy model," queries the planning model three times per action (world model) and five times per future state (value function) for 15 total value predictions, and selects the action with highest ensemble value via "majority mean" aggregation. Parallel decoding is used for direct policy deployment; autoregressive decoding is used during planning for higher-quality predictions.

Key Contributions

• Latent frame injection: a no-architectural-change mechanism to extend any latent video diffusion model with arbitrary robot modalities (proprioception, actions, values, extra camera views).
• Single-stage fine-tuning: eliminates multi-stage pipelines by unifying policy, world model, and value function within the native diffusion objective.
• Dual-checkpoint planning: decouples the policy model and planning model to leverage on-policy rollout data for refined world modeling without degrading the policy.
• State-of-the-art benchmarks: highest reported scores on LIBERO (98.5%), RoboCasa (67.1% with only 50 human demos vs. competitors using 1050), and real-world bimanual ALOHA (93.6% average).
• Model-based vs. model-free ablation: empirical comparison of world-model + state value vs. Q-value (model-free) planning variants.

Results

• LIBERO (simulation): 98.5% average success rate across four suites — outperforms π0.5, OpenVLA-OFT, CogVLA, UniVLA, DP-VLA, GR00T-N1.5, UVA, Video Policy, and diffusion policies trained from scratch.
• RoboCasa (simulation): 67.1% average success rate using only 50 human demos (vs. baselines using up to 1050); surpasses prior SOTA including FLARE and GR00T-N1.5.
• Real-world ALOHA (bimanual): 93.6% average across four challenging tasks; highest among all evaluated policies.
• Model-based planning improvement: +12.5 percentage point average task completion on two challenging real-world tasks relative to direct policy deployment.
• Auxiliary supervision ablation: removing future state and value prediction targets from policy training degrades performance (exact delta not excerpted but stated to be significant).

Limitations

• Compute at inference: best-of-\(N\) planning requires \(N\) GPUs running in parallel; ensemble calls (15 value predictions per proposal) substantially increase inference cost.
• Full action chunk execution: to keep planning cost tractable, the full 50-step action chunk is deployed without receding-horizon replanning, reducing mid-chunk correction ability.
• Rollout data dependency for planning: planning benefits only emerge after collecting on-policy rollouts; the base model trained purely on demonstrations cannot plan effectively due to narrow state-action coverage.
• No input history: the current formulation conditions only on the current timestep observation, not a history of frames, which may limit performance on partially observable or temporally aliased states.
• Scalar value encoding: values are encoded as tiled latent frames, a heuristic representation whose capacity for precise value regression is not theoretically grounded.

Relevance to Vision-Language Models

Cosmos Policy directly benchmarks against VLA models (π0.5, OpenVLA-OFT, CogVLA, UniVLA, GR00T-N1.5) and argues that video-pretrained backbones capturing spatiotemporal dynamics are superior foundations for low-level control compared to image-text-pretrained VLMs — a direct empirical challenge to the dominant VLA paradigm. The latent frame injection technique generalizes to any modality, suggesting a path for augmenting VLMs with temporal, proprioceptive, or reward signal modalities without architectural redesign. For VLM researchers, this work establishes a concrete performance ceiling that language-grounded robot policies must clear, and demonstrates that the pretraining data distribution (video vs. image-text) materially affects downstream control capability.

🔗 Related in this collection

• CogVLA: Cognition-Aligned Vision-Language-Action Models via Instruction-Driven Routing & Sparsification