Skip to content

RoboTALES: Learning Reasoning-Guided Robot Policies via Task-Aligned Simulated Futures

🕒 Published (v1): 2026-07-07 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RoboTALES is a single-stage framework that aligns video-generative world models with task intent for robot visuomotor control by coupling an LLM planner, a VLM-based reward critic, and an action policy in joint end-to-end training. The hierarchical planner decomposes long-horizon instructions into subtask tokens that condition a Stable Video Diffusion backbone, while a frozen VLM critic steers the generator's latent dynamics toward semantically faithful futures via differentiable policy optimization. On RoboCasa and LIBERO10, the approach consistently outperforms decoupled video-policy baselines, especially on long-horizon tasks.

Problem

Pretrained video generative models used as world models for robot control tend to (1) drift from task intent because they are optimized for visual realism rather than semantic faithfulness, and (2) fail to produce reliably action-conditional rollouts. Existing systems also treat language planning and predictive representations as loosely coupled modules: plans influence action selection but do not shape the latent dynamics of the world model, preventing closed-loop alignment between imagination and action.

Method

RoboTALES trains four tightly coupled components end-to-end in a single stage:

  1. LLM Planner \(F_P\) (Gemini-2.5-Pro): decomposes task instruction \(\tau\) into \(K \in [2,5]\) subtasks \(\{c^{(1)},\ldots,c^{(K)}\}\), concatenated into an augmented plan \(C^* = [\tau\;;\;c^{(1)}\;;\;\ldots\;;\;c^{(K)}]\).

  2. Video Generator \(G_\theta\) (Stable Video Diffusion): predicts future frame rollouts \(\hat{s}_{t+1:t+\Delta}\) conditioned on the current state and \(C^*\) via cross-attention over CLIP embeddings of the plan. Only cross-attention modules and a targeted subset of decoder layers (\(\theta^* \subset \theta\)) are trained.

  3. VLM Critic \(F_\mathcal{R}\) (frozen): scores decoded keyframes against \(\tau\) to produce a scalar reward \(r\). The generator is optimized using a DDPO-style REINFORCE objective over the denoising trajectory, treating each denoising step as an MDP action: $\(J_\text{DDPO}(\theta^*) = \mathbb{E}\!\left[A\cdot\frac{1}{K}\sum_{t\in\mathcal{T}_K}\overline{\ell_\theta(x_{t-1}|x_t,\tau)}\right], \quad A = r - b(\tau)\)$ where \(b(\tau)\) is a per-instruction running-mean baseline.

  4. Action UNet \(\pi_\phi\) (1D diffusion): conditioned on hidden decoder features \(f^{\{l\}}_{G_{\theta^*}}\) and \(C^*\); crucially, no stop-gradient is placed between \(\pi_\phi\) and \(G_{\theta^*}\), so action-level gradients propagate back into the video decoder.

The joint objective is: $\(L_\text{total}(\theta^*,\phi) = L_\text{video}(\theta^*) + \beta\,L_\text{DDPO}(\theta^*) + \gamma\,L_\text{action}(\phi,\theta^*)\)$

Key Contributions

  • Planner-conditioned video generation: subtask tokens from an LLM are injected into the video diffusion model's cross-attention, turning rollout generation into a structured milestone-driven process.
  • Critic-guided representational steering: a frozen VLM provides a language-conditioned reward signal that directly refines the video generator's latent dynamics via differentiable DDPO optimization.
  • Single-stage joint training: action gradients flow back into the video decoder, co-adapting representations so the world model learns to "imagine for acting" rather than purely for visual fidelity.
  • Reasoning-aligned policy learning: the action policy is conditioned on semantically enriched, task-aligned decoder features, yielding smoother and more coherent trajectories on long-horizon tasks.

Results

  • RoboCasa Pick-and-Place: 48% mean success rate, outperforming Video-Policy [35] and the strongest prior baseline [23].
  • RoboCasa turning tasks: 64% mean success rate, surpassing competing methods.
  • RoboCasa pressing tasks: 96% mean success rate, surpassing competing methods.
  • Highest average success across all 34 manipulation tasks on RoboCasa; largest margins on structure-sensitive tasks (doors, drawers) requiring ordered subgoal execution.
  • Also evaluated on LIBERO10; results consistent with RoboCasa trends (full tables truncated in provided text).

Limitations

  • Training requires 6–7 days on 2Ă— NVIDIA A100 80 GB GPUs with batch size 1 per GPU—high compute cost.
  • Each task uses only 50 human demonstrations; scalability to more data-scarce settings is untested.
  • The planner relies on Gemini-2.5-Pro at inference time, adding an LLM API dependency and latency.
  • The VLM critic is frozen throughout training; co-adapting it could improve alignment but is not explored.
  • The text is truncated before full quantitative tables are presented, so complete per-task breakdown and LIBERO10 numbers are not available for review.

Relevance to Vision-Language Models

RoboTALES demonstrates a concrete role for VLMs as frozen reward oracles within a diffusion-based world model, directly steering latent visual dynamics via RL rather than only supervising final actions—an underexplored use of VLMs in embodied AI. The approach shows that VLM semantic scores can be backpropagated through a denoising MDP (via DDPO) to reshape internal video model representations, offering a blueprint for grounding generative visual models in language-conditioned objectives. This connects to the broader VLM literature on reward modeling and process supervision, extending it from text-only or final-answer settings into continuous visuomotor control. For researchers tracking VLMs, the key takeaway is that a frozen VLM critic—without any task-specific fine-tuning—provides a sufficiently dense reward signal to meaningfully reshape the latent space of a video diffusion model.