Skip to content

Goal-Driven Reward by Video Diffusion Models for Reinforcement Learning

🕒 Published (v1): 2025-11-30 16:22 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GenReward is a reward framework for visual RL that uses pretrained video diffusion models (CogVideoX) to generate goal-conditioned videos and derive two complementary intrinsic reward signals—video-level trajectory alignment and frame-level goal-reaching via forward-backward representations—eliminating handcrafted reward engineering. It is the first reward framework combining generative model output, demo-free operation, and action-aware reward computation. Experiments on Meta-World and Distracting Control Suite show significant gains over prior reward models.

Problem

Designing programmatic reward functions for RL requires domain expertise and doesn't generalize across tasks. Prior approaches using VLMs or diffusion models either require expert demonstrations, lack action awareness, or fail to exploit generated videos as explicit goal-driven rewards, limiting performance on complex manipulation tasks.

Method

GenReward operates in three stages built on top of DreamerV3:

  1. VDM Adaptation: CogVideoX-5B-I2V (a video DiT with 3D VAE) is finetuned on domain-specific manipulation videos (RT-1, Bridge, RLBench) using standard diffusion loss conditioned on text + image prompts to enable goal-conditioned video generation.

  2. Video-Level Reward: The finetuned 3D causal VAE encodes both the agent's 16-frame historical observations and the generated goal video into latent vectors; cosine similarity between them is used as r_video, computed every 128 environment steps.

  3. Frame-Level Reward: CLIP selects the most task-relevant frame from the generated goal video as a goal image I*, encoded by DINOv3 into a 384-dim feature. A forward-backward (FB) network—F(s,a,z)^⊤ B(s') approximating discounted state occupancy—is trained for 100k steps on the replay buffer with the goal embedding ψ(I*) as target, then frozen. The FB inner product F(s,a,ψ(I*))^⊤ B(ψ(I*)) serves as r_FB. Final reward: r_gen = α·r_video + β·r_FB + r_env with α=1e-2, β=1e-5.

Key Contributions

  • Novel two-level reward framework exploiting pretrained video diffusion models without handcrafted reward design or expert demonstrations.
  • Action-aware frame-level reward via forward-backward representations, measuring probability of reaching a CLIP-selected goal state from a given state-action pair.
  • First reward framework combining all three properties: demo-free, generative-model-based, and action-aware (Table 1 comparison).
  • Demonstrated robustness to VDM hallucinations (object teleportation) and cross-domain video transfer (RT-1, Bridge, RLBench sources all improve performance).

Results

  • Meta-World dense reward (vs. DreamerV3 Dense Reward baseline):
  • Pick Out of Hole: 193 → 582
  • Bin Picking: 398 → 822
  • Shelf Place: 154 → 814
  • Meta-World sparse reward: GenReward consistently outperforms RoboCLIP, Diffusion Reward, TADPoLe, and Dense Reward on Bin Picking.
  • DCS/Adroit (model-free DrQv2 backbone):
  • Walker Walk: Dense=640, RoboCLIP=695, Diffusion Reward=28, GenReward=782
  • Hopper Stand: Dense=646, RoboCLIP=589, Diffusion Reward=776, GenReward=821
  • Adroit Door (success rate %): Dense=60, RoboCLIP=70, Diffusion Reward=0, GenReward=90
  • MW Reach: Dense=25, RoboCLIP=45, Diffusion Reward=60, GenReward=85
  • Hallucination robustness: GenReward w/ hallucination (574) vs. GenReward (796) vs. Dense Reward (454) on Pick Place.
  • Training time (1M steps, Meta-World Pick Place): Dense=102h, RoboCLIP=119h, Diffusion Reward=109h, GenReward=106h.

Limitations

  • Additional computational overhead from computing video-level and FB rewards during training (though wall-clock time is competitive with baselines at 1M steps).
  • VDM finetuning requires 7 days on 16 A100 GPUs; limits scalability of adaptation to new domains.
  • CLIP-based frame selection can fail to identify the most task-relevant goal frame (e.g., selects pre-grasp over successful-grasp frame in RT-1 Pick Apple), introducing imperfect goal specification.
  • Reward weighting hyperparameters (α, β) are sensitive and require tuning; too-large or too-small values degrade performance substantially.
  • Evaluation limited to robotic manipulation benchmarks (Meta-World, DCS, Adroit); generalization to broader RL domains (e.g., navigation, games) is untested.

Relevance to Agentic AI / LLM Agents

GenReward's approach of extracting structured goal representations from pretrained generative models directly parallels how LLM agents use world knowledge from pretraining to guide task-directed behavior without task-specific supervision. The forward-backward representation for measuring goal-reaching probability is a generalizable mechanism for any agent that must evaluate progress toward an underspecified goal—relevant to reward shaping in tool-using or multi-step LLM agents. The demo-free, language-conditioned reward framework contributes to the broader agenda of building agents that can bootstrap from natural-language task descriptions rather than engineered reward signals. This work also highlights how video generation models can serve as a world model proxy within agent training loops, an increasingly important capability as agentic systems tackle embodied and long-horizon tasks.