Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

BCV-LR is a two-stage framework for imitation learning from videos (ILV) that extracts latent actions from action-free expert videos via self-supervised encoding and unsupervised world-model dynamics, then aligns those latent actions to the real action space through reward-free online interactions. Without accessing expert actions or environmental rewards, it surpasses state-of-the-art ILV and reward-based RL baselines in sample efficiency across 24/28 visual control tasks.

Problem¶

Existing imitation learning from videos (ILV) methods face a core tension: inverse-RL approaches extract reward signals from videos but require extensive environment exploration, yielding poor sample efficiency; supervised action-prediction approaches achieve better sample efficiency in state-based settings but collapse under complex visual continuous control. No prior work demonstrated that videos alone—without expert actions, rewards, or other supervision—can guide highly sample-efficient visual policy learning.

Method¶

BCV-LR operates in two stages:

Offline pre-training (on action-free expert videos): 1. A self-supervised visual encoder f is pre-trained using task-appropriate objectives—contrastive + self-reconstruction loss (Eq. 1) for discrete Procgen tasks, and a prototype-based temporal association objective (Sinkhorn-Knopp) for partially observable continuous DMControl tasks. 2. A latent action predictor p and world model w are jointly trained via a dynamics-based reconstruction objective (Eq. 2): p predicts a latent action z between consecutive frame embeddings, vector quantization (VQ codebook) discretizes z to prevent trivial copying, and w reconstructs the next latent state from the current state and z.

Online fine-tuning (reward-free environment interactions): 1. A latent action decoder d aligns predicted latent actions to the real action space using collected environmental transitions with real action labels plus a world-model reconstruction regularizer (Eq. 3), providing expert dynamics constraints during adaptation. 2. A latent policy π performs behavior cloning directly on latent actions (Eq. 4), sharing f and d for environment execution. Better π collects higher-quality transitions, which further refine p and d—an iterative self-improvement loop. The final policy is the composition d ∘ π ∘ f.

Key Contributions¶

First demonstration that videos alone (no actions, no rewards) are sufficient for extremely sample-efficient visual policy learning.
A unified offline pre-training + online fine-tuning framework (BCV-LR) combining self-supervised visual encoding, unsupervised latent action extraction via VQ world models, and reward-free behavioral cloning with iterative self-improvement.
Task-adaptive self-supervised objectives: contrastive/reconstruction for discrete control, temporal association for continuous control.
Expert-dynamics regularization during online fine-tuning to prevent catastrophic forgetting and distribution shift between video and agent observations.
Demonstration of multi-task pre-training transferability to unseen tasks.

Results¶

Procgen (16 discrete tasks, 100k interaction budget): BCV-LR mean score 13.8 (video-normalized 0.79) vs. best ILV baseline UPESV 9.0 (0.58), vs. reward-based PPO 2.3 (0.22), vs. video-augmented LAPO 6.8 (0.48). Wins on all 16 tasks vs. other ILV methods; reaches expert-level on Maze, Bigfish, Fruitbot, Starpilot.
DMControl (8 continuous tasks, 100k steps): BCV-LR mean score 604 (video-normalized 0.78) vs. LAIFO 158 (0.20), BCO 336 (0.31), UPESV 18 (0.03), reward-based TACO 310 (0.45), DrQv2 232 (0.34). Wins on 6/8 tasks vs. reward-based RL.
Overall: BCV-LR surpasses baselines on 24/28 tasks; reaches effective policies in as few as 20k–50k environment steps.
Video data efficiency: 50k video transitions sufficient for near-expert performance; 20k still viable on some tasks; degrades at 5k.
Multi-task adaptation: Multi-task BCV-LR pre-trained on 3 tasks generalizes to 2 unseen tasks (Bossfight, Dodgeball) with competitive results.

Limitations¶

Susceptible to covariate shift (a fundamental behavior cloning limitation), restricting performance on complex sequential locomotion tasks.
Performance degrades significantly when video data drops below ~5k transitions—video data efficiency remains an open problem.
Multi-task BCV-LR underperforms single-task BCV-LR on some seen tasks (e.g., Starpilot: 44.3 vs. 54.8), suggesting capacity/interference trade-offs.
Does not extend to real-world settings with visually complex or morphologically diverse agents; evaluated only on simulation benchmarks.
Combining BCV-LR with inverse reward signals to address covariate shift is identified as future work, not yet validated.

Relevance to Agentic AI / LLM Agents¶

BCV-LR addresses the core bottleneck of agent skill acquisition from passive observation—directly relevant to embodied and autonomous agents that must learn from internet video rather than instrumented demonstrations. The iterative self-improvement loop (policy improves data quality, which refines policy) is a primitive form of the bootstrapped, self-supervised agent training paradigm increasingly central to agentic AI. The work's framing of videos as scalable, label-free supervision connects to broader efforts to pre-train generalist agents from web-scale video data. Its multi-task pre-training result suggests a path toward video-foundation models for policy initialization, a key open challenge for generalist LLM-based agents acting in physical or simulated environments.