Skip to content

Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations

🕒 Published (v1): 2025-12-25 09:11 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

BCV-LR is a sample-efficient imitation learning framework that learns control policies from action-free expert videos without any reward signals. It extracts latent actions offline via self-supervised visual encoding and unsupervised dynamics modeling, then aligns these latent actions to the real action space through reward-free online interactions and iterative behavior cloning.

Problem

Existing Imitation Learning from Videos (ILV) approaches either rely on inverse RL (requiring extensive environment exploration to train reward prediction networks, hurting sample efficiency) or supervised action-prediction methods (which degrade on complex continuous visual control). No prior method demonstrated that videos alone—without expert actions or rewards—can support sample-efficient visual policy learning.

Method

BCV-LR operates in two stages:

Offline pre-training (on action-free expert videos): 1. A self-supervised visual encoder f is trained using task-specific objectives: contrastive learning + reconstruction for discrete Procgen tasks; Sinkhorn-Knopp prototype-based temporal association for continuous DMControl tasks. 2. A latent action predictor p and world model w are jointly trained with a dynamics-based unsupervised objective: p predicts a latent action z between consecutive encoded frames, which is vector-quantized (VQ codebook) to prevent trivial solutions, then w reconstructs the next latent state from the current state and z.

Online fine-tuning (with reward-free environment interactions): 1. Real transitions (o_t, a_t, o_{t+1}) are collected by executing the cloned latent policy. p is fine-tuned using both a supervised action prediction loss (cross-entropy/MSE against real actions via decoder d) and the pre-trained world model's dynamics constraint to prevent distribution shift from expert video. 2. A latent policy π is trained via behavior cloning to map latent features to latent actions from expert videos; π shares f and d with the fine-tuner. The improving policy collects better transitions, enabling iterative self-improvement.

Final policy = f (encoder) + π (latent policy) + d (decoder).

Key Contributions

  • First framework demonstrating that videos alone (no action labels, no rewards) suffice for highly sample-efficient visual policy learning.
  • Self-supervised encoder pre-training with domain-adaptive objectives (contrastive/reconstruction for discrete; temporal association for continuous) followed by unsupervised latent action extraction via VQ-regularized world models.
  • Iterative online alignment loop: latent action fine-tuning ↔ behavior cloning policy improvement, using real transitions collected without rewards.
  • Achieves expert-level performance on multiple tasks (Maze, Bigfish, Fruitbot, Starpilot in Procgen; reacher_hard, point_mass_easy in DMControl) within 100k interaction steps.
  • Multi-task pre-training variant (BCV-LR-M) enables cross-domain adaptation to unseen tasks.

Results

  • Procgen (16 discrete tasks, 100k steps): BCV-LR mean score 13.8, video-normalized 0.79 vs. expert. Outperforms all ILV baselines (UPESV: 0.58, LAPO: 0.48, BCO: 0.38) and reward-based RL baselines (PPO: 0.22, LAPO: 0.48). Leads on 16/16 tasks vs. ILV baselines.
  • DMControl (8 continuous tasks, 100k steps): BCV-LR video-normalized mean 0.78 vs. best ILV baseline BCO at 0.31 and best RL baseline TACO at 0.45. Outperforms reward-based RL on 6/8 tasks at 100k steps; learns effective policies at 50k steps (some at 20k).
  • Overall: Surpasses state-of-the-art ILV and RL baselines on 24/28 tasks in terms of sample efficiency.
  • Video data efficiency: 50k expert video transitions sufficient for near-expert performance; degrades noticeably below 5k transitions.
  • Multi-task (BCV-LR-M): Effective policy imitation on both seen (Bigfish, Maze, Starpilot) and unseen (Bossfight, Dodgeball) tasks after pre-training on mixed videos from 3 tasks.

Limitations

  • Subject to covariate shift inherent to behavior cloning, limiting performance on long-horizon sequential tasks (e.g., complex robot locomotion).
  • Performance degrades substantially when video data falls below ~5k transitions; video data efficiency under very low data regimes remains an open problem.
  • Multi-task BCV-LR-M underperforms single-task BCV-LR-S on per-task metrics (e.g., Starpilot: 44.3 vs. 54.8), suggesting capacity/interference trade-offs.
  • Does not yet incorporate inverse reward components; integration with inverse RL to address covariate shift is left for future work.
  • Evaluation limited to simulated benchmarks (Procgen, DMControl, Metaworld); real-world robotics transfer not demonstrated.

Relevance to Agentic AI / LLM Agents

BCV-LR directly addresses the agent learning bottleneck of requiring expensive labeled data or reward engineering, showing that passive video observations—widely available from the internet—can substitute for both. For agentic systems that must acquire skills in new environments cheaply, the offline latent pre-training + minimal online interaction paradigm mirrors the broader trend of foundation-model pre-training followed by lightweight task adaptation. The iterative self-improvement loop (policy → data → better latent alignment → better policy) is structurally analogous to self-play and RLHF loops used in LLM agent training. Multi-task pre-training results also hint at a path toward scalable video-pretraining for generalist agents using internet video corpora.