Skip to content

Vision Language Models are In-Context Value Learners

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Generative Value Learning (GVL) repurposes frozen VLMs as universal temporal value function estimators by prompting them to autoregressively predict task-completion percentages over shuffled video frames. Shuffling breaks the VLM's implicit monotonic temporal bias, yielding coherent per-frame value predictions without any task- or robot-specific training. GVL generalizes zero-shot across 300+ real-world tasks and enables dataset filtering, success detection, and advantage-weighted policy learning.

Problem

Existing universal value/reward models trained on limited vision-only data lack the semantic and temporal grounding to generalize across diverse robot embodiments and tasks. Naively prompting a VLM with ordered video produces degenerate, monotonically increasing value predictions that ignore actual trajectory quality, making VLMs seemingly unsuitable for value estimation.

Method

GVL frames value estimation as an autoregressive next-token prediction task over a randomly shuffled set of video frames plus a natural-language task description. Three components work together:

  1. Autoregressive batch prediction: The VLM sees all frames and previously emitted values in context, producing v_t = VLM(o_{1..T}; v_{1..t-1}; l_task), enforcing global self-consistency without Bellman training.
  2. Input shuffling: Frame order is randomly permuted (anchored at frame 1 to resolve the arrow-of-time ambiguity). This forces the VLM to attend to semantic content rather than exploit chronological ordering cues, breaking the monotonicity shortcut.
  3. Multi-modal in-context learning: Few-shot examples of (shuffled frames, ground-truth values) are prepended at inference time. Examples can come from the same task, unrelated tasks, or a different embodiment (e.g., human videos), with no model fine-tuning.

The backbone is Gemini-1.5-Pro (long-context). Evaluation uses Value-Order Correlation (VOC), a rank correlation between predicted values and ground-truth chronological order on expert trajectories.

Key Contributions

  • GVL framework: universal value estimation via shuffled-frame autoregressive prompting of a frozen VLM.
  • Value-Order Correlation (VOC): a lightweight, scalable metric for evaluating value model quality without downstream policy rollouts.
  • Demonstration of cross-embodiment and cross-task in-context value learning (human videos improve robot value predictions).
  • Downstream applications requiring no model training: dataset quality ranking, success detection (GVL-SD), and advantage-weighted regression (AWR) for real-world offline RL.

Results

  • OXE (50 datasets, 1000 trajectories): GVL zero-shot VOC heavily right-skewed; significantly outperforms LIV on language-goal datasets where LIV is near-random.
  • ALOHA-250 (250 bimanual tasks, 500 trajectories): Zero-shot median VOC = 0.12 (>60% positive); one-shot median VOC = 0.37 (>90% positive).
  • In-context scaling (ALOHA-13): Average VOC improves monotonically from 0 to 5 in-context examples.
  • Cross-embodiment (ALOHA-13): Human-video one-shot raises VOC from ~0.36 (zero-shot) to ~0.44 vs. ~0.54 for same-embodiment one-shot.
  • Success detection (ALOHA-SIM-6): GVL-SD accuracy 0.71 (zero-shot), 0.75 (one-shot) vs. SuccessVQA 0.62 and SuccessVQA-CoT 0.63.
  • Filtered imitation learning: GVL-SD outperforms SuccessVQA on all 6 simulation tasks; SuccessVQA often hurts performance due to low precision.
  • AWR on real ALOHA (7 tasks): GVL+DP outperforms plain DP on majority of tasks; improvement correlates with VOC score.
  • Ablation (RT-1): GVL VOC = 0.74 vs. single-frame VLM VQA VOC = −0.08.

Limitations

  • Performance degrades on tasks with poor camera viewpoints (e.g., top-down views that occlude task-relevant motion), as in open-drawer and remove-gears.
  • Zero-shot performance on long-horizon bimanual tasks (ALOHA-250) is modest (median VOC 0.12); few-shot examples are needed for reliable quality.
  • Relies on availability of a first anchor frame to disambiguate temporal direction; fully shuffled inference is not possible.
  • Evaluation uses VOC as a proxy metric; direct policy learning experiments are limited to 7 tasks.
  • Computational cost of long-context VLM inference (150+ images per call with 5-shot) is not analyzed.

Relevance to Vision-Language Models

GVL is a concrete demonstration that long-context VLMs possess latent temporal grounding and semantic reasoning sufficient for dense value estimation—a capability not previously extracted from VLMs. The shuffled-frame prompting trick is a broadly transferable insight: it reveals that VLMs' monotonic bias in video understanding is a prompting artifact, not a fundamental architectural limit. For VLM researchers, GVL establishes in-context learning as a viable mechanism for zero-shot robot supervision, complementing work on VLMs as reward models, and raises questions about how pre-training data distribution shapes temporal reasoning. The VOC metric and cross-embodiment ICL findings also offer evaluation and data-efficiency tools relevant to any VLM application requiring temporal understanding.