Skip to content

ROVER: Recursive Reasoning Over Videos with Vision-Language Models for Embodied Tasks

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ROVER is a recursive inference framework that decomposes long robot-manipulation videos into subtask segments, enabling VLMs to reason over temporally bounded frame sequences rather than entire trajectories. Combined with a subtask-specific sliding context window, ROVER reduces hallucinations on non-expert trajectories and achieves linear time complexity versus the quadratic scaling of full-context baselines. It outperforms GVL and other baselines on task progress estimation, frame-level reasoning, and video QA across 543 RoboCasa videos and real-world OpenX Embodiment benchmarks.

Problem

VLMs excel at single-image understanding but degrade when reasoning over long video sequences: attending to only a few keyframes sacrifices global context, while concatenating all frames into one context is computationally expensive and causes hallucinations—especially when trajectories deviate from expected, optimal behavior. No prior method scales gracefully to arbitrary video lengths while maintaining temporal coherence for embodied manipulation tasks.

Method

ROVER implements a recursive function ROVER(c, Y) where c is a task context and Y is the growing token sequence of interleaved image and text tokens. At each timestep the VLM either retrieves the next frame via a special [next-frame] token or declares a new subtask via "The robot needs to: {subtask}". When a new subtask is declared, a child ROVER call is spawned; its context ϕ(Y) is initialized from the parent's last-frame + new subtask description, and its final frame+text output ψ is appended back to the parent sequence. Within each subtask, a sliding window retains exactly three frames—the subtask's first frame (with its text), the immediately preceding frame, and the incoming frame—capping context size and reducing per-step inference from O(N²) to O(N). The entire framework is deployed as in-context learning with no fine-tuning; Gemini-1.5-Pro is the primary backbone. Ground-truth progress labels for evaluation are computed geometrically: a combined distance metric d_t = y_t + u_t integrating goal-focused Euclidean distances and trajectory-focused distances to the nearest downstream expert keypoint, then inverted and normalized to [0, 1].

Key Contributions

  • ROVER framework: recursive subtask decomposition + sliding context window for VLM video reasoning, reducing inference complexity from O(N²) to O(N).
  • Non-expert trajectory generation protocol: injects random deviations and smooth interpolated recoveries into expert RoboCasa demonstrations to produce trajectories spanning near-expert to fully random behavior.
  • RoboCasa-derived evaluation dataset: 543 videos across 27 kitchen manipulation tasks, annotated with dense geometric value estimates and natural language descriptions.
  • Three-task video reasoning benchmark: frame-level progress estimation (Pearson correlation vs. ground truth), frame-level natural language reasoning (verifiable error rate), and video QA (precision/recall on event occurrence and timing).

Results

  • Task progress estimation: ROVER achieves consistently higher Pearson correlation with ground-truth value estimates on non-expert and partially-completed trajectories; baselines (GVL, TemporalConcat, LocalConcat) diverge sharply as trajectory quality decreases (Figure 3/4).
  • Frame-level reasoning error rate: ROVER's error rate remains low across all trajectory levels; GVL error rate climbs substantially on lower-level (more non-expert) videos; 83% of GVL perception errors occur when >10 frames are in context (Table 1, Figure 5).
  • Video QA: baselines show 20–50% precision with near-100% recall (systematic hallucination of events); ROVER significantly higher accuracy across all task groups; baselines frequently report events before they occur (negative frame-difference distribution in Figure 6).
  • OpenX real-world evaluation: ROVER achieves higher correlation with frame-number-based ground-truth value estimates across 1,000 videos from 50 OXE datasets.
  • Error analysis (100 real-world + 100 simulation videos): ROVER total error rate 26% (real) / 23% (sim) vs. GVL 76% (real) / 57% (sim); ROVER's primary failure mode is incorrect subtask specification (9% real / 7% sim).
  • Ablations: sliding-window-only variant matches or exceeds GVL on short-horizon tasks; recursive decomposition provides additional gains on multi-subtask pick-and-place tasks (Figure 7a). Results hold across Gemini-2.5-Pro-Preview, GPT-4o, and Qwen-2.5-VL-32B.

Limitations

  • Incorrect subtask decomposition (unnecessary or invalid subtasks) fragments reasoning and misaligns it with actual task progression.
  • Evaluated only under in-context learning; fine-tuned variants are left to future work.
  • Subtask boundary specification relies on the VLM generating structured special tokens ([next-frame], "The robot needs to: ..."), which may be brittle without fine-tuning.
  • Ground-truth value function requires known object geometry and expert keypoints from simulation; applicability to fully unstructured real-world tasks is unclear.
  • Dataset is limited to kitchen manipulation in RoboCasa simulation; generalization to other embodied domains (navigation, dexterous manipulation) is untested.

Relevance to Vision-Language Models

ROVER directly addresses a core bottleneck for VLMs in sequential decision-making: the hallucination and context-overload that arise when models must attend to long video sequences. It demonstrates that recursive decomposition—structuring long-horizon tasks as a tree of short-horizon reasoning chains—is an effective inference-time intervention requiring no model fine-tuning, making it immediately applicable to any sufficiently capable VLM backbone. For researchers tracking VLMs, this work connects decomposition-based prompting (a technique explored in static NLP) to the temporal-grounding problem, and provides a rigorous benchmark (progress estimation, NL reasoning, QA) and dataset for measuring VLM temporal understanding in embodied settings. The linear scaling result also has practical implications for deploying frontier VLMs in real-time robot control loops.