Skip to content

VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoning

🕒 Published (v1): 2025-03-17 17:59 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VideoMind is a video-language agent that decomposes temporal video reasoning into four specialized roles—Planner, Grounder, Verifier, Answerer—each implemented as a role-specific LoRA adapter on a shared backbone. The Chain-of-LoRA mechanism enables seamless role switching at inference time without maintaining separate full models, achieving state-of-the-art on 15 benchmarks including outperforming GPT-4o on long-video QA with a 2B model.

Problem

Existing VLMs and chain-of-thought methods handle static images well but fail on long videos because they cannot explicitly localize, revisit, or verify specific temporal segments; regression-based temporal grounding models lack textual interpretability; and modular multi-model agents are memory-inefficient and inflexible.

Method

VideoMind builds on Qwen2-VL and defines four roles, each fine-tuned with a distinct LoRA adapter on tailored datasets: 1. Planner: emits a JSON action plan (role sequence) and optionally rephrases the query; trained on 39K samples. 2. Grounder: predicts timestamps via a dedicated timestamp decoder — a transformer encoder over a 4-level temporal feature pyramid built from LLM hidden states, with a classification head (binary focal loss \(L_{cls}\)), a boundary regression head (\(L_{reg}\), L1), and a contrastive loss \(L_{con}\); trained on 210K samples. 3. Verifier: receives top-5 grounder candidates, zooms in (±50% boundary expansion), and scores each with \(\text{Sigmoid}(L_y - L_n)\) using teacher-forced logits; trained on 232K samples. 4. Answerer: the base model (no fine-tuning), processes the verified high-resolution segment.

At inference, all LoRA weights are cached; the planner's JSON output triggers adapter swaps. This "Chain-of-LoRA" requires no extra memory beyond the single backbone plus the LoRA parameter sets.

Key Contributions

  • Role-based agentic workflow with four distinct, complementary roles for grounded video reasoning.
  • Chain-of-LoRA: shared backbone + per-role LoRA adapters enable role switching with zero additional full-model memory overhead.
  • Timestamp decoder: specialized regression/classification heads operating on a temporal feature pyramid extracted from LLM hidden states, improving grounding accuracy beyond language-token approaches.
  • Moment-level test-time search: grounder generates top-5 candidates; verifier re-ranks them via zoom-in, providing a structured inference-time search strategy for video.
  • Strong empirical coverage: evaluated on 15 benchmarks across Grounded VideoQA, Video Temporal Grounding, and General VideoQA.

Results

  • CG-Bench (Grounded VideoQA, avg 27-min videos): VideoMind-2B achieves mIoU 5.94, rec.@IoU 8.50 — surpassing GPT-4o (5.62 / 8.30) and Gemini-1.5-Pro (3.95 / 5.81); VideoMind-7B sets new SOTA at mIoU 7.10, acc.@IoU 4.67.
  • NExT-GQA: VideoMind-7B reaches Acc@GQA 50.2 vs. VideoChat-TPO's 41.2 (7B).
  • ReXTime: VideoMind-7B achieves Acc 74.59 without fine-tuning, vs. fine-tuned VTimeLLM-7B's 57.58.
  • Charades-STA / ActivityNet-Captions: state-of-the-art on temporal grounding benchmarks (exact numbers truncated in provided text).
  • VideoMind-2B outperforms GPT-4o and Gemini-1.5-Pro on MLVU and LVBench (long-video benchmarks).

Limitations

  • The timestamp decoder introduces an additional architectural component (separate from the LoRA adapters) trained on domain-specific grounding data, adding complexity beyond pure adapter switching.
  • The planner relies on three pre-defined reasoning plans; unusual query types may not map cleanly.
  • Training data curation requires repurposing many datasets with synthetic labels (e.g., GPT-4o-mini for query rephrasing), introducing noise.
  • The verifier uses a fixed IoU threshold (0.5) for binary label assignment during training, which may not generalize uniformly across domains.
  • Full system evaluation on very long videos (>1 hour) is not demonstrated; CG-Bench averages 27 minutes.

Relevance to Harnesses / Meta-Harnesses

VideoMind is a concrete instantiation of a multi-role agent harness: a single model backbone orchestrated by a meta-controller (the Planner) that dynamically sequences specialized sub-agents (Grounder → Verifier → Answerer) based on query context. The Chain-of-LoRA mechanism is directly relevant as a lightweight harness pattern — adapter swapping substitutes for process-level tool dispatch, showing how harness-level role coordination can be collapsed into a single-model inference loop. The moment-level search (generate-then-verify) mirrors harness patterns like tool-use-with-reflection or Best-of-N sampling at the sub-task level, offering a template for harnesses where intermediate outputs require structured validation before downstream consumption.