Skip to content

VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; VideoMind: chain-of-LoRA agent for temporally grounded video reasoning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VideoMind is a video-language agent that decomposes temporal video reasoning into four specialized roles (Planner, Grounder, Verifier, Answerer) coordinated through a novel Chain-of-LoRA mechanism, where a single base model dynamically switches between role-specific LoRA adapters at inference time. The 2B model surpasses GPT-4o and Gemini-1.5-Pro on multiple long-video benchmarks while avoiding the memory overhead of maintaining separate full models.

Problem

Existing multimodal reasoning approaches—both tool-augmented LLM agents and visual CoT methods—fail on long-form videos because they cannot explicitly localize or revisit specific temporal segments. Regression-based temporal grounding models offer localization but lack textual interpretability, while general video LLMs lack temporal grounding precision. There is no unified, efficient framework that jointly performs temporal grounding, verification, and question answering in a single coherent system.

Method

VideoMind builds on Qwen2-VL and defines four roles, each activated via a dedicated LoRA adapter on a shared backbone:

  1. Planner: Outputs a JSON action plan selecting from three reasoning plans (Ground+Verify+Answer, Ground+Verify only, or Answer only) and optionally rephrases queries for better grounding.
  2. Grounder: Predicts temporal moments via a dedicated timestamp decoder. The decoder compresses visual tokens to one per frame via 1D average pooling, projects them with the <REG> token query embedding, fuses them through a 3-layer transformer encoder, then applies a 4-level temporal feature pyramid (Conv1D with stride 2 per level, sequence lengths \(T, T/2, T/4, T/8\) concatenated). Two heads output frame-level foreground scores (binary focal loss) and boundary offsets (L1 loss), with an additional contrastive loss on frame-query cosine similarities.
  3. Verifier: Takes the top-5 grounder candidates, zooms in by expanding boundaries 50% on each side, inserts <SEG-START>/<SEG-END> special tokens at boundary frames, and outputs binary Yes/No judgments. Confidence is computed as \(\text{Sigmoid}(L_y - L_n)\) from teacher-forced token likelihoods; highest-confidence candidate is passed forward.
  4. Answerer: Standard VLM inference on the selected cropped segment (or full video for Plan-3), no fine-tuning.

Chain-of-LoRA: All LoRA adapters are cached in memory; role switching is done by activating the corresponding adapter, incurring no additional memory for full model copies. Training is role-specific on tailored datasets (39K planner, 210K grounder, 232K verifier samples).

Key Contributions

  • Role-based agentic workflow decomposing video reasoning into Planner, Grounder, Verifier, and Answerer with distinct training objectives and datasets.
  • Chain-of-LoRA: a single unified LMM backbone with multiple cached role-specific LoRA adapters enabling zero-overhead role switching at inference time.
  • Timestamp decoder with temporal feature pyramid (\(T + T/2 + T/4 + T/8\) parallel prediction) and combined focal + L1 + contrastive losses for accurate moment localization.
  • Moment-level inference-time search: grounder produces top-5 candidates; verifier selects the best via zoom-in boolean evaluation—a form of test-time scaling for temporal reasoning.
  • Evaluation across 15 benchmarks spanning Grounded VideoQA, Video Temporal Grounding, and General VideoQA.

Results

Grounded VideoQA — CG-Bench (avg. 27-min videos): - VideoMind-2B: mIoU 5.94, rec.@IoU 8.50, acc.@IoU 4.02 — outperforms GPT-4o (mIoU 5.62, rec.@IoU 8.30, acc.@IoU 4.38) and Gemini-1.5-Pro (mIoU 3.95) on grounding metrics; 2B beats all open-source models up to 78B on mIoU - VideoMind-7B: mIoU 7.10, rec.@IoU 9.93, acc.@IoU 4.67 — new state-of-the-art on acc.@IoU

Grounded VideoQA — NExT-GQA: - VideoMind-7B: Acc@GQA 50.2, IoU mIoU 31.4, IoP mIoP 28.2 vs. prior best VideoChat-TPO at 41.2 / 27.7 / 25.5

Grounded VideoQA — ReXTime (zero-shot, no fine-tuning): - VideoMind-7B: Acc 74.59, Acc@IoU 20.20 vs. fine-tuned VTimeLLM-7B at 57.58 / 17.13

  • State-of-the-art also reported on Charades-STA, ActivityNet-Captions (Video Temporal Grounding), QVHighlights, and 5 General VideoQA benchmarks including MLVU and LVBench (exact numbers truncated in provided text).

Limitations

  • The grounder's timestamp decoder adds architectural complexity exclusive to that role, complicating unified training.
  • Performance still trails large proprietary models (e.g., GPT-4o long-acc 45.2 vs. VideoMind-7B 38.4 on CG-Bench long-acc), suggesting the QA accuracy gap is not fully closed.
  • The zoom-in verification strategy expands boundaries by a fixed 50%, which may be suboptimal for moments with highly variable durations.
  • Training requires role-specific curated datasets (481K samples total) across separate stages, increasing data engineering overhead.
  • The Chain-of-LoRA design assumes all roles can be adequately parameterized within low-rank adapters on a shared backbone; highly divergent capability requirements could limit expressiveness.

Relevance to Agentic AI / LLM Agents

VideoMind is a direct instantiation of multi-role agentic workflows applied to a modality (video) where agent-based decomposition is not merely beneficial but necessary—sequential temporal localization, verification, and answering cannot be collapsed into a single forward pass without sacrificing grounding fidelity. The Chain-of-LoRA mechanism is a lightweight alternative to tool-calling or model-routing architectures, showing that inference-time role switching via adapter activation is a viable paradigm for building specialized yet unified agents. The verifier-as-inference-time-search component connects to the broader o1-style test-time scaling literature, extending it from token-level to segment-level search in a structured domain. This work is directly relevant to researchers building video-grounded agents, multi-modal reasoning pipelines, and efficient multi-agent systems that must balance specialization with parameter efficiency.