VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoning¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

VideoMind is a video-language agent that decomposes temporal video reasoning into four specialized roles (Planner, Grounder, Verifier, Answerer) coordinated through a novel Chain-of-LoRA mechanism, where a single base model dynamically switches between role-specific LoRA adapters at inference time. The 2B model surpasses GPT-4o and Gemini-1.5-Pro on multiple long-video benchmarks while avoiding the memory overhead of maintaining separate full models.

Problem¶

Existing multimodal reasoning approaches—both tool-augmented LLM agents and visual CoT methods—fail on long-form videos because they cannot explicitly localize or revisit specific temporal segments. Regression-based temporal grounding models offer localization but lack textual interpretability, while general video LLMs lack temporal grounding precision. There is no unified, efficient framework that jointly performs temporal grounding, verification, and question answering in a single coherent system.

Method¶

VideoMind builds on Qwen2-VL and defines four roles, each activated via a dedicated LoRA adapter on a shared backbone:

Planner: Outputs a JSON action plan selecting from three reasoning plans (Ground+Verify+Answer, Ground+Verify only, or Answer only) and optionally rephrases queries for better grounding.
Grounder: Predicts temporal moments via a dedicated timestamp decoder. The decoder compresses visual tokens to one per frame via 1D average pooling, projects them with the <REG> token query embedding, fuses them through a 3-layer transformer encoder, then applies a 4-level temporal feature pyramid (Conv1D with stride 2 per level, sequence lengths \(T, T/2, T/4, T/8\) concatenated). Two heads output frame-level foreground scores (binary focal loss) and boundary offsets (L1 loss), with an additional contrastive loss on frame-query cosine similarities.
Verifier: Takes the top-5 grounder candidates, zooms in by expanding boundaries 50% on each side, inserts <SEG-START>/<SEG-END> special tokens at boundary frames, and outputs binary Yes/No judgments. Confidence is computed as \(\text{Sigmoid}(L_y - L_n)\) from teacher-forced token likelihoods; highest-confidence candidate is passed forward.
Answerer: Standard VLM inference on the selected cropped segment (or full video for Plan-3), no fine-tuning.

Chain-of-LoRA: All LoRA adapters are cached in memory; role switching is done by activating the corresponding adapter, incurring no additional memory for full model copies. Training is role-specific on tailored datasets (39K planner, 210K grounder, 232K verifier samples).

Key Contributions¶

Role-based agentic workflow decomposing video reasoning into Planner, Grounder, Verifier, and Answerer with distinct training objectives and datasets.
Chain-of-LoRA: a single unified LMM backbone with multiple cached role-specific LoRA adapters enabling zero-overhead role switching at inference time.
Timestamp decoder with temporal feature pyramid (\(T + T/2 + T/4 + T/8\) parallel prediction) and combined focal + L1 + contrastive losses for accurate moment localization.
Moment-level inference-time search: grounder produces top-5 candidates; verifier selects the best via zoom-in boolean evaluation—a form of test-time scaling for temporal reasoning.
Evaluation across 15 benchmarks spanning Grounded VideoQA, Video Temporal Grounding, and General VideoQA.

Results¶

Grounded VideoQA — CG-Bench (avg. 27-min videos): - VideoMind-2B: mIoU 5.94, rec.@IoU 8.50, acc.@IoU 4.02 — outperforms GPT-4o (mIoU 5.62, rec.@IoU 8.30, acc.@IoU 4.38) and Gemini-1.5-Pro (mIoU 3.95) on grounding metrics; 2B beats all open-source models up to 78B on mIoU - VideoMind-7B: mIoU 7.10, rec.@IoU 9.93, acc.@IoU 4.67 — new state-of-the-art on acc.@IoU

Grounded VideoQA — NExT-GQA: - VideoMind-7B: Acc@GQA 50.2, IoU mIoU 31.4, IoP mIoP 28.2 vs. prior best VideoChat-TPO at 41.2 / 27.7 / 25.5

Grounded VideoQA — ReXTime (zero-shot, no fine-tuning): - VideoMind-7B: Acc 74.59, Acc@IoU 20.20 vs. fine-tuned VTimeLLM-7B at 57.58 / 17.13

State-of-the-art also reported on Charades-STA, ActivityNet-Captions (Video Temporal Grounding), QVHighlights, and 5 General VideoQA benchmarks including MLVU and LVBench (exact numbers truncated in provided text).

Limitations¶

The grounder's timestamp decoder adds architectural complexity exclusive to that role, complicating unified training.
Performance still trails large proprietary models (e.g., GPT-4o long-acc 45.2 vs. VideoMind-7B 38.4 on CG-Bench long-acc), suggesting the QA accuracy gap is not fully closed.
The zoom-in verification strategy expands boundaries by a fixed 50%, which may be suboptimal for moments with highly variable durations.
Training requires role-specific curated datasets (481K samples total) across separate stages, increasing data engineering overhead.
The Chain-of-LoRA design assumes all roles can be adequately parameterized within low-rank adapters on a shared backbone; highly divergent capability requirements could limit expressiveness.

Relevance to Harnesses / Meta-Harnesses¶

VideoMind operationalizes a multi-role agent pipeline inside a single model by routing through LoRA adapters rather than orchestrating separate model instances — a direct architectural answer to the memory and latency costs of multi-model harnesses. The Chain-of-LoRA mechanism is essentially a lightweight meta-harness: a planner role decomposes the task, specialist roles execute subgoals, and control returns to a coordinator, all within one inference stack. For someone building agent harnesses, this is a concrete recipe for collapsing a multi-agent graph into a single-model substrate without sacrificing role specialization, which matters when deploying harnesses at the edge or under memory constraints. It also validates the decompose-ground-verify-answer loop as a reusable scaffold for temporally-grounded reasoning tasks beyond video.