Skip to content

Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understanding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Vgent is a training-free, graph-based Retrieval-Augmented Generation (RAG) framework for long video understanding that represents video clips as graph nodes connected by shared semantic entities. It adds a structured post-retrieval reasoning step that decomposes queries into binary/numerical sub-questions to filter hard negatives before final generation. Applied to open-source LVLMs (2B–7B), it yields consistent 3.0–5.4% gains over base models and outperforms state-of-the-art video RAG methods by 8.6% on MLVU.

Problem

Long videos (e.g., 30 min ≈ 200K tokens) exceed the context windows of LVLMs, forcing lossy sparse sampling or token compression. Existing video RAG methods treat each clip as an independent document, disrupting temporal dependencies and introducing retrieval noise; ~40% of failure cases occur when the correct clip is retrieved but irrelevant co-retrieved clips distract the model into wrong answers. Proprietary agent-based solutions (VideoAgent, DrVideo) rely on closed-source GPT-4 APIs, limiting flexibility.

Method

Offline Video Graph Construction: The video is segmented into clips of K=64 frames sampled at 1 FPS. An LVLM extracts entity–description pairs from each clip's visual content and subtitles. New entities are merged with existing ones in a global prototype set via text-embedding similarity (threshold τ=0.7, using BAAI/bge-large-en-v1.5); clips sharing an entity are connected by edges, forming a knowledge graph G=(V,E) built once and reused for all queries.

Graph-based Retrieval: The LVLM extracts keywords from the user query. For each keyword, entities with embedding similarity >θ=0.5 are matched; all clips associated with those entities form a candidate set, then re-ranked by multi-field similarity (entity, description, subtitle) to select top-N=20 clips.

Structured Reasoning: The LVLM generates binary (yes/no) or numerical sub-questions targeting presence and quantity of key entities. Each of the top-N clips is evaluated against all sub-questions; clips answering at least one sub-question positively are retained (up to r=5). The LVLM then aggregates structured answers across retained clips into an intermediate reasoning summary.

Multimodal Augmented Generation: The final LVLM call receives both the filtered video clips and the aggregated reasoning summary as multimodal context to produce the answer.

Key Contributions

  • Training-free graph-based video RAG where clips are nodes connected by recurring entities, preserving temporal dependencies for cross-clip retrieval.
  • Structured post-retrieval reasoning via divide-and-conquer sub-question verification, eliminating hard negatives and enabling explicit information aggregation across clips.
  • Query-independent offline graph construction (once per video), enabling 1.73× speedup over Video-RAG on multi-question benchmarks.
  • Demonstrated plug-and-play compatibility with five open-source LVLMs (InternVL2.5, Qwen2-VL, Qwen2.5-VL, LongVU, LLaVA-Video) at 2B–7B scales.

Results

  • MLVU: +3.0–5.4% over base LVLMs; Qwen2.5-VL-3B reaches 70.4%, surpassing its 7B base (68.8%); LongVU-7B: 65.4→70.8%.
  • VideoMME: +4.2% average gain over base models; +5.4% on long-video subset; Qwen2.5-VL-7B: 71.1/65.1 (w/ sub / w/o sub) → 74.3/68.9.
  • LongVideoBench: Qwen2.5-VL-7B: 56.0→59.7 (+3.7%).
  • vs. RAG baselines (MLVU): Qwen2.5-VL-7B + Vgent (72.1%) vs. Video-RAG (63.4%): +8.7%; LLaVA-Video-7B + Vgent (72.5%) vs. Video-RAG (71.3%): +1.2%.
  • Ablation (MLVU, Qwen2.5-VL-7B): NaïveRAG=65.4, GraphRAG=69.5, NaïveRAG+SR=68.6, GraphRAG+SR=72.1; GraphRAG over NaïveRAG: +4.1%; SR over GraphRAG alone: +2.6%.
  • Inference (online, per video-minute): Vgent=3.93 s vs. Video-RAG=20.81 s; offline graph build=20.13 s.

Limitations

  • Offline graph construction (20.13 s/video-minute) adds upfront latency, unsuitable for streaming or single-question scenarios with short videos.
  • Entity merging relies on text-embedding similarity thresholds (Ï„, θ) tuned on the same three benchmarks; generalization to domain-specific videos (medical, sports) is unverified.
  • Structured sub-questions are generated by the same LVLM used for retrieval verification; errors in sub-question generation propagate and cannot be self-corrected.
  • Performance degrades at r=1 (e.g., Count drops from 58.7 to 25.7), indicating the method still requires multi-clip fusion and is brittle when forced to rely on a single clip.
  • Evaluation is restricted to MCQ-format benchmarks; open-ended generation quality is not assessed.

Relevance to Vision-Language Models

Vgent directly addresses a core LVLM bottleneck—context-window overflow on long video—without any fine-tuning, making it a model-agnostic inference-time enhancement compatible with the latest open-source VLMs (Qwen2.5-VL, LLaVA-Video, InternVL2.5). The structured reasoning module highlights a critical finding for the VLM community: retrieval accuracy alone is insufficient, and intermediate chain-of-thought-style verification over retrieved multimodal chunks is necessary to prevent distraction from hard negatives. The graph-RAG paradigm offers a complementary approach to token compression and sparse sampling, the two dominant strategies in long-video VLM research, and demonstrates that structured knowledge representations can substitute for expensive proprietary LLM orchestration pipelines.