Skip to content

Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Video-RAG is a training-free, plug-and-play pipeline that augments any Large Video-Language Model (LVLM) with visually-aligned auxiliary texts—extracted via OCR, ASR, and object detection—retrieved via RAG to address query-relevant information needs. By adding only ~2K auxiliary tokens per query, it consistently improves long video understanding without fine-tuning or proprietary APIs. Applied to LLaVA-Video-72B, it surpasses GPT-4o and Gemini-1.5-Pro on Video-MME.

Problem

Existing LVLMs fail on long videos because their context windows cannot accommodate the high frame counts needed for full coverage. Fine-tuning long-context LVLMs requires massive compute and labeled data; GPT-based agent pipelines depend on expensive proprietary models (GPT-4o) and multi-turn interactions that are slow (~20 days and ~$2000 to run Video-MME with VideoAgent). Prior RAG-for-video approaches process video as plain text, losing visual alignment.

Method

Video-RAG operates in three phases without any model training:

  1. Query Decouple: The LVLM is prompted (text-only, no frames) to emit a JSON retrieval request specifying what ASR content to retrieve (Rasr), which physical entities to detect (Rdet), and what spatial properties to extract (Rtype: location, count, relation).

  2. Auxiliary Text Generation & Retrieval (parallel):

  3. OCR database: EasyOCR extracts text from sampled frames; Contriever encodes chunks into FAISS index.
  4. ASR database: Whisper transcribes audio; same Contriever+FAISS pipeline.
  5. DET database: CLIP similarity selects query-relevant keyframes (threshold t=0.3); APE open-vocabulary detector extracts object categories and bounding boxes; a scene graph converts raw detections into natural-language object-location, object-counting, and relative-relation strings.

  6. Retrieval: Contriever encodes the concatenated query+request; FAISS cosine similarity > t selects relevant text chunks from OCR/ASR databases; Rtype selects which scene-graph outputs to include.

  7. Integration & Generation: Retrieved auxiliary texts (OCR, ASR, DET) are concatenated chronologically and appended to the query; this combined text plus sampled video frames is fed into the LVLM in a single forward pass.

Key Contributions

  • Training-free RAG pipeline compatible with any LVLM as a drop-in augmentation with no architectural changes.
  • Three visually-aligned auxiliary modalities (OCR, ASR, object detection with scene-graph post-processing) extracted entirely with open-source tools (EasyOCR, Whisper, APE, Contriever, FAISS).
  • Single-turn parallel retrieval with minimal overhead: ~2K extra tokens and ~5 s additional inference time per query on a 7B model using 8 GB extra GPU memory.
  • Open-source 72B-scale performance exceeding GPT-4o and Gemini-1.5-Pro on Video-MME without any proprietary API.

Results

  • Video-MME: Average +2.8% across 7 LVLMs (vs. with-subtitle baseline); LLaVA-Video-72B reaches 77.4% overall vs. GPT-4o 77.2% and Gemini-1.5-Pro 81.3%. Largest individual gain: LongVA-7B +6.0% overall.
  • MLVU: LLaVA-Video-7B +1.6% (72.4%), LLaVA-Video-72B +0.7% (73.8%, new SOTA at time of submission).
  • LongVideoBench: LLaVA-Video-72B reaches 65.4%, surpassing Gemini-1.5-Pro (64.0%) by 1.4%; 7B variant +2.1% (58.7%).
  • Ablation (Long-LLaVA-7B, Video-MME): no aux texts 52.0% → DET only 52.9% → +OCR 54.3% → +ASR 61.6% → all+RAG 62.1%; removing RAG retrieval (random tokens) drops from 62.1% to 59.1%.
  • Threshold t=0.3 optimal: balances 1.9K tokens and 11 s inference; t=0.0 (all tokens, 3.6K) marginally lower at 62.0%.

Limitations

  • Performance is upper-bounded by the quality of external tool outputs (EasyOCR accuracy, Whisper transcription, APE detection precision); no adaptive mechanism to compensate for tool failures.
  • Object detection is restricted to CLIP-sensitive physical entities (filtered by spaCy), excluding abstract concepts—may miss queries requiring higher-level semantic understanding.
  • Retrieval is purely lexical/embedding-based (Contriever similarity); temporal ordering of retrieved chunks is reconstructed heuristically rather than attending to fine-grained temporal structure.
  • Frame sampling strategy is inherited from the base LVLM (no adaptive selection); gains are larger with fewer frames, suggesting suboptimality at high frame rates.
  • Evaluation omits interleaved input format for LongVideoBench, potentially underestimating the baseline.

Relevance to Vision-Language Models

Video-RAG directly addresses the cross-modality alignment bottleneck in LVLMs: it shows that retrieving text-form grounding signals (OCR, ASR, object graphs) improves LVLM attention toward query-relevant keyframes, as confirmed by Grad-CAM and t-SNE visualizations. The plug-and-play design is significant because it decouples long-video capability from model scale—any LVLM immediately benefits without retraining, lowering the barrier to deploying open-source models at proprietary parity. For researchers tracking VLMs, the work highlights that modality gap, not just context length, is a core bottleneck, and that structured textual grounding is a lightweight yet effective bridge.