Skip to content

Seeing the Scene Matters: Revealing Forgetting in Video Understanding Models with a Scene-Aware Long-Video Benchmark

🕒 Published (v1): 2026-03-28 12:44 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; scene-aware benchmark reveals catastrophic forgetting in long-video VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Current VLMs suffer from measurable long-context forgetting when required to reason across scenes in long videos: performance drops ~49.5% from clip-level to scene-level tasks. SceneBench is the first benchmark that systematically isolates scene-level understanding across 2,485 videos and 8,903 QA pairs. Scene-RAG, a retrieval-augmented framework built around semantic scene segments, partially mitigates this forgetting with consistent gains over baseline RAG approaches.

Problem

Existing long-video benchmarks either use short isolated clips or conflate temporal granularity with content variability by evaluating different temporal scales on entirely different videos. This makes it impossible to determine whether performance gaps reflect genuine temporal reasoning failures or merely content-domain shifts. No prior benchmark enforces scene-level queries—where clues are spread across multiple non-instantaneous minutes—within the same video, leaving the specific failure mode of long-context scene forgetting unmeasured.

Method

SceneBench constructs 6 tasks over 2,485 YouTube videos (avg. 1,978 s): (1) SceneQA — multi-choice questions whose answer clues span \(\geq 2\) minutes; (2) SceneQA-Audio — SceneQA extended with audio-modal cues; (3) ClipQA — short-clip (~30 s) baseline; (4) Inverse VQA (I-VQA) — given an answer, identify the correct question (forces effect-to-cause reasoning); (5) Comment Prediction — predict whether a viewer comment matches a video; (6) Title Prediction — global semantic abstraction.

Scene-RAG augments an MLLM with three stages: 1. Scene Tiling: groups frames into semantically coherent segments by denoising a visual-similarity sequence via TV-L1 minimization — \(\min_{x \in \mathbb{R}^n} \frac{1}{2}\sum_{t=1}^n (x_t - s_t)^2 + \lambda \sum_{t=2}^n |x_t - x_{t-1}|\) — then thresholds at \(k = \mu_x + \alpha\sigma_x\) to detect salient scene runs. 2. Multimodal Memory Construction: encodes each scene with InternVideo2 (visual) and Qwen-Audio2 (audio transcription/captioning) into a multimodal memory bank. 3. Query Retrieval: decomposes the user query into fine-grained clues with Qwen3-14B, retrieves top-\(k\) scene embeddings by similarity, and fuses them with the MLLM's uniformly sampled frames (ratio 0.5).

Key Contributions

  • SceneBench: first scene-level long-video benchmark with 2,485 videos (avg. 33 min, range 1 min–4+ hr), 8,903 QA pairs, 6 genre categories, and 6 task types; mean SceneQA answer-to-clue temporal distance 262 s (std 310 s).
  • Comprehensive evaluation of 19 MLLMs (proprietary + open-source) revealing a systematic ~49.5% performance collapse from clip-level to scene-level tasks.
  • Scene-RAG: a semantic-scene-aware retrieval-augmented framework that consistently outperforms clip-based Video-RAG on both SceneBench and Video-MME.

Results

  • Open-source MLLM averages on SceneBench: ClipQA 51.73, SceneQA 26.11, SceneQA-Audio 29.08 — SceneQA is 25.62 points below ClipQA (49.5% relative drop).
  • I-VQA mean across open-source models: 35.79; several models at or below random-chance (25%).
  • Gemini 2.5 Pro achieves 80.3 overall average; open-source models average 52.3.
  • Scene-RAG average gain over baseline on SceneBench: ~+1.40% across tasks.
  • Scene-RAG vs. Video-RAG on SceneBench (LongVA 7B): +1.1 pp vs. +0.1 pp over baseline.
  • Scene-RAG on Video-MME overall (LongVA): +11.0 pp vs. Video-RAG's +10.6 pp; (Long-LLaVA): +11.8 pp vs. +10.1 pp; (LLaVA-OneVision): +8.6 pp vs. +7.0 pp.
  • Accuracy degrades monotonically with SceneQA temporal span; Scene-RAG shows +6.1 pp advantage over Video-RAG at long spans (>1,000 s).

Limitations

  • Scene-RAG gains on SceneBench are modest (~+1.40% average), indicating retrieval augmentation alone is insufficient for robust long-context scene reasoning.
  • The benchmark covers only 6 genres (films, vlogs, documentaries, etc.) drawn from YouTube, limiting domain diversity.
  • Scene-RAG uses only 4 input frames per retrieved scene and runs on a single H800 GPU; scalability to very long (multi-hour) videos is untested.
  • SceneQA annotation at multi-minute granularity is acknowledged to be ambiguous; full methodology is deferred to supplementary material.
  • No fine-tuning experiments are reported; Scene-RAG is a purely inference-time augmentation.

Relevance to Vision-Language Models

SceneBench provides a controlled, within-video diagnostic that isolates temporal scene-level forgetting—a failure mode previously conflated with content variation in existing VLM benchmarks. The near-50% accuracy collapse from clip-level to scene-level tasks across 19 models, including frontier proprietary systems, quantifies a fundamental limitation of current MLLM architectures in long-context video grounding. Scene-RAG demonstrates that semantic-scene-aware retrieval (rather than uniform frame sampling) is the right inductive bias for mitigating this forgetting, pointing toward memory-augmented VLM architectures as a necessary research direction. The benchmark's multimodal design (visual + audio) and its inverse-reasoning task (I-VQA) push the field beyond simple temporal localization toward causal, narrative-level understanding.