Skip to content

LongVQUBench: Benchmarking Long-Term Video Quality Understanding of Vision-Language Models

🕒 Published (v1): 2026-07-01 15:40 UTC · Source: Arxiv · link

Why this paper was selected

Long-video quality understanding benchmark for LVLMs; temporal distortion evaluation gap

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LongVQUBench is a new benchmark for evaluating long-term video quality understanding (LVQU) in large vision-language models (LVLMs), covering 1,200+ videos from minutes to two hours in length. It introduces three hierarchically structured evaluation levels—local, cross-event, and global—plus a needle distortion question-answering (NDQA) paradigm for fine-grained perceptual sensitivity testing. Experiments on 14 state-of-the-art LVLMs reveal consistent performance degradation as video length and reasoning depth increase.

Problem

Existing video quality benchmarks focus on short clips with isolated distortions (e.g., KoNViD-1k, Q-Bench-Video), and LVLM-oriented long-video benchmarks (e.g., LongVideoBench, MLVU) assess semantic understanding rather than perceptual fidelity. No benchmark jointly evaluates temporal coherence, cumulative degradation, and hierarchical perceptual reasoning over long-form videos—leaving a critical gap in assessing whether LVLMs can reason about video quality over extended durations.

Method

LongVQUBench aggregates 1,200 videos (several minutes to ~2 hours) from LongVideoBench, MLVU, and LongVideoReason, covering films, documentaries, surveillance, egocentric recordings, and CGI. High-quality source videos are segmented into 15-second clips; controlled spatial (14 types) and temporal (4 types) distortions at three intensity levels are applied according to three configurations before clips are reconstructed into full-length videos. The benchmark defines three progressive evaluation levels: - LQU (Local Event Quality Understanding): single distorted clip (5–20 s); probes detection, localization, classification, severity assessment, and open reasoning. - CQR (Cross-Event Quality Reasoning): 2–4 distorted clips; probes comparison, cumulative effect, integration, temporal relations, and open reasoning. - GQU (Global Quality Understanding): all clips with systematic distortion; probes stability, dominant factor identification, trend assessment, overall evaluation, and open reasoning.

Overlaid across all three levels is the NDQA paradigm, in which spatial or temporal artifacts are sparsely embedded and models must detect/interpret them via multiple-choice (Yes/No, What, Which, How) and open-ended questions. QA pairs are constructed in two stages: automated generation from distortion metadata followed by multi-expert review with consensus resolution; open-ended responses are scored for relevance and completeness with GPT-based evaluation.

Key Contributions

  • LongVQUBench: the first benchmark explicitly designed for long-term video quality understanding in LVLMs, with 1,200+ videos (minutes to ~2 hours) and 1,500 MCQ + open-ended QA pairs.
  • A three-level hierarchical evaluation framework (LQU → CQR → GQU) that progressively tests localized detection, multi-event integration, and holistic perceptual judgment.
  • The NDQA paradigm: sparsely embedded spatial/temporal distortions at controlled intensities to probe fine-grained perceptual sensitivity independently of semantic content.
  • A large-scale empirical study of 14 LVLMs (3 proprietary, 11 open-source) under zero-shot conditions, exposing fundamental limitations in temporal localization, perceptual attribution, and global reasoning.

Results

The paper text is truncated before the quantitative results table (Section 4.1 is incomplete). Based on the text available: - All 14 evaluated LVLMs show consistent performance degradation as reasoning depth increases from LQU → CQR → GQU. - Models exhibit limited capacity for long-range temporal integration and perceptual attribution over extended durations. - Specific accuracy numbers, model rankings, and per-level breakdowns are not present in the provided text.

Limitations

  • Dataset construction relies on applying controlled distortions to high-quality source clips, which may not fully replicate organically degraded real-world content.
  • The majority of source videos (888/1,200) are high quality before distortion injection, so naturally low-quality long videos are underrepresented (\(n=94\) low, \(n=218\) medium).
  • Open-ended question scoring depends on GPT-based evaluation, inheriting biases and inconsistencies of that judge model.
  • The benchmark covers video-level perceptual quality but does not assess audio-visual quality interactions.
  • Evaluation is zero-shot only; fine-tuned or retrieval-augmented settings are not explored.

Relevance to Vision-Language Models

LongVQUBench directly stress-tests a blind spot of current LVLMs: their inability to maintain coherent perceptual reasoning over long temporal contexts, which is distinct from the semantic understanding tasks on which most models are trained and evaluated. By constructing a hierarchical, distortion-aware evaluation that scales from seconds-long local events to hour-long holistic judgments, it establishes a principled diagnostic for the temporal modeling and cross-modal perceptual integration capabilities of LVLMs. The NDQA paradigm is particularly relevant as a needle-in-a-haystack analogue for perceptual quality, complementing existing "needle-in-a-haystack" semantic retrieval benchmarks. This benchmark will drive development of LVLMs with stronger long-context visual attention and explicit temporal quality reasoning mechanisms.