Skip to content

QSVideo: Query-Conditioned Semantic Temporal Retrieval for Video Understanding

🕒 Published (v1): 2026-07-06 00:19 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; query-conditioned temporal retrieval tackles core long-video VLM degradation problem

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

QSVideo is a plug-and-play retrieval framework that addresses VLM degradation on long/streaming video by jointly optimizing relevance, diversity, and temporal coverage when selecting frames under a strict budget. It introduces QSRanker—a two-stage module that rewrites queries into retrieval-friendly form and scores frames along Object, Action, and Location dimensions—combined with QSRetrieval, a greedy diversity-aware search with tailored temporal traversal strategies. On LVBench, an 8B backbone with QSVideo matches or exceeds 72B models using far fewer frames.

Problem

VLM accuracy degrades with video duration because irrelevant frames confuse the language component, yet naively increasing frame count wastes memory without consistent accuracy gains. Existing retrieval methods suffer from three compounding failures: (1) biased relevance—using raw questions (especially multiple-choice) as retrieval queries introduces distractor noise; (2) limited diversity—ranking by relevance alone yields near-duplicate evidence; (3) temporal collapse—selections cluster in a narrow temporal region, missing distant but critical evidence.

Method

QSVideo decouples question understanding from visual ranking via two components:

QSRanker (query-conditioned semantic ranker): - A Question Analyzer (Qwen3-4B LLM) rewrites the original question \(\hat{q}\) into a clean, retrieval-friendly query \(q\) and assigns discrete importance weights \((w_o, w_a, w_l) \in \{1,\ldots,5\}^3\) for Object, Action, and Location dimensions. - A Semantic Ranker (Qwen3-VL-4B-Instruct, fine-tuned via SFT) takes a \((q, v_i)\) pair and outputs structured scores \((o, a, l) \in \{1,\ldots,5\}^3\). - A unified per-frame score is: \(\text{Score}(q, v) = \tilde{w}_o \cdot o + \tilde{w}_a \cdot a + \tilde{w}_l \cdot l\) where weights are softmax-normalized. - The ranker is trained on Video-Ranker-35K (35K video–query pairs sampled from LLaVA-Video-178K), with frame-level semantic labels generated by Qwen-VL-30B. SFT minimizes cross-entropy over the 5-class discrete scores for each dimension.

QSRetrieval (relevance–diversity temporal retrieval): - Greedily selects frames by maximizing: $\(v^* = \arg\max_{v \in A^c}\!\left\{\lambda \cdot \text{Score}(q, v) + (1-\lambda)\min_{v' \in A}\|f(v) - f(v')\|_2\right\}\)$ using L2 distance in visual embedding space for diversity (L2 chosen over cosine for sensitivity to magnitude changes). - Temporal windowing partitions \(\mathcal{V}\) into \(K\) disjoint windows; at most one frame is selected per window to enforce coverage. - Two traversal strategies: long video—iterative anchor-and-expand (global anchor = highest-scoring frame, then local search within radius \(r\), mask visited windows, repeat); streaming video—recency-first traversal from the most recent window backward, using the previous selection as the diversity reference.

Key Contributions

  • QSVideo framework: plug-and-play retrieval pipeline compatible with any video VLM backbone, requiring no model scaling or frame budget expansion.
  • QSRanker: decoupled question-to-query transcription + structured Object/Action/Location relevance scoring, mitigating biased relevance from raw questions.
  • Video-Ranker-35K: automatically annotated frame-level ranking dataset enabling SFT of a lightweight 4B VLM ranker.
  • QSRetrieval: greedy relevance–diversity selection with temporal windowing; distinct traversal strategies for long vs. streaming video.

Results

LVBench (long video, ~67-min average duration): - QSVideo (8B backbone, 32 frames): 49.1% overall, outperforming the base MiniCPM-o 2.6 backbone by 6.9–7.1 pp. - Outperforms SEAL (34B, 16 frames, 45.9%) and GPT-4o (60 frames, 48.9%) using only 8B parameters and 32 frames. - Best sub-72B model on entity recognition (51.1%) and key information retrieval (54.6%). - Competitive with Qwen2-VL-72B (41.3%) and InternVL2.5-78B (43.6%) at a fraction of the model size. - Deep Video Discovery (200B, 200 frames) achieves 74.2%; AdaReTaKe (72B, ≤1024 frames) achieves 64.6%—QSVideo (8B, 32 frames) reaches 49.1%.

StreamingBench: - QSVideo achieves state-of-the-art performance with an efficient 8B backbone (specific numbers truncated in provided text).

Gains are consistent across multiple VLM backbones (plug-and-play verified).

Limitations

  • Temporal windowing enforces exactly one frame per window, which may be suboptimal when evidence clusters legitimately within a single temporal segment.
  • The semantic ranker is fine-tuned only on videos \(\leq 60\) seconds; generalization to truly long videos (\(\geq\) hour-scale) relies on the retrieval strategy, not the ranker's own training distribution.
  • Video-Ranker-35K annotation uses a 30B teacher model (Qwen-VL-30B); label quality is bounded by the teacher's capabilities.
  • The \(\lambda\) trade-off between relevance and diversity and the temporal radius \(r\) are hyperparameters requiring tuning per setting.
  • Computational overhead, though described as modest and accelerable via batched inference, is not fully ablated in the provided text.

Relevance to Vision-Language Models

QSVideo directly attacks a core bottleneck in deploying VLMs for video: the mismatch between fixed frame budgets and variable-length video content, which degrades language grounding. The structured Object/Action/Location relevance formulation is a principled alternative to the binary yes/no VLM-as-ranker paradigm used in SeViLA and FRAG, showing that lightweight fine-tuned rankers can outperform much larger VLM backbones when retrieval is properly conditioned. The plug-and-play design decouples retrieval from VLM architecture, making QSVideo directly composable with next-generation VLM backbones. For researchers tracking VLMs, this work establishes that retrieval-side engineering—rather than context-window scaling—may be the more efficient axis for long-video understanding gains.