A Training-Free Framework for Long Video Understanding via Video-Query-Options Similarity¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; training-free long-video understanding via video-query-options similarity

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

MLLMs struggle with hour-long videos because context windows cannot accommodate the required frame tokens. This work proposes a training-free plug-in combining adaptive frame sampling, dynamic resolution allocation, and a query-option similarity mechanism to focus retrieval on task-relevant segments. Applied to LLaVA-Video and Qwen2.5-VL (7B/72B), it achieves state-of-the-art on five long-video benchmarks without any fine-tuning.

Problem¶

Current MLLMs are limited to fixed context lengths, forcing uniform downsampling of long videos and losing critical temporal details. Training-based solutions (context extension, token compression) are costly to retrain for each new architecture. Existing training-free retrieval methods use image-text (CLIP-level) similarity, which fails on action-dependent and temporally structured questions and lacks mechanisms to refine retrieved frame quality.

Method¶

The framework has three components built on a video-text retrieval (VTR) backbone (PerceptionEncoder PE-G/14):

Video-Query-Options Similarity (VQOS): The video is divided into $m$ equal-length 16-second segments. An initial cosine similarity $S_i^0 = \frac{f_{v_i} \cdot f_q}{\|f_{v_i}\|\,\|f_q\|}$ is computed between each segment embedding and the query. The original MLLM then generates $z$ candidate answer hypotheses. Each hypothesis is concatenated with the question to form statements $T = \{T_1,\dots,T_z\}$, and the refined segment score is: $$S_i = \max_{f \in F} \frac{f_{v_i} \cdot f}{\|f_{v_i}\|\,\|f\|}$$ where $F$ collects the VTR encodings of all statements. For multiple-choice questions, ground-truth options bypass generation.

Adaptive Frame Sampling (AFS): The top-$k$ segments by $S_i$ are selected. Segments are sorted and partitioned into $L_1$ priority levels; each level $l$ contains $m_l$ segments each contributing $c_l$ frames ($c_1 > c_2 > \dots > c_{L_1}$), subject to $\sum m_l c_l = N$ total frames. A linear program (PuLP) finds a feasible $\{m_l\}$.

Dynamic Resolution Allocation (DRA): Given a token budget $P$ and $L_2$ resolution levels with scaling factors $\alpha_1 > \dots > \alpha_{L_2}$, each level $i$ gets $\hat{n}_i \approx \frac{P}{L \cdot H_i W_i}$ frames, with higher-similarity frames assigned higher resolution $(H_i, W_i)$. This trades resolution for temporal coverage adaptively.

The pipeline runs $R$ rounds of option generation (parallelizable), then performs a final top-$N$ retrieval with VQOS scores to generate the answer.

Key Contributions¶

VQOS: a hypothesis-then-verify similarity mechanism that fuses query with MLLM-generated candidate answers to produce more discriminative segment relevance scores.
AFS: similarity-guided non-uniform frame sampling that allocates more frames to high-relevance segments under a fixed frame budget.
DRA: similarity-guided spatial resolution allocation that assigns higher resolution to relevant frames under a fixed token budget.
Training-free integration into LLaVA-Video and Qwen2.5-VL at 7B and 72B scales; all three components are complementary to token-compression methods like AdaReTake.

Results¶

LLaVA-Video-7B (Ours-GO): +5.3% average over 5 benchmarks; +8.5% on hour-long video benchmarks (LVBench, VideoEval-Pro) vs. baseline.
Qwen2.5-VL-7B (Ours-GO): +5.0% average; +8.3% on LVBench + VideoEval-Pro.
72B scale: +3.6% (LLaVA-Video) and +3.2% (Qwen2.5-VL) average.
vs. AKS (training-free): Ours-GO lags by 0.4% on shorter-video benchmarks but surpasses by 3.0% on longer ones (LVBench, VideoEval-Pro), based on LLaVA-Video-7B.
vs. AdaReTake (training-free): Ours-GO outperforms by 0.6% on shorter and by 2.8% on longer benchmarks.
Stacking with AdaReTake: +1.3% / +1.0% additional gain over Ours-GO alone (Qwen2.5-VL-7B).
Ablation (Qwen2.5-VL-7B): Each component contributes incrementally — AFS: +0.8%, DRA: +2.1%, generated options: +0.4%, provided options: +1.2% (cumulative $\Delta_\text{avg}$ over 5 metrics).
Achieves overall SOTA among open-source training-free methods across all five benchmarks.

Limitations¶

Option generation adds latency: $R$ rounds of MLLM inference for hypothesis generation increase wall-clock time; parallelization mitigates but does not eliminate this cost.
Ours-PO is not general-purpose: using provided (ground-truth) options as hypotheses only applies to multiple-choice benchmarks, limiting deployment in open-ended settings.
DRA not applicable to fixed-resolution encoders: LLaVA-Video uses a fixed 384×384 input, so DRA was disabled for that model.
VTR model dependency: quality of segment scoring depends on the video-text retrieval backbone (PE-G/14); weaker retrievers would degrade performance.
Gains diminish at 72B scale: larger models already better handle long context, reducing the headroom for retrieval-based improvements.
No evaluation on truly real-world open-ended retrieval outside structured benchmarks with fixed candidate pools.

Relevance to Vision-Language Models¶

This paper directly addresses a fundamental bottleneck for VLMs in video: the mismatch between fixed context windows and long-form temporal content. The VQOS mechanism is notable because it uses the VLM itself to generate query expansions that sharpen retrieval, a self-referential loop absent in prior CLIP-based frame selectors. For researchers tracking VLMs, the training-free framing is practically significant — it decouples architectural innovation from deployment cost, making it immediately applicable to any new MLLM backbone. The complementarity with KV-cache compression (AdaReTake) also suggests a convergence between input-level and token-level efficiency strategies as the field matures toward longer video understanding.