One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding

🕒 Published (v1): 2026-04-15 17:59 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Why this paper was selected

Extreme compression to one token per frame enables long-video VLMs; NeurIPS 2025

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

XComp addresses the token bottleneck in long video understanding by compressing each video frame down to a single LLM-layer token. It combines supervised learnable-progressive token compression (LP-Comp) with question-conditioned frame selection (QC-Comp) that mitigates LLM position bias via segmented local attention. Fine-tuned from VideoChat-Flash-2B using only 2.5% of its SFT data, XComp raises LVBench accuracy from 42.9% to 46.2%.

Problem

VLMs for long video understanding are constrained by LLM context length: each frame encodes into tens-to-hundreds of tokens, making dense temporal sampling infeasible. Existing token compression methods rely on heuristics (pooling, attention-score selection) that were not trained to consolidate dropped information, causing severe degradation at the extreme compression ratios required for hour-long videos.

Method

LP-Comp (token-level): The LLM layers are supervised end-to-end via a supervised compression tuning (SCT) stage to progressively reduce per-frame token count \(N^{(l)}\) from \(N^{(1)}=16\) at the input down to \(N^{(L)}=1\) at the final layer. The reduction schedule follows a cosine curve: $\(N^{(\ell)} = \left\lceil \frac{N^{(1)}+1}{2} \cos\!\left(\frac{\ell}{L}\pi\right) + \frac{N^{(1)}-1}{2} \right\rceil, \quad \ell=1,\ldots,L.\)$ At each compression step, the prefix tokens of each frame are dropped (suffix-preservation, compatible with causal attention). SCT uses only 2.5% of VideoChat-Flash's SFT data.

QC-Comp (frame-level, inference-time): To select the \(T'\) most question-relevant frames from \(T\) total, the model computes cross-attention scores between question tokens and video tokens. To counteract "lost-in-the-middle" position bias, the full video is split into short segments of 64 frames; attention is computed locally within each segment (segmented local attention), then aggregated per frame. Top-\(k\) frames are retained.

Both stages combine to yield one token per highly selective frame, enabling sampling of up to 2048 frames for videos longer than one hour.

Key Contributions

• LP-Comp: learnable and progressive intra-LLM token compression supervised via a data-efficient SCT stage (2.5% data), achieving a 16× per-frame compression ratio.
• QC-Comp: inference-time, question-conditioned frame selection using segmented local attention to neutralize LLM position bias in long-context attention scoring.
• XComp: a unified VLM combining both, the first to sustain performance improvement as frame count scales (unlike baselines that degrade beyond a threshold).
• Demonstrates that heuristic compression fails at extreme ratios, motivating the learned-compression paradigm.

Results

• LVBench (avg. duration 4101 s): 46.2% vs. VideoChat-Flash-2B baseline 42.9%; vs. InternVL3-2B 42.9% — all at 2B scale.
• LongVideoBench: 59.7% vs. VideoChat-Flash-2B 58.3%, InternVL3-2B 55.4%.
• MLVU: 66.7% vs. VideoChat-Flash-2B 65.7%, InternVL3-2B 64.2%.
• VideoMME (Long): 45.6% vs. VideoChat-Flash-2B 44.9%.
• Competitive with several 7–8B-scale models (e.g., Kangaroo 8B: 39.4% on LVBench, Eagle2.5-8B: 48.2%).
• Performance continues to improve as input frame count increases; VideoChat-Flash degrades after a certain frame count.
• Training cost: ~24 hours on 8× H100 GPUs using only 2.5% of the baseline SFT dataset.

Limitations

• Evaluated only at the 2B model scale; scaling behavior to larger LLMs is not studied.
• The cosine compression schedule and suffix-preservation rule are empirically chosen; theoretical justification is absent.
• QC-Comp relies on LLM attention scores as a relevance proxy, which may not generalize reliably across architectures or question types.
• Frame selection is query-conditioned but static at inference; no iterative or adaptive refinement is explored.
• Ablation data is reported on LVBench only, leaving uncertainty about compression behavior on other benchmarks.

Relevance to Vision-Language Models

XComp directly tackles the per-frame token explosion that prevents VLMs from processing dense temporal streams, a fundamental scalability barrier for video-VLMs. By supervising LLM layers to perform compression — rather than delegating it entirely to the visual encoder or fixed heuristics — it demonstrates that the LLM backbone is an active participant in representation compression, reshaping assumptions about where compression should occur in the VLM stack. The position-bias correction via segmented local attention is broadly applicable to any long-context VLM that uses attention-score-based selection. For researchers tracking video VLMs, this work establishes a new efficiency frontier (one-token-per-frame at 2B scale) and raises the question of how extreme compression interacts with tasks requiring fine-grained spatial reasoning.