Skip to content

One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

XComp achieves extreme video token compression by reducing each selected frame to a single token at the final LLM layer, enabling 2×–4× denser frame sampling. It combines learnable-and-progressive token compression (LP-Comp) within LLM layers with question-conditioned frame selection (QC-Comp) that uses segmented local attention to avoid "lost-in-the-middle" position bias. Fine-tuned from VideoChat-Flash-2B on only 2.5% of SFT data, XComp surpasses its base model on all evaluated long-video benchmarks.

Problem

Long-video VLMs are constrained by LLM context length: each frame expands into 16–hundreds of tokens, making dense temporal sampling infeasible. Existing token-reduction methods rely on heuristic (training-free) selection or pooling; at extreme compression ratios (16× per frame), these heuristics fail to transfer dropped token information into retained tokens, causing accuracy degradation. An additional problem is LLM positional bias in long sequences ("lost in the middle"), which corrupts attention-based frame relevance scoring.

Method

LP-Comp (Learnable and Progressive Token Compression): Supervised compression tuning (SCT) trains the LLM layers end-to-end to progressively reduce per-frame tokens from N⁽¹⁾=16 to N⁽ᴸ⁾=1 following a cosine schedule across all L transformer layers. Critically, only suffix tokens of each frame are preserved at each step, compatible with causal attention (later tokens absorb context from earlier ones). SCT uses just 2.5% of VideoChat-Flash's SFT data.

QC-Comp (Question-Conditioned Frame Compression): At inference time, the long video is split into fixed-size segments (64 frames each). Within each segment, local attention scores between question tokens and video tokens are computed independently in parallel. Each frame's relevance score is the average cross-attention from any question token to any of its video tokens. Top-k frames are retained. Segmented local attention replaces global attention to prevent position bias from suppressing mid-video frames.

The combined pipeline: (1) dense frame sampling → (2) LP-Comp reduces each frame to 1 token across LLM layers → (3) QC-Comp removes low-relevance frames, yielding one token per highly selective frame.

Key Contributions

  • LP-Comp: supervised, cosine-scheduled progressive token compression inside LLM layers, achieving 16× per-frame compression without heuristics.
  • Suffix-preservation strategy: dropping prefix tokens per frame to align compression with causal attention direction.
  • QC-Comp: query-conditioned frame selection using segmented local attention to correct LLM positional bias over long sequences.
  • Data-efficient SCT: full framework learned with only 2.5% of VideoChat-Flash's fine-tuning data.
  • Demonstrated generalizability to LLaVA-Next-Video in addition to VideoChat-Flash.

Results

  • LVBench: XComp 46.2% vs. VideoChat-Flash-2B 42.9% (baseline), InternVL3-2B 42.9%; comparable to some 7B models (e.g., Kangaroo-8B 39.4%, mPLUG-Owl3-7B 43.5%).
  • LongVideoBench: 59.7 vs. VideoChat-Flash-2B 58.3 and InternVL3-2B 55.4.
  • MLVU: 66.7 vs. VideoChat-Flash-2B 65.7 and InternVL3-2B 64.2.
  • VideoMME (Long): 45.6 vs. VideoChat-Flash-2B 44.9.
  • Ablation (LVBench, 1024 frames): learnable progressive compression 44.3 vs. heuristic compression 41.1 vs. no-compression baseline 41.8.
  • Suffix-preservation (44.3 LVBench / 65.2 MLVU) beats uniform-drop (43.6 / 64.1).
  • Segmented local attention (QC-Comp) outperforms global attention on LongVideoBench (59.7 vs 59.5), MLVU (66.7 vs 66.3), VideoMME Long (45.7 vs 44.8).
  • LLaVA-Next-Video + XComp: VideoMME Long 63.2 vs. baseline 62.7; LVBench 43.1 vs. baseline 40.6.

Limitations

  • Experiments limited to 2B-scale VLMs (VideoChat-Flash-2B, LLaVA-Next-Video); scaling to larger models is future work.
  • QC-Comp frame selection is inference-only; joint training of frame selection with LLM layers could yield further gains.
  • Compression strategy was not integrated into VLM pre-training, only fine-tuning.
  • Evaluation restricted to four benchmarks; behavior on other video tasks (e.g., dense captioning, temporal grounding) is untested.

Relevance to Vision-Language Models

XComp directly addresses a core scalability bottleneck in VLMs: the quadratic cost of processing hundreds of visual tokens per frame in long-context LLMs. By showing that LLM layers themselves can be trained as compression modules—not just the visual encoder/projector—it opens a new design axis distinct from context-extension or external resampler approaches. The segmented local attention strategy for mitigating "lost-in-the-middle" bias is transferable to any VLM with attention-based token selection. This work advances the line of efficient long-video VLMs alongside methods like VideoChat-Flash and LLaVA-Next-Video, with practical impact for any application requiring temporal understanding over minute-to-hour scale videos.