Skip to content

FlowCut: Rethinking Redundancy via Information Flow for Efficient Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FlowCut is a training-free visual token pruning framework for large vision-language models (LVLMs) that analyzes information flow across ViT layers—rather than relying on single-layer attention scores—to identify and progressively remove redundant tokens. It achieves 94.4%–96.0% of full-model accuracy on LLaVA-1.5-7B with 88.9% token reduction, and delivers a 3.2× prefilling speedup on LLaVA-NeXT-7B with 94.4% reduction.

Problem

Existing visual token pruning methods (FastV, SparseVLM, VisionZip, PDrop) assess redundancy using attention scores from a single layer and a single criterion. This is insufficient: token importance evolves dynamically across layers, different criteria (attention strength, value norm, semantic similarity) can produce contradictory rankings at any given layer, and applying a fixed pruning ratio ignores the layer-wise progression of attention concentration.

Method

FlowCut models information flow as per-token inflow/outflow (argmax of attention rows/columns) across ViT layers and identifies three structural properties: (1) the CLS token acts as a global information relay, making its attention entropy a tractable proxy for overall flow concentration; (2) redundancy emerges progressively as attention entropy decreases layer by layer; (3) single-criterion, single-layer scoring is unstable.

Three modules instantiate these insights:

  • Attention-entropy-adaptive pruning ratio: At each prunable layer, CLS attention entropy is normalized by maximum entropy (log N) to compute r_H; layers with high entropy prune conservatively, low-entropy layers prune aggressively. The number of tokens to prune P is computed as (N−T)/L · (1 − r_H²).
  • Multi-criteria evaluator: Token importance score S = (I_a/Σ + I_s/Σ) × I_d combining CLS attention strength I_a, value-vector cosine similarity to CLS value I_s, and L1 norm of value vectors I_d (information density).
  • Cumulative importance tracking: Running score S_cum^(l) = 0.5·S_cur^(l) + 0.5·S_cum^(l-1) accumulates history across layers, pruning every two layers to smooth noise.

For LLaVA-1.5, pruning completes at the penultimate ViT layer; for LLaVA-NeXT and Qwen2-VL, pruning spans from the vision encoder into the second LLM layer.

Key Contributions

  • Unified information-flow analysis of visual token redundancy in LVLMs, establishing CLS as an effective relay proxy.
  • Empirical demonstration that single-layer, single-criterion redundancy scoring is both insufficient and contradictory across criteria.
  • FlowCut framework combining entropy-adaptive layer-wise pruning ratios, multi-criteria scoring, and cumulative importance aggregation—all training-free.
  • State-of-the-art results across image and video understanding benchmarks, plus demonstrated efficiency gains during training (1.5–1.6× speedup with ≥99.8% accuracy retention).

Results

  • LLaVA-1.5-7B, 64 tokens (88.9% reduction): FlowCut 96.0% vs. VisionZip 94.4% (previous SoTA); +1.6% average over 12 benchmarks.
  • LLaVA-NeXT-7B, 160 tokens (94.4% reduction): FlowCut 91.9% vs. VisionZip 87.6%; +4.3%.
  • Qwen2-VL-7B, 88.9% reduction: FlowCut 56.4 GQA / 72.6 MMB / 81.8 POPE vs. FastV 51.9 / 70.1 / 76.1.
  • Video-LLaVA, 256 tokens (87.5% reduction): 98.6% accuracy vs. VisionZip 94.4%; +4.2%.
  • Inference efficiency (LLaVA-NeXT-7B): 3.0× total, 3.2× prefilling speedup; 4.29 TFLOPs vs. 31.03 TFLOPs for uncompressed baseline.
  • Training efficiency (LLaVA-1.5-7B, 192 tokens): 1.5× speedup, 101.1% of baseline performance; 128 tokens: 1.6×, 99.8%.
  • Ablation confirms all three components (adaptive ratio, cumulative tracking, multi-criteria) independently contribute; full integration achieves best results.

Limitations

  • Evaluation focuses on LLaVA family and Qwen2-VL; generalization to proprietary or closed-source models (GPT-4V, Gemini) is untested.
  • The CLS-token proxy assumption breaks for vision encoders without a CLS token (handled via averaging, but this may be a weaker proxy).
  • Pruning schedule spans vision encoder to LLM layer 2 for high-resolution models; this tight integration may complicate adoption in architectures that separate the two stages differently.
  • No evaluation on tasks requiring dense spatial understanding (e.g., pixel-level grounding, OCR on high-resolution document images) where 88–94% token removal may be more harmful.
  • Computational overhead of computing multi-criteria scores and cumulative updates across layers is not separately quantified.

Relevance to Vision-Language Models

FlowCut directly addresses a central bottleneck in deploying LVLMs at scale: the quadratic cost of large visual token sequences from high-resolution images and video. By grounding pruning in information-flow theory rather than heuristic single-layer scores, it advances the principled understanding of how visual information propagates through ViT encoders—a mechanistic insight relevant to the entire LVLM family. The 3–4× prefilling speedup without fine-tuning lowers the barrier for real-world LVLM deployment and complements orthogonal efficiency work (KV cache compression, quantization). For researchers tracking VLMs, this establishes a new SoTA reference for training-free token reduction and offers a reusable analytical lens (entropy-based flow concentration, CLS-as-relay) applicable to future LVLM architectures.