Skip to content

Attention-Free and Lightweight Token Reduction for Efficient Vision-Language Models

🕒 Published (v1): 2026-07-15 06:55 UTC · Source: Arxiv · link

Why this paper was selected

Attention-free token reduction; directly enables VLM deployment on edge/resource-constrained devices

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ALTR is a training-free, plug-and-play visual token reduction framework for VLMs that replaces attention-map-based importance estimation with second-order Rényi entropy scoring and replaces pairwise similarity computation with a stride-based diversity sampling scheme keyed on MLP transformation consistency. It is fully compatible with FlashAttention-style acceleration frameworks and achieves competitive accuracy at aggressive compression ratios (up to 88.9% token reduction).

Problem

Existing token reduction methods for VLMs either (1) rely on materialised attention maps, making them incompatible with modern acceleration kernels like FlashAttention, or (2) use pairwise feature-similarity comparisons with \(O(N^2)\) overhead, undermining the efficiency gains of token reduction. No prior work simultaneously achieves attention-free operation, sub-quadratic overhead, framework compatibility, and strong task accuracy.

Method

ALTR operates between the vision encoder and the LLM projector as a one-shot, two-stage selection operator \(S: X \mapsto X^* \subset X\), \(|X^*| = K \ll N\).

Stage 1 — Importance-Aware Selection via Rényi Entropy. For each token \(x_i \in \mathbb{R}^D\), feature magnitudes are \(\ell_1\)-normalised channel-wise to form a discrete distribution: $\(p_{i,j} = \frac{|x_{i,j}|}{\sum_{k=1}^D |x_{i,k}|}\)$ Second-order Rényi entropy then quantifies activation dispersion: $\(H_2(x_i) = -\log \sum_{j=1}^D p_{i,j}^2\)$ Tokens with high \(H_2\) have more uniform activations across latent components and are considered more informative. The top \(K_1 = \lfloor\lambda K\rfloor\) tokens by \(H_2\) form \(X^{\text{imp}}\).

Stage 2 — Diversity-Aware Selection via Transformation Consistency. For each remaining token \(x_i \in X \setminus X^{\text{imp}}\), the cosine similarity between its pre-MLP representation \(x'_i\) and post-MLP representation \(x''_i = W_2\,\sigma(W_1 x'_i)\) (from the vision encoder's final layer, reusing the forward pass at zero extra cost) is computed: $\(s_i = \cos(x'_i, x''_i) = \frac{x_i^{\prime\top} x_i^{\prime\prime}}{\|x'_i\|_2\,\|x''_i\|_2}\)$ Tokens with similar semantics produce similar \(s_i\); sorting by \(s_i\) groups semantically close tokens adjacently. Stride-based sampling at interval \(t = \lfloor|X_{\text{rem}}|/K_2\rfloor\) then selects \(K_2 = K - K_1\) tokens spanning the entire latent-semantic axis, yielding \(X^{\text{div}}\). The final set is \(X^* = X^{\text{imp}} \cup X^{\text{div}}\).

A scalar ratio hyperparameter \(\lambda \in [0,1]\) governs the importance–diversity balance.

Key Contributions

  • Training-free, attention-free token reduction framework (ALTR) compatible with FlashAttention; operates as a plug-and-play module between vision encoder and LLM.
  • Entropy-based importance criterion using second-order Rényi entropy that runs in linear time and requires no attention map.
  • Transformation-consistency diversity signal derived from the MLP forward pass at negligible additional cost, enabling stride-based diverse sampling with no pairwise comparisons.
  • State-of-the-art average accuracy across four benchmarks on LLaVA-v1.5 at multiple token budgets, with demonstrated generalisation to LLaVA-Next and Qwen2.5-VL.

Results

Evaluated on LLaVA-v1.5-7B (baseline: 576 tokens) vs. SparseVLMs (ICML'25), PyramidDrop (CVPR'25), MustDrop, VisionZip (CVPR'25), VisPruner (ICCV'25) on MME, TextVQA, ScienceQA, GQA. Normalised averages:

  • 192 tokens (↓66.7%): ALTR 98.86% vs. SparseVLMs 98.61%, VisPruner 98.34%, VisionZip 98.01%, MustDrop 96.66%, PyramidDrop 96.97%
  • 128 tokens (↓77.8%): ALTR 97.95% vs. VisPruner 97.74%, SparseVLMs 97.55%, VisionZip 97.18%, MustDrop 95.26%, PyramidDrop 94.00%
  • 64 tokens (↓88.9%): ALTR 94.52% vs. VisPruner 94.34%, VisionZip 94.08%, SparseVLMs 92.04%, MustDrop 91.17%, PyramidDrop 80.20%
  • ALTR is the only method maintaining >94% relative performance at 88.9% token reduction while remaining FlashAttention-compatible.
  • Generalisation experiments on Qwen2.5-VL reported (full numbers truncated in provided text).

Limitations

  • Rényi entropy measures intrinsic token information without access to the text query; tokens may be retained that are irrelevant to the specific downstream instruction.
  • The balance hyperparameter \(\lambda\) requires per-model or per-task tuning; no automatic selection strategy is proposed.
  • Evaluation is primarily on LLaVA-v1.5; generalisation to Qwen2.5-VL and LLaVA-Next is described as preliminary.
  • Performance at very aggressive budgets (64 tokens, ↓88.9%) still incurs ~5–6% absolute average degradation, which may be unacceptable for OCR-heavy or fine-grained tasks.
  • High-resolution and multi-image scenarios are acknowledged as future work; scalability at \(N \gg 576\) (e.g., 2K+ token encoders) is not evaluated.

Relevance to Vision-Language Models

This work directly targets a key bottleneck in deploying modern VLMs — the quadratic cost of processing dense visual token sequences — without retraining or architectural changes, making it immediately applicable to frozen models like InternVL, Qwen2.5-VL, and LLaVA. The incompatibility of prior attention-based pruning with FlashAttention is a practically critical but under-addressed problem; ALTR's entropy+transformation-consistency design offers a principled resolution. For researchers tracking VLMs, this contributes both a deployment-ready efficiency primitive and a new perspective on using information-theoretic token characterisation as a substitute for cross-modal attention signals. It connects to a growing line of work on efficient multimodal inference alongside DyMU, VisionZip, and PyramidDrop, positioning itself as the first method in this family that achieves full acceleration-framework compatibility.