Combating Textual Noise and Redundancy: Entropy-Aware Dense Visual Token Pruning

🕒 Published (v1): 2026-07-02 17:50 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; entropy-aware visual token pruning under dense/fine-grained instructions — practical efficiency

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EADP identifies two root causes of failure in visual token pruning for VLMs: dense cross-modal scoring is corrupted by a "noise floor" from dispersed functional words and punctuation, and standard Top-K selection causes feature fragmentation by over-concentrating on a few discriminative local peaks. EADP resolves both via entropy-guided textual denoising followed by reformulating token selection as a facility location submodular maximization problem, achieving state-of-the-art accuracy-efficiency trade-offs across multiple VLM families.

Problem

Existing visual token pruning methods degrade severely under strict budgets, dense instructions, and fine-grained or negative queries. Two specific failure modes are identified: (1) naive dense cross-modal scoring aggregates similarity from functional words and punctuation, which exhibit uniformly dispersed spatial similarity distributions, creating an indiscriminate noise floor that drowns out the sparse activations of semantic entity tokens; (2) even with perfect scores, greedy Top-K selection over-fits to redundant local maxima (e.g., repeatedly sampling a single body part of a target object), wasting the token budget while leaving other semantic parts unrepresented.

Method

EADP is a plug-and-play, training-free two-stage framework operating between the vision encoder and the LLM.

Stage 1 — Dispersion-Aware Dense Scoring.
For each non-EOS CLIP text token \(t_i\), cosine similarities to all \(N\) projected visual tokens are computed, yielding \(C \in \mathbb{R}^{M \times N}\). Each row is converted to a probability distribution via temperature-scaled softmax and its Shannon entropy is computed: $\(h_i = -\sum_{j=1}^N p_{i,j} \log p_{i,j}\)$ Tokens with \(h_i\) above the \(q\)-quantile threshold \(\lambda = Q_q(H)\) (high entropy = dispersed/noisy tokens) are discarded. Retained tokens receive entropy-inverse weights \(\alpha_i = \text{softmax}(-h_i/\gamma)\), yielding a dense guidance score \(S^D_j = \sum_{i \in \mathcal{T}'} \alpha_i \cdot c_{i,j}\). This is fused with the global EOS-based score \(S^G\) as \(s_j^I = \mu s_j^G + (1-\mu) s_j^D\), followed by min-max normalization.

Stage 2 — Spatial Smoothing, Score Polarization, and Submodular Selection.
The 1D score map \(S^I\) is reshaped to 2D and convolved with a \(3 \times 3\) Gaussian kernel for spatial continuity, then flattened and exponentiated by polarization factor \(\beta > 1\) to re-amplify salient peaks suppressed by smoothing: $\(\hat{s}_j^I = \left(s^I_{\text{smooth},j}\right)^\beta\)$ Token selection is cast as facility location submodular maximization: $\(\arg\max_{\mathcal{Y} \subset \mathcal{V}} \sum_{j=1}^N \hat{s}_j^I \cdot \max_{v_i \in \mathcal{Y}} \text{Sim}(v_i, v_j)\)$ This objective forces the selected subset \(\mathcal{Y}\) to cover all visual tokens (every \(v_j\) has a similar representative in \(\mathcal{Y}\)), naturally penalizing redundant selections. Solved greedily with a \(1 - 1/e\) approximation guarantee using cached marginal gains \(\text{Curr}(v_j) = \max_{v_i \in \mathcal{Y}} \text{Sim}(v_i, v_j)\).

Key Contributions

• Systematic diagnosis of the dispersion phenomenon: functional words and punctuation produce near-uniform spatial similarity maps that dominate naive dense guidance aggregation.
• Entropy-guided denoising: uses information entropy as a parser-free proxy for token semantic concentration, filtering dispersed text tokens and reweighting retained ones.
• Facility location submodular maximization for token selection, replacing heuristic Top-K with a principled objective that guarantees holistic coverage and penalizes redundancy.
• Spatial smoothing + score polarization prior that propagates structural context while preventing peak suppression.
• Demonstrates SoTA on LLaVA-1.5, LLaVA-NeXT, LLaVA-Video, and Qwen2.5-VL across 10 benchmarks.

Results

All results are on LLaVA-1.5-7B (baseline: 576 tokens, avg. over 9 benchmarks = 64.9) vs. published baselines:

• 128 tokens retained (↓77.8%): EADP avg 63.5 vs. CDPruner (NeurIPS25) 63.2, DivPrune (CVPR25) 62.5, VisionZip (CVPR25) 62.5, SparseVLM (ICML25) 62.1.
• 64 tokens retained (↓88.9%): EADP avg 62.2 vs. CDPruner 61.9, DivPrune 60.9, VisionZip 60.3.
• 32 tokens retained (↓94.4%): EADP avg 60.9 vs. CDPruner 60.4, DivPrune 58.4, TRIM 56.0.
• On VizWiz (a challenging negative-query benchmark), EADP at 128 tokens scores 58.0 vs. CDPruner 57.3 and DivPrune 57.5.
• Results on LLaVA-NeXT-7B (2,880-token baseline, avg 66.3) also reported; the text was truncated before full NeXT and Qwen2.5-VL tables appeared.

Limitations

• The greedy submodular algorithm scales as \(O(K \cdot N)\) per step; optionally precomputing the \(N \times N\) token-similarity matrix trades \(O(N^2)\) memory for speed, which may be costly at very high resolutions.
• Dense scoring relies on CLIP's text encoder, which is capped at \(M \leq 77\) tokens — very long prompts cannot be fully encoded.
• Hyperparameters (\(\mu\), \(q\), \(\gamma\), \(\beta\)) require tuning; sensitivity analysis is deferred to supplementary material.
• Results tables for LLaVA-NeXT and Qwen2.5-VL were truncated in the provided text; full generalization evidence is not fully visible here.
• Method is training-free and plug-and-play, but the CLIP projection layer is borrowed from CDPruner, introducing an indirect dependency on that design.

Relevance to Vision-Language Models

Visual token pruning is a central efficiency challenge for deployment-scale VLMs, where high-resolution inputs produce thousands of patch tokens driving quadratic attention costs. EADP's entropy-based denoising directly tackles a previously undiagnosed failure mode of dense cross-modal scoring — a technique widely used in recent pruning work — making the diagnosis broadly applicable. The submodular selection formulation offers a principled alternative to Top-K that is relevant beyond pruning, connecting to the broader question of how VLMs should construct compact visual representations for LLMs. The plug-and-play, training-free design means these insights can be layered on top of existing VLM architectures without retraining.