Skip to content

AgilePruner: An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; empirical study of attention+diversity for adaptive visual token pruning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AgilePruner is an empirical study that uses effective rank (erank) and attention-entropy to characterize the behavior of attention-based vs. diversity-based visual token pruning in LVLMs. It finds that higher retained diversity correlates with more hallucinations, and that image complexity determines which pruning paradigm works better. These insights are operationalized into a simple adaptive threshold-based pruning method that matches or surpasses existing methods across nine benchmarks.

Problem

Visual token pruning for LVLMs is dominated by attention-based and diversity-based approaches, but their actual behavioral characteristics—how much feature diversity each truly preserves, how that diversity links to hallucination, and when each paradigm is preferable—have not been systematically studied. Hybrid methods exist but lack principled justification for combining the two strategies.

Method

The analysis framework uses two complementary metrics applied to vision encoder outputs:

  • Attention entropy: Shannon entropy \(H(p) = -\sum_i p_i \log p_i\) of the [CLS] token's head-averaged attention scores from the penultimate vision encoder layer, renormalized after excluding the self-attention term. Lower entropy = more concentrated attention.
  • Effective rank (erank): \(\text{erank}(A) = \exp\!\left(-\sum_{i=1}^{L} q_i \log q_i\right)\) where \(q_i\) are the normalized singular values of the token embedding matrix \(A \in \mathbb{R}^{N \times d_l}\). Higher erank = more diverse token embeddings.

These metrics are used to: (1) measure how much diversity existing methods actually preserve; (2) correlate retained diversity with hallucination rates on CHAIR; and (3) identify that image complexity (measured by attention entropy and erank) predicts which pruning paradigm will excel.

From these insights, the authors derive AgilePruner: a threshold-based method that traverses tokens in descending attention-score order and removes a token if its cosine similarity to any already-selected token exceeds a threshold \(\tau\), where \(\tau\) is set adaptively based on the image's attention entropy (simple images → lower threshold, permitting more aggressive attention-biased selection; complex images → higher threshold, encouraging diversity).

For existing methods, a simple image-aware adjustment routes simple-image samples toward attention-biased selection and complex-image samples toward diversity-biased selection, consistently improving hybrid baselines.

Key Contributions

  • First erank-based quantitative characterization showing that diversity-oriented pruning methods often preserve far less diversity than intended (e.g., DivPrune erank 21.84 vs. PruMerge+ 10.91 at 64 tokens).
  • Discovery that higher retained token diversity is positively correlated with hallucination frequency on CHAIR, whereas attention-biased pruning suppresses hallucinations at the cost of lower object recall.
  • Demonstration that image complexity (measured by attention entropy/erank) systematically determines attention vs. diversity preference: simple images favor attention-based, complex images favor diversity-based pruning.
  • A minimal adaptive pruning instantiation (AgilePruner) that translates these principles into a training-free method achieving strong performance across nine benchmarks and reduced hallucination.
  • Validation that all findings are model-agnostic across LLaVA-1.5-7B, LLaVA-1.5-13B, LLaVA-NeXT-7B, and Qwen2.5-VL-7B.

Results

  • Diversity vs. hallucination (CHAIR, 64 tokens, LLaVA-1.5-7B): DivPrune achieves \(C_S = 57.4\), \(C_I = 18.0\); FasterVLM (attention-based) achieves \(C_S = 45.4\), \(C_I = 13.5\). Increasing attention-selection ratio \(R\) from 0→0.75 in DivPrune monotonically reduces \(C_S\) from 57.4→45.2 and \(C_I\) from 18.0→14.1.
  • Image-complexity preference (Table 4): On simple-image tasks (OCR, Numerical Calc.), attention-based pruning outperforms diversity-based (e.g., OCR: 140 vs. 130 MME score). On complex-image tasks (POPE, Scene, Count), diversity-based wins (POPE: 86.0 vs. 77.4).
  • Adaptive combination (Table 6, 64 tokens): DivPrune+FasterVLM adaptive (97%→adaptive split) achieves 94.87% relative MME vs. 94.08% for fixed combination; POPE improves from 82.2 → 83.5.
  • VisPruner + image-aware adjustment (128 tokens): POPE improves from 84.6→85.5 (+0.9), MME from 1768→1787 (+19).
  • AgilePruner matches or surpasses existing pruning methods on nine standard datasets while further reducing hallucination, consistent with empirical predictions.

Limitations

  • The adaptive threshold in AgilePruner requires tuning; the paper describes it as "intentionally minimal," leaving room for more principled threshold derivation.
  • The finding that diversity increases hallucination is tested primarily on CHAIR (image captioning); generalization to other task types (VQA, grounding) is less thoroughly analyzed.
  • Effective rank computation via SVD adds computational overhead that is not fully quantified in the paper.
  • The image-complexity signal (attention entropy) is computed from the vision encoder's penultimate layer, making it architecture-coupled; transferability to encoders without a [CLS] token (e.g., SigLIP variants) is not discussed.
  • Analysis is limited to pre-LLM pruning; dynamic pruning within the LLM layers is not addressed.

Relevance to Vision-Language Models

AgilePruner directly addresses the inference efficiency bottleneck in LVLMs caused by quadratically scaling visual token attention, which is a central challenge as models like LLaVA-NeXT and Qwen2.5-VL process higher-resolution inputs. The erank-hallucination correlation is a non-obvious, empirically grounded insight that challenges the naive assumption that more diverse tokens always produce better outputs—directly relevant to reliability and hallucination research in VLMs. The image-complexity-dependent switching principle provides a theoretically grounded rationale for adaptive compute allocation, connecting to broader work on dynamic inference in multimodal models. The training-free nature of AgilePruner makes it immediately applicable as a plug-in to existing LVLM deployments without fine-tuning.