Skip to content

Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CDPruner is a training-free, model-agnostic visual token pruning method for MLLMs that replaces attention-based or similarity-based selection with a Determinantal Point Process (DPP) that jointly maximizes feature diversity and instruction relevance. Applied to LLaVA-NeXT-7B, it reduces FLOPs by 95% and CUDA latency by 78% while retaining 94% of original accuracy at a 94.4% token reduction ratio. It achieves state-of-the-art across multiple VQA and video benchmarks.

Problem

Existing visual token pruning methods fall into two inadequate families: (1) attention-based methods retain many duplicate tokens because high attention score does not imply diversity; and (2) similarity-based methods ignore instruction relevance, pruning the same tokens regardless of the query. Neither balances the dual objectives of covering diverse visual content and attending to query-relevant regions, leading to suboptimal performance at high reduction ratios.

Method

CDPruner reformulates token selection as a conditional diversity maximization problem using Determinantal Point Process (DPP).

  1. Instruction relevance scores: Cosine similarity is computed between each visual token embedding and the mean text embedding of the instruction (from CLIP text encoder or LLM projector output). Scores are min-max normalized to [0, 1].

  2. Conditional kernel matrix: The base DPP kernel L (pairwise cosine similarity of visual tokens) is modulated by relevance scores: L̃ = diag(r̃) · L · diag(r̃). This reweights the kernel so tokens irrelevant to the query are deprioritized.

  3. Greedy MAP inference: Exact MAP inference for DPP is NP-hard; a greedy algorithm with (1 − 1/e) approximation guarantee and O(nm²) complexity via Cholesky decomposition is used, adding <10 ms overhead per sample.

The method requires no attention scores (compatible with FlashAttention), no retraining, and no architecture-specific modifications.

Key Contributions

  • CDPruner: plug-and-play, training-free visual token pruning via conditional DPP that jointly models feature similarity and instruction relevance.
  • A formulation of token subset selection as conditional diversity maximization, extending DPP from unconditional to instruction-conditioned settings.
  • Empirical demonstration of state-of-the-art performance across image VQA (10 benchmarks), high-resolution inputs, video understanding (4 benchmarks), and multi-turn dialogue, applied to LLaVA-1.5, LLaVA-NeXT, LLaVA-Video, and Qwen2.5-VL.
  • Ablation confirming that DPP-based global diversity modeling outperforms MMDP-based diversity (DivPrune), and that instruction conditioning provides further gains.

Results

  • LLaVA-1.5-7B, 32/576 tokens retained (94.4% reduction): CDPruner 94.3% Rel. vs. DivPrune 91.3%, VisionZip 88.4%, TRIM 86.2%.
  • LLaVA-NeXT-7B, 320/2880 tokens retained (88.9% reduction): CDPruner 98.0% Rel. vs. DivPrune 96.0%, VisionZip 95.7%.
  • LLaVA-NeXT-7B, 160/2880 tokens retained (94.4% reduction): CDPruner 96.0% Rel. vs. DivPrune 92.9%, second-best.
  • LLaVA-Video-7B, 64×64 tokens/frame (62.1% reduction): CDPruner 98.6% Rel. vs. PDrop 97.1%.
  • LLaVA-Video-7B, 64×16 tokens/frame (90.5% reduction): CDPruner 89.7% Rel. vs. DivPrune 88.3%, SparseVLM 79.9%.
  • Qwen2.5-VL-7B, 128/1296 tokens (90.1% reduction): CDPruner 85.2% Rel. vs. DivPrune 79.9%, FastV 79.0%.
  • Efficiency (LLaVA-NeXT-7B, 320 tokens): ×9.9 FLOPs reduction, ×6.6 prefill speedup, GPU memory 13.8 GB vs. 16.7 GB baseline; POPE F1 87.2 vs. 86.8 (original, unpruned).
  • Multi-turn dialogue (MMDU, 88.9% reduction): CDPruner 99.8% Rel. vs. TRIM 97.2%.
  • POPE hallucination benchmark: CDPruner exceeds original unpruned LLaVA-1.5-7B performance at 128-token retention.

Limitations

  • DPP-based conditional diversity offers reduced advantage on benchmarks with uninformative queries (e.g., VizWiz "What is this?"), where the instruction provides insufficient conditional signal.
  • Optimal handling of multi-turn dialogues remains an open problem; the current method partially addresses it via diversity modeling but no principled multi-turn-aware strategy is proposed.
  • Qwen2.5-VL shows greater performance drop after pruning compared to LLaVA series, because its projector already compresses visual tokens, leaving less redundancy to exploit.
  • Greedy DPP approximation is O(nm²); for very large m this may introduce non-negligible overhead, though the paper reports <10 ms in typical settings.

Relevance to Vision-Language Models

Visual token count is the dominant computational bottleneck in modern MLLMs, scaling quadratically with attention and reaching millions of tokens in video or high-resolution settings; CDPruner directly addresses this without retraining or architectural changes, making it immediately applicable to deployed VLMs. The instruction-conditioned DPP formulation is a principled departure from the attention-vs-similarity dichotomy that has defined prior pruning work, offering a unified diversity–relevance objective. The finding that pruning can reduce hallucination (POPE gains over the unpruned model) is a notable side result relevant to VLM reliability research. Its compatibility with FlashAttention and model-agnostic design make it broadly applicable across the LLaVA, InternVL, and Qwen2.5-VL families.