Skip to content

SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SCOPE is a training-free visual token pruning method for MLLMs that jointly optimizes token saliency (attention scores) and semantic coverage (pairwise cosine similarity) to avoid the incomplete visual representations produced by saliency-only approaches. Applied to LLaVA-1.5 7B, it retains 96.0% of full-model performance while discarding 88.9% of visual tokens (576→64). It consistently outperforms FastV, SparseVLM, VisionZip, and PDrop across image and video benchmarks.

Problem

Existing visual token pruning methods for MLLMs select tokens purely by attention-based saliency (CLS-token or text-to-vision attention), which causes two problems: (1) semantic incompleteness—salient tokens cluster around dominant objects and ignore essential context; (2) skewed attention distributions—most tokens receive near-uniform low attention, making discrimination among tail tokens unreliable. The result is significant performance degradation under aggressive compression budgets.

Method

SCOPE defines a set-coverage score f(S) = Σ_{u∈V} max_{s∈S} sim(u, s) over the selected subset S using pairwise cosine similarities among visual tokens. For each candidate token v, a marginal coverage gain Δ(v; S) = Σ_u [max(sim(u,v), C(u,S)) − C(u,S)] quantifies how much additional semantic space v would cover. The SCOPE score multiplies this gain by the attention-based saliency raised to a scaling factor α: Δ(v, A_v^α; S) = Δ(v; S) · A_v^α. Tokens are selected iteratively (greedy) by argmax SCOPE score until the budget K is reached. The module is inserted after the vision encoder (at layer −2 of CLIP ViT), requires no additional training, and is compatible with any MLLM using a separate visual encoder.

Key Contributions

  • Formal analysis of the semantic incompleteness of saliency-only pruning via a θ-coverage metric (proportion of full tokens with a cosine-similar counterpart in the selected set); shows saliency-only can underperform random selection on this metric.
  • SCOPE score: a multiplicative combination of submodular marginal coverage gain and attention saliency, enabling greedy token selection with complementary objectives.
  • Training-free integration into LLaVA-1.5, LLaVA-Next, and Video-LLaVA; 3.2× inference speedup on LLaVA-Next 7B at 18× token compression.
  • Demonstration that token redundancy can improve performance upon removal: SCOPE exceeds the full-token upper bound on POPE (100.2%) and MMVet (104.5%) at 192 tokens on LLaVA-1.5 7B.

Results

LLaVA-1.5 7B (576 tokens baseline): - 192 tokens (−66.7%): 99.5% of upper bound vs. VisionZip 101.9%, SparseVLM 101.3%, FastV 89.7% (SCOPE best avg) - 128 tokens (−77.8%): 98.1% vs. VisionZip 104.8%, SparseVLM 96.5%, FastV 90.4% - 64 tokens (−88.9%): 96.0% vs. VisionZip 93.5%, SparseVLM 85.1%, PDrop 73.5%, FastV 83.0%

LLaVA-Next 7B (2,880 tokens baseline): - 640 tokens (−77.8%): 98.9% vs. VisionZip 97.4%, SparseVLM 96.0% - 320 tokens (−88.9%): 97.1% vs. VisionZip 95.0%, SparseVLM 93.6% - 160 tokens (−94.4%): 95.1% vs. VisionZip 92.5%, SparseVLM 86.9%

Video-LLaVA (2,048 → 136 tokens): 100.5% avg vs. VisionZip 99.8%, SparseVLM 98.8%, FastV 71.0%

Efficiency: 3.2× latency reduction on LLaVA-Next 7B (601.9s → 188.8s) at 18× token compression; POPE 81.3% vs. PDrop 53.2% at same budget.

Ablation: Coverage-only > Saliency-only > Random; joint SCOPE consistently best across GQA, MMB, MME, POPE, TextVQA at 64 tokens on LLaVA-1.5.

Limitations

  • Greedy iterative selection with O(K·N) pairwise similarity evaluations per image; precomputing the full N×N similarity matrix is required, which may be costly for very long token sequences (e.g., multi-frame video with thousands of tokens).
  • α (saliency scaling factor) is fixed at 1.0 by default; sensitivity to this hyperparameter across diverse tasks is not fully characterized.
  • Evaluated only on LLaVA-family models; generalization to architectures with cross-attention fusion (e.g., Flamingo, BLIP-2 Q-Former) is not demonstrated.
  • Performance advantage over VisionZip at moderate budgets (192 tokens on LLaVA-1.5) is marginal; gains are most pronounced under extreme compression.

Relevance to Vision-Language Models

SCOPE directly addresses the computational bottleneck of visual token sequences in MLLMs, which is a fundamental scalability challenge for high-resolution and video VLMs. The paper formalizes the semantic coverage problem with the θ-coverage metric, providing a principled diagnostic tool beyond task accuracy for evaluating pruning quality—a contribution useful for future compression research. By framing token selection as a submodular coverage maximization problem weighted by saliency, it bridges combinatorial optimization and attention-based VLM internals. The training-free, plug-and-play nature makes it immediately applicable to deployed LLaVA-family models and relevant to anyone working on efficient inference for MLLMs in resource-constrained settings.