Skip to content

AnchorPrune: Relevance-Anchored Contextual Expansion for Visual Token Pruning

🕒 Published (v1): 2026-07-08 06:07 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; query-relevance-anchored visual token pruning cuts VLM inference cost

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AnchorPrune is a training-free visual token pruning framework that resolves the conflict between relevance-driven and diversity-driven token selection by enforcing a strict two-stage ordering: first construct a protected relevance anchor of query-critical tokens, then expand it greedily with globally informative, non-redundant context. On LLaVA-NeXT-7B it preserves 97.6% of full-token performance retaining only 160 of 2,880 tokens, exceeding the second-best training-free baseline by 4.7 percentage points.

Problem

High-resolution VLM inputs produce thousands of visual tokens, most of which are redundant for a given query but still incur prefilling cost. Existing pruning methods either (a) use saliency/relevance alone, over-concentrating the budget on correlated local tokens, (b) use diversity alone, fragmenting coherent evidence or retaining visually distinct but query-irrelevant regions, or (c) jointly optimize relevance and diversity within a unified objective (e.g., relevance-conditioned determinantal point processes), which provides no explicit protection for indispensable evidence — diversity can displace non-substitutable query cues under aggressive compression with no separate compensation stage.

Method

AnchorPrune selects \(S = S_\text{rel} \cup S_\text{ctx}\) with \(|S|=K\) in two sequential, non-interleaved stages.

Stage 1 — Protected Relevance Anchoring. Tokens are ranked by an architecture-specific query-conditioned priority score \(s_i\): - CLIP-aligned VLMs: negated patch–text cosine similarity in the pre-projector CLIP space, \(s_i = -\frac{1}{M}\sum_m v_i^\top t_m\), where prior work found raw CLIP local similarities over-respond to background. - Non-CLIP VLMs: token-wise maximum cosine similarity between projected visual tokens and language-model instruction embeddings, \(s_i = \max_m \langle z_i/\|z_i\|, e_m/\|e_m\| \rangle\).

The anchor size \(K_\text{rel}\) is determined adaptively rather than by a fixed ratio. Starting from a minimum anchor \(A_\text{min}\) of size \(K_\text{min}\), the method counts subsequent relevance-ranked candidates \(\pi_t\) whose novelty \(\Delta(\pi_t; A_\text{min}) = \min_{j \in A_\text{min}}(1 - h_{\pi_t}^\top h_j / \|h_{\pi_t}\|\|h_j\|)\) exceeds a threshold \(\tau\). Anchor expansion stops at the \(P\)-th such novelty event, or defaults to \(K_\text{max} = \lfloor K/2 \rfloor\) if fewer than \(P\) novel candidates are found (ensuring at least half the budget remains for Stage 2).

Stage 2 — Contextual Expansion. Each remaining slot is filled greedily: $\(i^\star = \arg\max_{i \notin S}\; p_i \cdot \Delta(i; S), \quad S \leftarrow S \cup \{i^\star\}\)$ where \(p_i\) is a global importance prior (mean CLS-to-patch attention for encoders with a [CLS] token; mean received attention otherwise, aggregated over pre-merge groups) and \(\Delta(i;S)\) is novelty relative to the current retained set (anchor + all previously chosen context tokens). The multiplicative form jointly suppresses uninformative-but-diverse tokens and informative-but-redundant tokens.

Key Contributions

  • Identifies an ordering limitation in existing pruning methods: jointly optimizing relevance and diversity neither protects non-substitutable query evidence nor provides a separate compensation stage when relevance guidance is weak or diffuse.
  • Introduces AnchorPrune: adaptive protected relevance anchoring followed by importance-weighted novelty expansion, enforcing asymmetric selection order.
  • Architecture-aware, training-free design applicable to CLIP-aligned and non-CLIP image VLMs and video VLMs without model modification or retraining.

Results

  • LLaVA-1.5-7B (576 tokens, CLIP-aligned):
  • 128 tokens: 98.7% relative retention (best; CDPruner next at 94.2%)
  • 64 tokens: 96.2% (best; CDPruner next at 94.2% — gap +2.0 pp)
  • 32 tokens: 93.9% (best; CDPruner next at 92.6% — gap +1.3 pp)
  • LLaVA-NeXT-7B (2,880 tokens, CLIP-aligned):
  • 640 tokens: 99.4% (best; CDPruner next at 95.9% — gap +3.5 pp)
  • 320 tokens: 98.3% (best; CDPruner next at 95.9% — gap +2.4 pp)
  • 160 tokens: 97.6% (best; CDPruner next at 92.9% — gap +4.7 pp)
  • Qwen2.5-VL-7B (1,296 tokens, non-CLIP):
  • Best Rel. at all three budgets (256/128/64 tokens), with largest advantages on DocVQA (e.g., 51.0 vs. 49.8 at 128 tokens, 31.3 vs. 31.6 at 64 tokens — competitive)
  • LLaVA-Video-7B: evaluated on Video-MME, EgoSchema, TempCompass (detailed numbers in Table 4, not reproduced in the provided text)
  • Gains increase under more aggressive compression across all models, consistent with the anchor protecting non-substitutable evidence that is most easily lost at low budgets.

Limitations

  • The anchoring-priority signal for CLIP-aligned models uses negated CLIP similarity — a heuristic motivated by background-sensitivity observations, not by a theoretically grounded relevance measure; behavior may vary across CLIP variants.
  • The adaptive anchor procedure introduces three hyperparameters (\(K_\text{min}\), \(\tau\), \(P\)) whose sensitivity across different image domains or VLM families is not fully characterized in the provided text.
  • Stage 2 greedy selection is \(O(NK)\) per layer, which may scale poorly for extremely long video sequences beyond the evaluated settings.
  • No analysis of failure modes where the relevance anchor is systematically miscalibrated (e.g., for queries with entirely implicit visual grounding).
  • Evaluation limited to 7B-scale models; scalability behavior for larger backbones is unverified.

Relevance to Vision-Language Models

AnchorPrune directly addresses the inference bottleneck that limits practical deployment of high-resolution and video VLMs, demonstrating that the ordering of selection criteria — not just their combination — is a critical design axis. The architecture-aware scoring (pre- vs. post-projector similarity depending on whether a CLIP encoder is present) makes it applicable across the current diversity of VLM architectures without retraining, which is important as the field consolidates around multiple distinct encoder paradigms. The result that 160 of 2,880 tokens suffice to retain 97.6% performance on LLaVA-NeXT-7B has direct implications for efficient serving and mobile/edge deployment of VLMs. The principle of asymmetric, ordered selection (protect non-substitutable evidence first, then expand) is likely generalizable to other modalities or token-reduction problems in multimodal models.