Moving Beyond Diversity: Visual Token Pruning as Subspace Reconstruction for Efficient VLMs¶

🕒 Published (v1): 2026-06-17 04:45 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; novel subspace-reconstruction reframing of token pruning; critical efficiency gain

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

SPARE reformulates visual token pruning in VLMs as a Column Subset Selection Problem (CSSP), using greedy subspace reconstruction via rank-revealing QR (RRQR) factorization to minimize reconstruction error rather than maximizing angular diversity. It additionally exploits an empirically discovered "anti-relevance" phenomenon—tokens with lower CLIP-based image-text similarity score better preserve contextual information—as a multiplicative guidance term. Applied to LLaVA, SPARE removes up to 94% of visual tokens while retaining 95% of baseline performance, training-free.

Problem¶

Dominant diversity-based token pruning methods (DivPrune, DART, CDPruner) rely on cosine-normalized similarity, which discards token magnitude entirely. This treats pruning as a dispersion problem rather than an approximation problem, leading to large reconstruction errors of the original feature matrix and degraded performance on compositional multi-skill tasks (e.g., simultaneous OCR + spatial reasoning + recognition). Attention-based methods are incompatible with FlashAttention and can underperform random pruning.

Method¶

Subspace Reconstruction via RRQR. Given visual embeddings $\mathbf{X}_v \in \mathbb{R}^{N \times d}$, SPARE minimizes the Frobenius-norm reconstruction error: $$\min_{\mathcal{S} \subset \{1,\dots,N\}} \left\| \mathbf{X}_v^\top - \mathbf{\Pi}_{\mathcal{S}} \mathbf{X}_v^\top \right\|_F^2 \quad \text{s.t. } |\mathcal{S}| = k,$$ where $\mathbf{\Pi}_{\mathcal{S}}$ is the orthogonal projector onto the span of selected tokens. Because exact CSSP is NP-hard, SPARE uses a greedy RRQR strategy: at each step, it selects the token $i$ with the largest projection residual $r_i = \|\mathbf{x}_i\|_2^2 - \sum_{\mathbf{q} \in \mathcal{Q}} (\mathbf{x}_i^\top \mathbf{q})^2$, then updates an orthonormal basis via Gram–Schmidt. This captures both direction and magnitude, unlike normalized cosine similarity.

Anti-Relevance Guidance. The anti-relevance score for token $n$ is: $$\rho_n = \frac{1}{M}\sum_{j=1}^{M} \left(-\cos(\mathbf{E}_{I,n}, \mathbf{E}_{T,j})\right),$$ z-score normalized per image and passed through softplus. At each iteration, the selected token jointly maximizes $r_n \cdot \rho_n$, biasing selection toward tokens that both expand the reconstruction subspace and lie in low CLIP-similarity regions (empirically found to carry more complementary contextual information).

Key Contributions¶

First formulation of visual token pruning as CSSP, explicitly accounting for token magnitude during selection.
Discovery and characterization of the anti-relevance phenomenon: low CLIP-similarity tokens consistently outperform high-similarity tokens as selection targets across multiple benchmarks.
SPARE algorithm: training-free, plug-and-play, with provable approximation guarantees inherited from RRQR-based CSSP and a Lipschitz bound linking embedding reconstruction error to LLM output divergence.
State-of-the-art results on nine benchmarks across four VLM architectures (LLaVA-1.5-7B/13B, LLaVA-NeXT-7B, Qwen2.5-VL-7B), with especially strong gains on MM-Vet compositional tasks.

Results¶

LLaVA-1.5-7B, 32 retained tokens (94.4% reduction): SPARE achieves 94.7% relative accuracy vs. CDPruner 93.0%, DART 85.6%. MM-Vet: SPARE 31.4 vs. CDPruner 27.9.
LLaVA-1.5-13B, 32 tokens: SPARE 94.7% rel. acc. vs. CDPruner 92.6%, DART 86.0%. MM-Vet: SPARE 34.7 vs. CDPruner 29.5.
LLaVA-NeXT-7B, 160 tokens (94.4% reduction from 2880): SPARE 96.2% rel. acc. vs. CDPruner 95.4%, DART 90.1%.
128-token regime (77.8% reduction, LLaVA-1.5-7B): SPARE 98.3% vs. CDPruner 98.0%, SparseVLM 97.0%.
MM-Vet gains over CDPruner at 128 tokens on LLaVA-1.5-7B: +0.8 points; performance gap on multi-skill combinations in MM-Vet reaches up to +33.3 points for certain 4-skill combinations.

Limitations¶

Theoretical Lipschitz bound (Eq. 6) provides a loose, qualitative link between embedding reconstruction error and LLM output divergence; tightness depends on the unknown local Lipschitz constant $L$.
Anti-relevance scores rely on CLIP embeddings, introducing a dependency on a separate pretrained model and adding minor inference overhead.
RRQR greedy selection is $O(Nkd)$ per image; at very high token counts (e.g., video with tens of thousands of tokens), computational cost may become non-trivial compared to simpler diversity heuristics.
Evaluated only on image VLMs; generalization to video models with temporal token accumulation is claimed but not fully demonstrated in the presented results.

Relevance to Vision-Language Models¶

SPARE directly addresses a core efficiency bottleneck in modern VLMs: the quadratic attention cost incurred by large visual token sequences from high-resolution or multi-frame inputs. By reframing pruning as subspace reconstruction, it provides a principled, training-free alternative to the currently dominant diversity-based paradigm, and its anti-relevance finding challenges the standard assumption that CLIP-aligned tokens are the most task-relevant—a result with implications for how VLM inference pipelines should be designed. The method's compatibility with multiple architectures (LLaVA, Qwen2.5-VL) and its plug-and-play nature make it immediately applicable to deployed VLM systems. The strong gains on compositional multi-skill benchmarks (MM-Vet) are particularly relevant for tracking progress on harder VLM reasoning tasks that go beyond single-skill evaluation.