Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; synergistic importance-diversity token compression for VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

PruneSID is a training-free visual token compression framework for VLMs that resolves the tension between retaining semantically important tokens and maintaining information diversity. It uses a two-stage pipeline — PCA-based semantic clustering followed by intra-group Non-Maximum Suppression — plus an image-complexity-adaptive token budget. It achieves 96.3% of baseline accuracy on LLaVA-1.5 while retaining only 11.1% of visual tokens.

Problem¶

Existing visual token reduction methods face a fundamental trade-off: attention-guided selection preserves salient tokens but discards background context and retains redundant near-duplicate patches; duplication-aware pruning improves diversity but may discard high-attention, semantically critical tokens. Neither approach simultaneously optimizes importance (retaining semantically salient content) and diversity (ensuring broad concept coverage), causing significant accuracy degradation at high compression ratios.

Method¶

PruneSID operates entirely at the vision-encoder output stage (before the LLM) with no fine-tuning.

Stage 1 — Principal Semantic Components Analysis (PSCA): Given token embeddings $X \in \mathbb{R}^{T \times D}$, features are sigmoid-activated and zero-mean centered across the token dimension. Low-rank SVD is applied to $X_{\text{ctr}}^\top$, yielding right singular vectors $V \in \mathbb{R}^{T \times K}$. Each token is assigned to the principal direction with the largest absolute loading: $$g(i) = \arg\max_j |V_{i,j}|$$ producing $K$ semantically coherent groups $\{G_1, \ldots, G_K\}$.

Stage 2 — Intra-group NMS: Within each group, tokens are ranked by their principal-component contribution score $s_i = |V_{i,g(i)}|$. A greedy NMS pass suppresses a token if its cosine similarity to any already-selected token exceeds threshold $\tau = \lambda \cdot \rho$, where $\rho$ is the average pairwise token similarity (global redundancy score) and $\lambda = N/32$ scales with the token budget $N$. Final group quotas $\{n_k\}$ are allocated proportionally to surviving group sizes.

Dynamic compression ratio: An image-level information score $\varphi = 1 - \rho$ drives per-image token budget adjustment, allocating more tokens to complex scenes and fewer to simple ones, while keeping the dataset-average budget fixed.

The pruning objective is grounded theoretically via the Inclusion-Exclusion Principle: PSCA maximizes $\sum_{s_i \in S'} I(s_i)$ while NMS enforces $R(s_i, s_j) \leq \epsilon$, jointly approximating: $$\max_{|S'|=N} \sum_{s_i \in S'} I(s_i) \quad \text{s.t.} \quad R(s_i, s_j) \leq \epsilon$$

Key Contributions¶

Training-free two-stage compression (PSCA + intra-group NMS) that jointly optimizes semantic importance and information diversity.
Information-aware dynamic compression ratio using global token redundancy $\rho$ to adaptively assign per-image token budgets.
Theoretical justification via the Inclusion-Exclusion lower bound on token set informativeness.
Demonstrated generalization across LLaVA-1.5, LLaVA-NeXT, Mini-Gemini, Qwen-VL, and Video-LLaVA (image and video modalities).

Results¶

LLaVA-1.5 @ 64 tokens (11.1% retention): 96.3% of full-model accuracy (avg. across 11 benchmarks); +1.9% over VisionZip, +7.9% over HiRED.
LLaVA-NeXT @ 5.6% retention (160 tokens): 92.8% of full-model accuracy; +2.5% over VisionZip.
LLaVA-NeXT @ 11.1% retention: +1.5% over VisionZip; @ 22.2%: +0.9%.
Mini-Gemini @ 64 tokens (88.9% reduction): 94.4% of baseline; +2.0% over VisionZip.
Video-LLaVA @ 6.6% per-frame retention (17/256 tokens): 95.5% of full-model average across TGIF/MSVD/MSRVTT/ActivityNet; +2.3% over VisionZip, vs. SparseVLM at 86.5%.
$7.8\times$ faster prefilling speed vs. uncompressed LLaVA-NeXT (cited in abstract).
PruneSID-Dyn (with dynamic ratio) provides consistent additional gains of +0.1–0.5% over the fixed-budget variant.

Limitations¶

Hyper-parameter $\lambda = N/32$ for NMS threshold scaling is empirically determined; behavior under out-of-distribution compression budgets or novel VLM architectures is uncharacterized.
Number of PCA components $K$ is a fixed design choice; sensitivity analysis is not detailed in the main text.
Dynamic compression ratio adds per-image overhead from computing $\rho$ (average pairwise similarity over all $T$ tokens), which scales as $O(T^2)$ and may be costly for very high-resolution encoders.
Evaluated at the vision-encoder output only; no analysis of interaction with LLM-internal token pruning methods.
Video experiments are limited to 8-frame clips; long-video scalability is untested.

Relevance to Vision-Language Models¶

PruneSID directly addresses the inference-efficiency bottleneck that becomes acute as VLMs shift toward higher-resolution inputs (e.g., LLaVA-NeXT's 2880 tokens per image), making it immediately applicable to production deployment. The PSCA mechanism offers a principled alternative to CLS-attention-based selection (dominant in VisionZip, HiRED, LLaVA-PruMerge), demonstrating that PCA over the token dimension — rather than the feature dimension — is an effective proxy for semantic grouping without any learned parameters. The adaptive compression ratio is a practically important contribution, since fixed-ratio compression implicitly assumes uniform image complexity, a poor fit for real-world datasets. For researchers tracking VLM efficiency, this work clarifies that the importance–diversity trade-off is the central axis of the sub-field and provides a clean two-component decomposition of that trade-off.