Skip to content

Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval

🕒 Published (v1): 2026-07-06 02:19 UTC · Source: Arxiv · link

Why this paper was selected

Object-evidence token merging for dense retrieval; practical efficiency for deploying VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SaMer (Semantic-aware Merging) is a token merging framework for multi-vector vision-language retrieval that compresses image-side post-projector tokens into \(K\) object-aware centroids, preserving the late-interaction MaxSim interface. At \(K=64\), it eliminates >93% of image tokens and achieves a 16.09× storage reduction while improving R@1 on both Flickr30K and MSCOCO over the uncompressed ColPali baseline.

Problem

Multi-vector retrievers (ColPali, ColQwen2) store hundreds of patch embeddings per image for MaxSim late interaction, creating O(\(ND\)) index storage and O(\(MND\)) scoring cost per query. Existing token compression methods — pruning, pooling, and feature-based merging — reduce this cost but risk collapsing object-instance evidence: two visually similar patches from different object instances can merge into a single representative, destroying the token-level discriminability that query tokens need to select under MaxSim. The key failure mode is cross-instance collapse, which cannot be detected by feature similarity alone.

Method

SaMer operates on post-projector visual tokens \(V = \{v_i\}_{i=1}^N\) and maps them to \(K\) representatives \(R(I) = \{r_k\}_{k=1}^K\) in three stages:

1. Feature-spatial soft assignment. Each token \(v_i\) at spatial coordinate \(p_i\) is assigned to representative \(k\) via a combined distance: $\(d(i,k) = \underbrace{1 - v_i^\top \mu_k}_{\text{feature similarity}} + \gamma\underbrace{\|p_i - s_k\|_2^2}_{\text{spatial coherence}}\)$ Soft assignment weights \(a_{i,k} = \text{softmax}_k(-d(i,k)/\tau_s)\) form each centroid as a normalized weighted average.

2. Object-aware merge prior (training only). Using bounding-box annotations, SaMer estimates per-representative bbox-label distributions via stop-gradient hard assignments and computes a cross-instance penalty \(P_\text{inst}(i,k) = 1 - P_k(b_i)\), which is large when assigning token \(i\) would mix it with tokens from a different object instance. This penalty is added to \(d(i,k)\) before soft assignment, reshaping gradients without an auxiliary loss. No bbox annotations or detectors are used at inference.

3. Projection-only adaptation. Only the shared image-text projection layer is updated using a multi-positive InfoNCE loss over compressed MaxSim scores \(S_K(q,I) = \frac{1}{M}\sum_{j=1}^M \max_{k \in [K]} q_j^\top r_k\). Vision encoder and language backbone remain frozen; no new learnable merge parameters are introduced.

Key Contributions

  • Frames retrieval-side token compression as an evidence-preservation problem, identifying cross-instance collapse as the primary failure mode of feature-based merging.
  • Proposes SaMer: training-time object annotations serve as a merge prior (not a loss), enabling annotation-free inference while instilling object-consistent grouping into the adapted projection space.
  • Demonstrates that projection-only adaptation with an object-aware merge rule outperforms all pruning/pooling compression baselines under matched \(K=64\) budget with the same adaptation setting.
  • Introduces grounding-oriented evaluation to verify that compressed tokens retain phrase-level visual evidence, not just aggregate retrieval scores.

Results

  • Storage: \(K=64\) removes >93% of ColPali image-side tokens, reducing storage by 16.09×.
  • Flickr30K R@1: ColPali 77.0 → SaMer 82.4; ColQwen2 73.6 → 79.3 (in-domain).
  • MSCOCO R@1: ColPali 47.4 → 51.6; ColQwen2 43.3 → 47.5 (cross-dataset).
  • ImageCoDe R@1: ColPali 5.4 → 5.9; nDCG@10 13.2 → 14.4 (compositional near-miss retrieval).
  • Outperforms all three compression baselines (H-Pool, SAP, HPC) on natural-image benchmarks under identical \(K=64\) budget and same adaptation protocol (Table 2).
  • Compression budget analysis shows diminishing returns beyond \(K=64\); R@5 on Flickr30K rises only from 96.3 (\(K=64\)) to 96.8 (\(K=512\)).
  • DocVQA: SaMer remains competitive with compressed baselines but does not improve over uncompressed ColPali, as document OCR/layout tokens are sparse and not redundant like natural-image patches.

Limitations

  • Object-aware merge prior requires annotated bounding-box data during training (Flickr30K-Entities); adaptation benefits may not transfer to domains lacking such annotations.
  • DocVQA performance does not improve: the object-centric compression assumption fails when sparse OCR tokens constitute the discriminative evidence.
  • Evaluated only on two multi-vector backbones (ColPali/ColQwen2); generalizability to other late-interaction VLM retrievers is not demonstrated.
  • The paper is truncated before the full merge component ablation (Table 4) results are presented, leaving the isolated contribution of the spatial coherence term somewhat unclear.
  • K is a fixed global budget; no mechanism to allocate tokens unevenly across semantically rich vs. sparse regions.

Relevance to Vision-Language Models

SaMer directly addresses the efficiency bottleneck of patch-based VLM retrievers (ColPali, ColQwen2) by showing that how tokens are merged — respecting object-instance boundaries — matters more than simply reducing count. For VLM researchers, the finding that compressed representations can outperform the uncompressed baseline is significant: it suggests that redundant patch tokens add noise to MaxSim scoring rather than signal. The projection-only adaptation strategy (frozen backbone) also provides a lightweight recipe for specializing existing VLMs to compressed retrieval without full fine-tuning. This work complements the broader trend of efficient VLM deployment by targeting the index/serving cost rather than inference compute.