Skip to content

To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Mechanistic analysis of visual information pathways and attention sinks in LVLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Large Vision-Language Models (LVLMs) contain two distinct classes of attention sinks: those originating in the ViT backbone that propagate into the LLM, and those emerging natively in the LLM. This paper shows that propagated ViT sinks encode coarse, high-level semantic summaries of images and are selectively beneficial — useful for global reasoning tasks, detrimental for fine-grained local tasks. The authors propose DIYSink, a dual-MLP projection framework with dynamic token selection, to adaptively exploit ViT sink tokens.

Problem

Prior work conflates two fundamentally different sink token populations in LVLMs: ViT-originating sinks that propagate into the LLM, and LLM-emerged sinks. Existing methods either discard visual sinks entirely (Kang et al., 2025) or absorb them with register tokens (Darcet et al., 2024), ignoring their semantic content. No prior work systematically analyzes whether ViT sinks carry exploitable visual information, how they propagate, or how to adaptively leverage them for downstream tasks.

Method

Sink identification: ViT sinks \(\hat{I}^l_{\text{vit}}\) are defined as tokens whose feature norm \(\phi(x_j^{l-1}) = \|x_j\|\) exceeds threshold \(\tau = 100\); typically 3–5 tokens per image. LLM-emerged sinks use a sink-dimension value criterion with \(\tau = 20\).

Analysis tools: Relevance maps (vertical attention columns reshaped to patch grids) and a vocabulary-projection technique (masking all other visual tokens and forwarding isolated sink embeddings through the LLM head) reveal that ViT sinks encode coarse, object-level semantic content, whereas non-sink tokens attend locally.

Training-free — sink-to-the-front: Sink tokens (with their positional embeddings) are reordered to the beginning of the visual token sequence before the connector and LLM, exploiting the LLM's causal structure so all subsequent tokens can attend to them.

Training-based — DIYSink: - Dual-MLP Projector: Two separate MLP connectors \(f_{\text{sink}}: \mathbb{R}^{D'} \to \mathbb{R}^D\) and \(f_{\text{non-sink}}: \mathbb{R}^{D'} \to \mathbb{R}^D\) are trained independently on disjoint token sets to avoid conflation of their distinct activation statistics. - Dynamic token selection: Two mechanisms gate which tokens enter the LLM at inference. CoT-Reweighting uses Chain-of-Thought routing to hard-select sink-only, non-sink-only, or both, based on GPT-4o-annotated task type (global vs. local vs. mixed). MLP-Reweighting trains a lightweight sentence-encoder-based MLP to produce soft reweighting coefficients over sink vs. non-sink token streams, requiring only a small amount of cross-task data.

Key Contributions

  • Empirical demonstration that ViT high-norm sinks propagate into the LLM as a distinct sink population, activating different hidden dimensions ({982, 2494, 3263}) than LLM-native sinks ({2533, 1415} in LLaVA-7B), and receiving on average \(\sim7\times\) more attention than regular visual tokens.
  • Quantitative and qualitative evidence that ViT sinks encode coarse-grained, global semantic features (confirmed via relevance maps and vocabulary-projection word distributions).
  • Task taxonomy: 600 GPT-4o-annotated image-query pairs categorized by image complexity and query globalness into Global, Local, and Mixed clusters; demonstrating that sink tokens benefit Global tasks and hurt Local tasks.
  • Plug-in, training-free sink-to-the-front reordering applicable post hoc to closed-source LVLMs.
  • DIYSink: dual-MLP projector + CoT or MLP dynamic selection, applicable to any open-source LVLM.

Results

  • Sink-to-the-front (training-free) — consistent MME improvements across five closed-source LVLMs tested without any fine-tuning:
  • InternVL2.5-4B: MME +19.17
  • Phi 3.5V: MME +3.37
  • Molmo-7B-D: MME +30.28
  • Gemma3-12B-it: MME +33.81
  • MathVista ALG improvements: Molmo-7B-D +1.80, Gemma3-12B-it +0.20
  • DIYSink on TinyLLaVA-0.5B-SigLIP-Qwen2-0.5B (full benchmark):
  • Baseline AVG: 48.34 → DIYSink (CoT): 49.04 (+0.70), DIYSink (ReW): 49.13 (+0.79)
  • MME: baseline 1381.10 → CoT 1456.78 (+75.68), ReW 1451.87 (+70.77)
  • MathVista overall improved; SQA and TextVQA also improve slightly
  • Preliminary experiment: Sink-only inference yields strong performance on Global tasks; Non-sink-only yields better performance on Local tasks, validating the task-conditional hypothesis.

Limitations

  • Sink identification requires choosing threshold \(\tau\) (set to 100 for feature norm), which may not generalize across all ViT architectures without tuning.
  • DIYSink is a train-from-scratch approach; it cannot be directly retrofitted to existing pre-trained LVLMs without retraining with the dual-MLP objective.
  • The CoT-Reweighting path depends on GPT-4o annotations for task taxonomy, introducing an expensive oracle at train time and a routing LLM call at inference.
  • Benefits of sink tokens appear highly conditional (Global vs. Local task split), meaning gains on global benchmarks may come with regressions on fine-grained localization tasks.
  • Analysis is conducted primarily on LLaVA-7B; generalization across all LVLM families is validated but the paper text provided is truncated before full DIYSink benchmark results on larger models.

Relevance to Vision-Language Models

This paper directly addresses a core mechanistic question in LVLM research: how visual information encoded by the ViT backbone actually flows into and is used by the language model. By disentangling ViT-propagated sinks from LLM-native sinks — two populations previously conflated — it offers new interpretability into the visual-language alignment interface, specifically the MLP connector bottleneck. The task-conditional utility of sink tokens (global vs. local) has direct implications for benchmark design and token pruning strategies, as methods that uniformly discard or retain sink tokens will systematically fail on one task class. DIYSink's dual-MLP design also challenges the standard single-projector connector paradigm in models like LLaVA, suggesting that structurally heterogeneous visual tokens warrant structurally heterogeneous projection.