Enhancing Multi-Image Understanding through Delimiter Token Scaling¶
🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link
Why this paper was selected
Delimiter token scaling resolves cross-image leakage in multi-image VLMs
Ask a follow-up
Open an assistant pre-loaded with this paper's context.
💬 Ask ChatGPT✦ Ask Claude
TL;DR¶
Large Vision-Language Models (LVLMs) suffer from cross-image information leakage when processing multiple images simultaneously, causing outputs to incorrectly mix details across images. This paper identifies that existing image delimiter tokens fail to fully block such leakage and proposes a training-free fix: scaling the hidden states of delimiter tokens by a factor \(\lambda > 1\) to strengthen their boundary-enforcing role. The method yields consistent gains on multi-image, multi-document, and multi-table benchmarks with zero additional training or inference cost.
Problem¶
When multiple images are fed to LVLMs, performance degrades relative to single-image settings due to cross-image information leakage: the model conflates information from different images. Existing LVLMs insert special delimiter tokens (e.g., <|vision_start|>, <|vision_end|>) to demarcate image boundaries, but attention analysis shows cross-image interaction persists, meaning delimiters are necessary but insufficient. Prior training-free fixes either require external modules (AVAM) or \(n+1\) forward passes for \(n\) images (FOCUS), imposing structural or computational overhead.
Method¶
The core insight comes from analyzing two properties of image delimiter tokens:
- Property 1 (attention sink): The \(i\)-th delimiter token receives strong, localized attention predominantly from tokens of the \(i\)-th image, forming a one-to-one image-to-delimiter mapping—unlike global BOS sink tokens.
- Property 2 (image-tagging): Because all tokens in image \(i\) share the common additive term \(p_{d_i} v_{d_i}\) in their attention output (Eq. 1), delimiter tokens act as a localized shared bias that reinforces intra-image interaction.
Delimiter Token Scaling: At every transformer layer \(l\), the hidden state of each delimiter token index \(t \in D\) is multiplied by \(\lambda > 1\): $\(h_t^{(l)*} = \begin{cases} \lambda \cdot h_t^{(l)} & t \in D \\ h_t^{(l)} & \text{otherwise} \end{cases}\)$
Scaling amplifies both properties simultaneously: higher activations pull more attention toward the correct delimiter (softmax suppresses cross-image weights), while also increasing \(|v_{d_i}|\), which amplifies the intra-image tagging term and counteracts the softmax suppression of within-image attention. The method modifies only hidden states, leaving the attention kernel untouched and preserving FlashAttention compatibility.
Key Contributions¶
- Analysis showing delimiter tokens exhibit a unique localized sink behavior distinct from global BOS sinks, and identifying two properties (attention absorption + image-tagging) governing their role.
- DelimScaling: a single-line modification—hidden-state multiplication by \(\lambda\)—that requires no fine-tuning, no architectural change, and no additional forward passes.
- Empirical demonstration that the method transfers to text-only multi-instance tasks (multi-document summarization, multi-table QA) by strengthening analogous delimiter structures.
- Evidence that a smaller model with DelimScaling can match or exceed a larger baseline (Qwen2.5-3B + Ours \(>\) Qwen2.5-7B on TQABench).
Results¶
Multi-image benchmarks (Table 1, baselines = unmodified models): - Qwen2.5-VL-3B on MuirBench: 37.31 → 42.42 (+5.1 pp) - Qwen2.5-VL-32B on Mantis: 68.20 → 70.05 (+1.85 pp) - InternVL3-2B on Mantis: 52.07 → 54.38 (+2.31 pp) - Gains consistent across Qwen2.5-VL (3B–32B), InternVL3 (1B–14B), LLaVA-OneVision (0.5B–7B)
Multi-document summarization (WCEP-10, Table 2): - Qwen2.5-3B ROUGE-2: 9.75 → 9.99; ROUGE-L: 18.42 → 18.47
Multi-table QA (TQABench, Table 4): - Qwen2.5-3B: 37.38 → 37.84; Qwen2.5-7B: 37.50 → 38.14
Cross-image interaction analysis: After scaling, interaction from Image 3 to Images 1 and 2 drops ~50%; intra-image interaction within Image 3 is preserved.
Ablation (Table 5, Qwen2.5-VL-3B on Mantis): removing delimiters → 53.92; adding M-RoPE without delimiters → 62.21; baseline with delimiters → 59.91; + Ours → 63.13.
Limitations¶
- \(\lambda\) is a fixed scalar hyperparameter; the paper does not provide a principled method for selecting it or show sensitivity analysis across tasks.
- The assumption underlying intra-image preservation (that within-image attention to the corresponding delimiter grows most after scaling) is stated as an assumption and not formally proven (see Appendix A.2).
- Gains on larger models (e.g., 32B) and on text-only tasks are modest in absolute terms, suggesting diminishing returns at scale.
- Analysis and experiments are conducted primarily on Qwen2.5-VL; generalization to architectures with different tokenization or attention structures (e.g., interleaved or diffusion-based VLMs) is unexplored.
- Replacing delimiter tokens with other special tokens was shown to eliminate triangular attention patterns, but the method's dependence on delimiter token identity (vs. position or magnitude) is not fully disentangled.
Relevance to Vision-Language Models¶
This work directly addresses a structural failure mode of transformer-based LVLMs in multi-image regimes—a setting increasingly common as VLMs are deployed in document understanding, visual comparison, and long-context reasoning tasks. The analysis of delimiter tokens as localized sink-like structures provides a new mechanistic lens on how LVLM attention organizes multi-modal context, complementing prior sink-token literature focused on text-only LLMs. The training-free, kernel-compatible nature of DelimScaling makes it immediately applicable on top of any existing LVLM (Qwen2.5-VL, InternVL3, LLaVA-OV) without any retraining. For VLM researchers, the finding that delimiter hidden-state magnitude controls boundary sharpness opens a broader design space for controlling cross-modality and cross-instance information flow.