Skip to content

Aligning What Vision-Language Models See and Perceive with Adaptive Information Flow

🕒 Published (v1): 2026-04-17 08:07 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

Adaptive info flow fixes VLMs that see but don't perceive; CVPR 2026

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLMs often "see" the correct image region via attention but fail to produce correct answers because text tokens spread attention across irrelevant visual tokens, corrupting the decoding path. This paper proposes AIF, a training-free method that modulates the causal mask at inference time to block text-to-irrelevant-visual-token connections, improving perception without retraining. It improves LLaVA-1.5-7B and Qwen2.5-VL-7B across VQA, OCR, grounding, counting, and hallucination benchmarks.

Problem

VLMs exhibit a seeing-perception misalignment: cross-attention maps confirm the model locates the relevant image region, yet the output is wrong. Prior fixes—visual cropping (ViCrop), high-resolution inputs, or causal-mask redesign via fine-tuning—require either heavy compute, extra inference passes, or retraining. The root cause, suboptimal cross-modal information flow during autoregressive decoding, had not been directly addressed as a test-time control signal.

Method

Token Dynamics. For each visual token \(v_i\), AIF runs a single auxiliary decoding step and records image-to-text attention values \(a^l_{i,j}\) (visual token \(i\) to text token \(j\), layer \(l\)). Token dynamics are: $\(D_{v_i} = \{d^l_{v_i}\}_{l=1}^{L}, \quad d^l_{v_i} = \max_j a^l_{i,j}\)$ with mean \(\mu_{v_i} = \frac{1}{L}\sum_l d^l_{v_i}\).

Entropy-based importance scoring. Object-relevant tokens show concentrated activation at certain layers; irrelevant tokens are erratically activated (high entropy). AIF computes: $\(\text{Ent}_{v_i} = \sum_{l=1}^{L} -\frac{d^l_{v_i}}{L \cdot \mu_{v_i}} \log\!\left(\frac{d^l_{v_i}}{L \cdot \mu_{v_i}}\right)\)$ High \(\text{Ent}_{v_i}\) → low importance → candidate for masking.

Adaptive thresholding. Candidate mask ratios (0.1–0.9 in steps of 0.1) are evaluated by measuring the Shannon entropy of the retained tokens' \(\mu_{v_i}\) distribution (Eq. 5). The ratio that maximally shifts the distribution away from the baseline (most centralized attention) is selected.

Causal mask modulation. The selected irrelevant visual tokens have their text-to-visual attention edges blocked in the causal mask \(C_{i,j} \leftarrow -\infty\) for those pairs. Vision-to-vision aggregation is left intact so no image content is discarded. The standard autoregressive decoding then proceeds with the modulated mask—one extra decoding pass total.

Key Contributions

  • Identifies suboptimal information flow (text tokens over-attending to irrelevant visual tokens) as the cause of seeing-perception misalignment in VLMs.
  • Proposes AIF: a training-free, architecture-agnostic causal mask modulation method guided by token dynamics and entropy.
  • Demonstrates that causal mask modulation is a broadly applicable control signal for improving perception, not just an interpretability tool.
  • Comprehensive evaluation across VQA, OCR, visual grounding, counting, and hallucination benchmarks on two representative VLMs (LLaVA-1.5-7B and Qwen2.5-VL-7B).

Results

General VQA (LLaVA-1.5-7B → +AIF): - V*: 42.4 → 50.3 (+7.9) - RealWorldQA: 55.6 → 60.5 (+4.9) - MMStar: 33.1 → 39.5 (+6.4)

General VQA (Qwen2.5-VL-7B → +AIF): - V*: 78.5 → 84.8 (+6.3) - RealWorldQA: 68.5 → 74.5 (+6.0) - MMStar: 63.9 → 70.9 (+7.0)

OCR: - TextVQA (LLaVA-1.5): 47.8 → 49.9 (+2.1) - SeedBench2-Plus (Qwen2.5-VL-7B): 70.4 → 76.5 (+6.1)

Visual Grounding (Qwen2.5-VL-7B, RefCOCO avg): +2.3 overall; outperforms Grounding-DINO-L on some splits.

Counting (CountBench): ~+2–3% for both models.

Hallucination (POPE): - LLaVA-1.5-7B: 85.4 → 88.7 - Qwen2.5-VL-7B: 87.8 → 89.5

Token selection quality (RefCOCO, top-20% recall of object tokens): Low-entropy selection (ours) vs. high-attention baseline: +20.9 pp small, +16.3 pp medium, +9.8 pp large objects.

Limitations

  • Requires one extra forward pass to compute token dynamics, adding latency (though the paper claims the rest of inference is unchanged).
  • Mask ratio search over 9 candidates (0.1–0.9) at each inference call introduces additional compute proportional to the candidate grid size.
  • Evaluated on two VLMs only (LLaVA-1.5-7B and Qwen2.5-VL-7B); generalizability to other architectures (e.g., encoder-decoder VLMs, proprietary models) is unverified.
  • The entropy-based importance measure relies on image-to-text attention, which may be unreliable in very long-context or highly compressed token regimes.
  • OCR gains are smaller than VQA/grounding gains, and the method's behavior on tasks requiring dense, uniform attention over the full image is not fully characterized.

Relevance to Vision-Language Models

This work directly addresses the core modality-alignment challenge in autoregressive VLMs: the decoder's attention to visual tokens does not automatically prioritize task-relevant content. By reframing information flow as a controllable inference-time signal rather than a fixed architectural property, AIF opens a new direction that is complementary to both data-centric (instruction tuning) and architecture-centric (mask redesign, high-res inputs) approaches. The entropy-based token dynamics formulation is lightweight and architecture-agnostic, making it readily applicable across the growing ecosystem of LLM-backbone VLMs. For researchers tracking VLMs, this also provides a new interpretability lens: the gap between localization (seen via attention) and generation (perceived via output) can now be quantified and closed within a single inference call.