LASER: A Corrective Lens for LVLMs via Visual Attention Preservation and Sink Suppression¶
🕒 Published (v1): 2026-07-02 04:59 UTC · Source: Arxiv · Venue: ECCV 2026 · link
Why this paper was selected
ECCV 2026; LASER attention-sink suppression fixes visual forgetting in long LVLM decoding
Ask a follow-up
Open an assistant pre-loaded with this paper's context.
TL;DR¶
LASER is a post-training framework that addresses visual forgetting in large vision-language models (LVLMs) — the progressive drift of attention away from visual tokens during long-horizon reasoning. It introduces two complementary GRPO rewards that jointly regulate the temporal trajectory and intra-visual distribution of attention, achieving consistent gains across eight multimodal benchmarks.
Problem¶
During extended chain-of-thought generation, LVLMs progressively reduce attention to visual tokens (visual forgetting), degrading grounding. Existing remedies treat this as a late-stage decay problem or apply heuristic reminders, but overlook two causal factors: (1) early-stage attention decay is disproportionately damaging because errors propagate autoregressively; and (2) even when total visual attention is preserved, it collapses onto a small set of semantically uninformative visual sink tokens, preventing effective grounding of task-relevant regions.
Method¶
LASER augments GRPO training with two attention-shaping rewards applied to Qwen-2.5-VL-7B-Instruct over 45K RL samples.
Sink token identification. Tokens with anomalously large hidden-state activation magnitudes (exceeding a top-\(k\) percentile threshold \(\eta\) across massive-activation dimensions \(\mathcal{D}^*\)) are labeled sink tokens \(\mathcal{S}\); the remainder are non-sink tokens \(\mathcal{V}' = \mathcal{V} \setminus \mathcal{S}\).
Visual Grounding Reward \(R_\text{vis}\). At each generation step \(t\), the non-sink Visual Attention Proportion is: $\(\alpha'_t = \frac{1}{LH}\sum_{l=1}^L\sum_{h=1}^H\sum_{j\in\mathcal{V}'} a_{t,j}^{(l,h)}\)$ The reward penalizes decay from the peak attention level: $\(R_\text{vis} = \sum_{t=1}^T \exp\!\left(-\lambda\left(1 - \frac{\alpha'_t}{\alpha'_{\max}}\right)\right) + \beta\)$ where \(\beta\) discourages reward hacking through redundant reasoning.
Sink Suppression Reward \(R_\text{supp}\). Penalizes excessive per-token attention concentration on sink tokens relative to the visual-token average: $\(R_\text{supp} = \exp\!\left(-\gamma \cdot \max\!\left(0,\, \frac{\bar{a}_\mathcal{S}}{\bar{a}_\mathcal{V}} - 1\right)\right)\)$
Combined objective. Visual rewards are gated on correctness to prevent rewarding grounding on wrong answers: $\(R = R_\text{acc} + \omega \cdot \mathbf{1}_{[R_\text{acc}>0]} \cdot (R_\text{vis} + R_\text{supp})\)$
Key Contributions¶
- Causal analysis via KV-cache scaling interventions demonstrating that early-stage visual attention suppression (at the \(\frac{1}{8}\) response position) degrades MMStar accuracy by 13.9% vs. 2.0% for \(\frac{3}{4}\)-stage suppression — confirming strong temporal asymmetry.
- Empirical finding that visual sink tokens absorb 2–3× more per-token attention than the visual average, and that redistributing attention away from sinks outperforms uniform magnitude scaling.
- LASER framework: trajectory-level, training-time attention shaping via dual GRPO rewards (\(R_\text{vis}\), \(R_\text{supp}\)) targeting distinct pathologies of visual forgetting.
- Correctness-gating of attention rewards, ensuring grounding signals are only reinforced on correct reasoning trajectories.
Results¶
- MathVision: 44.6 vs. best prior open-source 33.9 (Reflection-V) — +12.7% absolute, largest reported margin.
- MMStar: 64.1, best among all open-source models; outperforms Reflection-V (63.0) and VAPO-Thinker (not reported on MMStar in table).
- HallusionBench: 60.7, +3.3% absolute over the primary baseline Qwen2.5-VL-7B (53.6 → 60.7).
- MathVerse: 55.2 vs. 53.3 (Reflection-V) and 52.4 (VisionR1).
- LogicVista: 51.2 vs. 50.9 (OpenVLThinker), second-best open-source.
- Reported aggregate gains: +1.9% on visual reasoning benchmarks and +5.7% on visual perception tasks vs. strong baselines.
- Evaluated on 8 benchmarks (MathVista, MathVerse, MathVision, WeMath, MMMU, MMStar, LogicVista, HallusionBench) using Qwen2.5-VL-7B-Instruct as the backbone; ablation confirms both \(R_\text{vis}\) and \(R_\text{supp}\) are individually necessary.
Limitations¶
- Single backbone evaluated (Qwen2.5-VL-7B-Instruct); generalization to other LVLM architectures is unverified.
- Sink token identification relies on a manually set threshold \(\eta\) (top-\(k\) percentile) and fixed massive-activation dimensions, which may not transfer across architectures without re-tuning.
- The penalty term \(\beta\) in \(R_\text{vis}\) to prevent reward hacking through redundant reasoning is introduced but its interaction with response length is not deeply analyzed.
- Training requires extracting per-layer, per-head attention weights at every generation step, imposing non-trivial computational overhead during RL training.
- Evaluated only on static image benchmarks; video or multi-image settings with even longer contexts are not addressed.
Relevance to Vision-Language Models¶
LASER directly addresses a structural failure mode of RL-trained LVLMs — visual forgetting — which becomes increasingly consequential as models are pushed toward longer reasoning chains. The empirical finding that early-stage attention decay is causally dominant (not merely correlational) refines the community's understanding of how visual grounding errors propagate during CoT generation, which has broad implications for RLVR training recipes. The sink suppression mechanism complements magnitude-focused approaches (e.g., Reflection-V, VAPO-Thinker) by targeting attention distribution, establishing that total visual attention mass is a misleading proxy for grounding quality. For researchers tracking VLMs, this work advances the methodological toolkit for attention-aware post-training and provides diagnostic tools (KV-cache scaling interventions, VAP trajectory analysis) applicable beyond LASER itself.