Skip to content

ActiveScope: Actively Seeking and Correcting Perception for MLLMs

🕒 Published (v1): 2026-06-23 08:17 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML 2026; training-free fine-grained perception correction for MLLMs — drop-in applicable

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ActiveScope is a training-free framework that improves MLLM fine-grained perception in high-resolution images by diagnosing two root causes of localization failure—contextual dominance and semantic bias—and addressing each with dedicated modules. It achieves 96.34% accuracy on V*Bench, outperforming the prior best by 3.15%.

Problem

MLLMs struggle with fine-grained visual perception in high-resolution images. Existing training-free methods (attention-map localization, coarse-to-fine search) suffer from two failure modes: (1) contextual dominance, where visually salient distractors overwhelm the attention landscape so the true target is never activated; and (2) semantic bias, where the first-answer-token attention aggregates global query semantics and fixates on the dominant concept, causing incomplete localization in multi-object queries. These failures cause both inaccurate single-target localization and incomplete coverage of multiple targets.

Method

ActiveScope is a plug-and-play, training-free wrapper applied at inference time around an existing MLLM (tested on Qwen3-VL and InternVL2.5).

Semantic Anchor Localization (SAL): Instead of using the attention map of the first predicted answer token (which carries global semantic bias), SAL prompts the LLM to enumerate needed objects \(O = \{O_1, \ldots, O_K\}\), then extracts cross-attention maps from the \(l\)-th decoder layer for each object's span tokens \(O_k = \{t_{s_k}, \ldots, t_{e_k}\}\):

\[A(O_k, X)^{(l)} = \frac{1}{|O_k|} \sum_{t_i \in O_k} \text{Softmax}\!\left(\frac{t_i^{(l)} \cdot (X^{(l)})^\top}{\sqrt{d_k}}\right)\]

A relative attention map is computed by dividing by a reference attention map from a generic instruction (e.g., "Describe the image"), suppressing background noise. Multi-scale sliding windows extract bounding boxes per object independently.

Interference-Suppressed Refinement (ISR): For each candidate box, the MLLM verifies presence of the target via a yes/no crop query. On failure, ISR updates a global attention mask \(M \in \mathbb{R}^{(N+L)\times(N+L)}\) to set attention scores between the target token span and the rejected visual tokens to \(-\infty\), forcing redistribution in the next inference pass. This cycles up to \(C \leq 3\) times.

Composition: Verified bounding boxes are unioned into a minimum enclosing crop \(X_\text{zoom}\); the MLLM receives \((X, X_\text{zoom}, q)\) for final answer generation.

Key Contributions

  • Systematic diagnosis identifying contextual dominance (primary cause of inaccurate localization) and semantic bias (cause of incomplete multi-target localization) through controlled ablation experiments.
  • SAL module using per-object semantic token attention maps to independently ground each target, decoupling global semantic bias.
  • ISR module with iterative verification-and-suppression cycles that correct localization via attention masking without parameter updates.
  • State-of-the-art results on V*Bench, HR-Bench 4K/8K with Qwen3-VL 4B and InternVL2.5 4B.

Results

All results are on 4B-scale base models unless noted.

  • V*Bench (Qwen3-VL 4B): 96.34% average (Attr 95.65%, Spatial 97.37%); base model 89.01%; best prior method (ICoT) 93.19%; +3.15% over prior best, +7.33% over base.
  • HR-Bench 4K (Qwen3-VL 4B): 78.13% avg; base 76.88%; ZoomEye 76.25%; ICoT 54.00%.
  • HR-Bench 8K (Qwen3-VL 4B): 74.75% avg; base 71.75%; ZoomEye 73.50%; ICoT 55.00%.
  • InternVL2.5 4B on V*Bench: 80.63% avg; base 68.59%; ZoomEye 79.58%; ICoT 60.21%.
  • Ablation: key token guidance (SAL core) achieves F1=65.0 on multi-object localization vs. 40.9 baseline and 41.2 with iterative localization from first-answer-token attention.
  • Interference Suppression converges within 3 iterations for the majority of correctable errors.

Limitations

  • Inference overhead: SAL requires an additional LLM forward pass to enumerate objects; ISR requires up to \(C=3\) extra inference cycles per target with attention mask recomputation.
  • Verification step adds another MLLM call per candidate box, multiplying latency proportionally to object count.
  • Performance tested only on Qwen3-VL and InternVL2.5; generalization to architectures with different attention mechanisms (e.g., those without accessible cross-attention maps) is not demonstrated.
  • \(C \leq 3\) cycle cap is a fixed heuristic; complex scenes with many distractors may not converge.
  • Bounding box extraction via multi-scale sliding windows is described only briefly; edge cases (overlapping targets, textureless regions) are not analyzed.

Relevance to Vision-Language Models

ActiveScope directly addresses a well-known VLM weakness: attention-based localization in high-resolution MLLMs degrades under distractor-rich scenes, which is a bottleneck for VQA, spatial reasoning, and document understanding tasks. The causal decomposition into contextual dominance vs. semantic bias provides a diagnostic vocabulary applicable to any attention-based MLLM. The training-free design is particularly relevant because it composes orthogonally with resolution scaling techniques (dynamic tiling in Qwen2-VL/InternVL), meaning it can be layered onto future model generations without retraining. For VLM researchers, the finding that per-object token attention maps vastly outperform first-answer-token attention for grounding (F1 65.0 vs. 40.9) has implications for how attention is interpreted and leveraged in chain-of-thought and tool-use agentic VLM pipelines.