Skip to content

Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding

🕒 Published (v1): 2026-03-24 05:37 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; identifies instruction-relevant image regions for information-dense VLM inputs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PinPoint is a two-stage framework for Large Vision-Language Models (LVLMs) that identifies instruction-relevant image regions via contrastive alignment before re-encoding them at high resolution, rather than pruning tokens post-hoc. It simultaneously improves accuracy and reduces FLOPs on information-dense document/infographic VQA tasks, outperforming the vanilla (all-token) baseline on InfoVQA by 18.5% while using only 65.3% of the compute.

Problem

Existing token pruning methods rely on LLM attention weights to discard visual tokens, but these maps are unreliable in complex, text-rich images (infographics, document layouts), causing hallucinations, semantic fragmentation (multi-token visual elements broken up), and contextual entanglement (answer-relevant tokens polluted by global self-attention). No prior method addresses instruction-aware region selection before the LLM sees visual tokens; instead they all prune after-the-fact.

Method

PinPoint operates in two stages:

Stage 1 — Region Selection via Instruction-Region Contrastive Alignment. After the ViT encoder + projector, visual tokens \(\mathbf{V} \in \mathbb{R}^{T \times d}\) are reshaped into a 2D grid and processed by a sliding window (size \(W \times H\), stride \(S\)) to produce \(N_r\) region-level representations \(\mathbf{R} = \{\mathbf{R}_i\}_{i=1}^{N_r}\). Text is embedded as \(\mathbf{T} \in \mathbb{R}^{M \times d}\). A small set of \(K\) learnable guidance queries \(\mathbf{E} \in \mathbb{R}^{K \times d}\) attends to each region via scaled dot-product attention to produce visual-aware embeddings \(E^v_i\), and attends to the text to produce \(E^t\). Cosine similarity between \(E^v_i\) and \(E^t\) ranks regions; the adaptive top-\(k\) regions are greedily selected until they cover a predefined fraction \(r\) of the total image area.

Stage 2 — Region Refinement. Selected regions are cropped from the original image and re-encoded independently by the ViT, stripping the global self-attention context that contaminates tokens with irrelevant information. The resulting compact, re-encoded tokens are fed to the LLM for final answer generation.

Training. Only the \(K\) learnable queries and two small MLP projection layers are trained (all other components frozen) using a combined loss: $\(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{inter}} + \lambda \mathcal{L}_{\text{intra}}\)$ \(\mathcal{L}_{\text{inter}}\) is a symmetric cross-modal contrastive loss pulling paired \((E^v_{\text{pos}}, E^t)\) together across batch negatives. \(\mathcal{L}_{\text{intra}}\) separates the positive region from irrelevant regions within the same image.

Dataset. New bounding-box annotations are provided for InfographicVQA, SPDocVQA, and MPDocVQA that label both the answer element and surrounding supporting elements, generated via an OCR + VLM pipeline with minimal manual effort.

Key Contributions

  • PinPoint two-stage framework: lightweight instruction-region alignment (query + MLP only, frozen backbone) followed by isolated re-encoding of selected regions.
  • Adaptive top-\(k\) region selection based on cosine similarity between learned visual and text representations, covering at least fraction \(r\) of image area.
  • Dual contrastive training: inter-modal loss across batch + intra-image loss separating relevant from irrelevant regions.
  • New multi-element bounding-box annotations for InfoVQA, SPDocVQA, and MPDocVQA capturing answer + supporting context regions.
  • Demonstrated compatibility with two backbones: LLaVA-NeXT-7B and Qwen2-VL-7B.

Results

  • InfoVQA (LLaVA-NeXT-7B): PinPoint ANLS 0.3024 vs. vanilla 0.2552 (+18.5%) at 65.3% FLOPs; outperforms SparseVLM (0.2428) by 24.5% with 4.9% fewer FLOPs.
  • SPDocVQA (LLaVA-NeXT-7B): ANLS 0.6472 vs. vanilla 0.6628 (slight drop) at 55.0% FLOPs; beats all pruning baselines (FastV: 0.6099, PDrop: 0.5507, SparseVLM: 0.5726).
  • MPDocVQA (LLaVA-NeXT-7B): ANLS 0.3866 vs. vanilla 0.3758 at 56.5% FLOPs; best among all methods.
  • GQA (LLaVA-NeXT-7B): Accuracy 0.7608 vs. vanilla 0.7598 at 59.1% FLOPs; best among all methods.
  • Latency (LLaVA-NeXT-7B): PinPoint 381.6 ms vs. vanilla 569.5 ms; comparable to FastV (386.0 ms) and faster than SparseVLM (471.1 ms).
  • InfoVQA (Qwen2-VL-7B): ANLS 0.7140 vs. vanilla 0.7399 at 55.5% FLOPs; vs. FastV 0.5130, PDrop 0.4506.
  • SPDocVQA (Qwen2-VL-7B): ANLS 0.8977 vs. vanilla 0.9359 at 44.2% FLOPs; outperforms FastV (0.8106) by large margin.
  • PinPoint + Global (re-encoded regions + low-res global tokens): consistently best accuracy across all InfoVQA/SPDocVQA/MPDocVQA/GQA at 72–88% of vanilla FLOPs.
  • ViCrop (iterative region-focusing baseline): 378–452% of vanilla FLOPs with lower accuracy than PinPoint.

Limitations

  • On SPDocVQA and some Qwen2-VL benchmarks, PinPoint accuracy still falls below the vanilla all-token baseline, indicating region selection is not perfect.
  • Sliding window region granularity is fixed by hyperparameters (\(10 \times 10\) window, stride 7); may not generalize to arbitrarily structured or very large documents without tuning.
  • The approach requires new region-level ground-truth annotations for training; the automated pipeline depends on OCR and advanced VLMs, introducing potential annotation noise.
  • Evaluation confined to four benchmarks (InfoVQA, SPDocVQA, MPDocVQA, GQA); generalization to natural scene understanding or video tasks is not assessed.
  • Latency measurements for Qwen2-VL-7B show PinPoint (555.5 ms) is slower than the vanilla (907.4 ms) but also slower than FastV (532.7 ms), suggesting re-encoding overhead.

Relevance to Vision-Language Models

PinPoint directly addresses a core efficiency bottleneck in high-resolution LVLMs: the quadratic cost of processing dense visual token sequences in document and infographic understanding. Unlike attention-based pruning, it introduces an explicit instruction-grounded region localization step — a design choice that connects to the broader trend of spatial grounding and selective attention in VLMs (e.g., region-conditioned generation, visual grounding models). The contrastive training of lightweight adapter queries over frozen backbones is a parameter-efficient paradigm relevant to any researcher studying VLM adaptation and token efficiency. The new multi-element annotation datasets and evaluation on both LLaVA-NeXT and Qwen2-VL provide reusable benchmarks for the community studying information-rich image understanding.