Skip to content

ERGO: Efficient High-Resolution Visual Understanding for Vision-Language Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ERGO efficient high-resolution VLM processing via selective token allocation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ERGO introduces a coarse-to-fine pipeline for efficient high-resolution visual understanding in LVLMs, where a downsampled image is first analyzed to identify task-relevant regions, then only those regions are re-encoded at full resolution. Training uses a reinforcement learning framework with a novel Task-driven Contextual Exploration (TCE) reward that encourages reasoning-driven perception rather than precise object localization. ERGO surpasses Qwen2.5-VL-7B on V* by 4.7 points while using only 23% of the vision tokens and achieving a 3× inference speedup.

Problem

Existing LVLMs processing high-resolution images incur prohibitive computational cost from large vision token counts. Naive resolution reduction discards fine-grained detail. Existing "thinking with images" models follow a perception-driven reasoning paradigm—they first localize targets then reason—which fails when input images are downsampled because visually indiscernible objects cannot be reliably localized. No prior work explicitly trains models to handle the perceptual uncertainty inherent in low-resolution inputs for coarse-to-fine pipelines.

Method

ERGO trains a 7B policy model (Qwen2.5-VL-7B-Instruct) via GRPO in a two-stage multi-turn pipeline:

  1. Coarse stage: The policy receives a downsampled image \(I_\text{orig}\) and query \(q\), generating bounding-box coordinates \(o_\text{region}\) and a reasoning trace.
  2. Fine stage: The predicted region \(I_\text{region} = \text{crop}(I_\text{orig}, o_\text{region})\) is re-encoded at original resolution; the policy then answers conditioned on both turns.

The reward function \(R = r_\text{TCE} + r_\text{acc} + r_\text{format}\) combines: - Region-verification reward \(r_\text{region} = \mathbb{1}[\text{match}(R(\cdot|I_\text{region},q),\, o_\text{GT})]\): a frozen 72B reward model evaluates only the cropped region (not the original image) against the ground-truth answer, incentivizing self-contained crops. - Box adjustment reward \(r_\text{box} = \mathbb{1}\!\left[\frac{\text{Area}(I_\text{region})}{\text{Area}(I_\text{orig})} \leq \gamma\right]\) with \(\gamma = 0.6\), preventing the degenerate "crop the whole image" policy. - TCE reward \(r_\text{TCE} = \alpha \cdot r_\text{region} + \beta \cdot r_\text{box}\) (\(\alpha=1, \beta=0.5\)). - Standard accuracy (\(r_\text{acc}\)) and format (\(r_\text{format}\)) rewards.

A key design choice is excluding \(I_\text{orig}\) from the reward model, forcing the model to select crops that are independently sufficient for answering the query.

Key Contributions

  • Two-stage coarse-to-fine reasoning pipeline that explicitly targets vision-token efficiency while preserving fine-grained visual detail.
  • Region-verification reward that evaluates crops without the original image, promoting self-contained and informative region selection.
  • Box adjustment reward with a data-driven threshold (\(\gamma=0.6\)) that prevents degenerate full-image cropping.
  • Demonstration that reasoning-driven perception (leveraging contextual cues to locate invisible objects) outperforms perception-driven reasoning (requiring clear visual signal before reasoning) under low-resolution constraints.
  • State-of-the-art performance on six high-resolution VQA benchmarks under pixel constraints with fewer tokens than competing methods.

Results

  • V* benchmark (1280×28×28): ERGO scores 83.8 vs. Qwen2.5-VL-7B-Inst at 77.0; vs. best competing efficiency-oriented method MGPO at 77.5; vs. best non-efficiency method MiniO3 at 81.2.
  • V* benchmark (640×28×28): ERGO scores 81.7 vs. Qwen2.5-VL-7B at 56.5; vs. MiniO3 at 74.9; uses 1,025 vision tokens vs. 4,471 for the full-resolution baseline (≈23%).
  • Average across 6 benchmarks (1280×28×28): ERGO 58.4 vs. MGPO 52.6, VisionThink 49.4, MiniO3 52.2, Qwen2.5-VL-7B-Inst 52.4.
  • Latency (vLLM, single H100, V*): ERGO at 1.61 s/sample vs. DeepEyes (1 tool call) at 2.18 s, MiniO3 (1 tool call) at 2.03 s, and Qwen2.5-VL-7B at 4.89 s.
  • Pareto-optimal: At 640×28×28, ERGO outperforms all baselines evaluated at 1280×28×28 while using fewer tokens.
  • Conventional benchmarks: ERGO maintains or improves base model on CV-Bench, MMVP, Hallusion-Bench, POPE, MMBench, AI2D, ChartQA.
  • Ablation: TCE-reward-only training (no explicit answer optimization) still outperforms accuracy-reward-only training (average 54.9 vs. 53.5), demonstrating region quality is the bottleneck.

Limitations

  • Single tool call constraint: trained for one crop operation per query; multi-hop or spatially complex queries may require iterative cropping, which is excluded.
  • Reward model dependency: region-verification uses a frozen 72B model (Qwen2.5-VL-72B-Instruct), introducing significant training infrastructure cost.
  • Fixed \(\gamma = 0.6\) threshold is data-driven from specific training sets; generalization to domains with systematically large or unusual object sizes is untested.
  • Evaluation limited to 7B policy scale; it is unclear whether TCE reward benefits scale or degrade at larger model sizes.
  • Training data restricted to ArxivQA and V* training splits, potentially limiting domain coverage.

Relevance to Vision-Language Models

ERGO directly addresses the token efficiency bottleneck in high-resolution VLMs, a central challenge as models are deployed on dense-detail tasks (document understanding, medical imaging, satellite imagery). The "reasoning-driven perception" paradigm it establishes—where contextual language and visual priors guide spatial attention rather than requiring clear pixel signal—extends the "thinking with images" line (DeepEyes, PixelReasoner) toward practical deployment by decoupling reasoning quality from input resolution. The reward engineering insights (excluding the original image from the reward model, regularizing crop area) are directly applicable to any RL-trained LVLM adopting multi-turn visual grounding. For VLM researchers, the result that TCE reward alone outperforms accuracy reward alone challenges the standard assumption that final-answer supervision is the primary signal needed for visual reasoning.