Skip to content

GRIT: Teaching MLLMs to Think with Images

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GRIT trains multimodal large language models (MLLMs) to produce reasoning chains that interleave natural language with explicit bounding box coordinates, grounding each reasoning step in specific image regions. The method uses a novel RL algorithm (GRPO-GR) that requires only image-question-answer triplets—no bounding box annotations or reasoning chain supervision—enabling effective training with as few as 20 samples.

Problem

Open-source visual reasoning models generate purely textual chain-of-thought when processing multimodal inputs, failing to explicitly reference visual evidence during reasoning. This produces unverifiable, visually ungrounded rationales. Prior approaches that incorporate visual grounding either require expensive dense annotations linking reasoning steps to bounding boxes (VisCoT, CogCoM) or treat grounding and reasoning as separate phases rather than a unified generative process.

Method

Grounded Reasoning Paradigm. Given image \(I\) and query \(q\), the model generates an interleaved output \((c, a)\) where reasoning chain \(c\) freely mixes natural language tokens \(T\) and bounding box quadruplets \(B\) (normalized integer coordinates). Crucially, after generating bounding boxes, the model receives no additional pixel crops—it must internally interpret its own grounding actions to inform subsequent tokens. The chain is structured with <think>…</think> and <rethink>…</rethink> token pairs.

GRPO-GR. Built on Group-Relative Policy Optimization (GRPO), it optimizes policy \(\pi_\theta\) using three reward components: - Format reward \(r_\text{format} = s_\text{st} + s_\text{bf}\): rewards correct use of <think>/<rethink> token pairs (+0.5 each) and presence of ≥1 syntactically valid bounding box (+0.5). - Counting reward \(r_\text{count}\) (optional): +0.5 if the number of generated bounding boxes matches the ground-truth object count. - Answer accuracy reward \(r_\text{ans} = s_\text{GPT} + 0.1 s_\text{BLEU}\): binary GPT-4o judge correctness plus down-weighted BLEU-1.

Group-normalized advantage \(A_i = (r_i - \mu) / (\sigma + \delta)\) drives the standard clipped GRPO objective with KL penalty against a reference policy.

Models trained: Qwen2.5-VL-3B and InternVL3-2B, using 20 triplets from VSR and TallyQA, 200 steps, batch 128, 8Ă—A100 80GB, ~12 hours per model.

Key Contributions

  • Grounded reasoning paradigm: interleaved NL + bounding-box coordinate generation within a single autoregressive chain, without multi-turn pixel augmentation.
  • GRPO-GR: RL training from task-level rewards only (answer accuracy + format), requiring no intermediate annotation.
  • Extreme data efficiency: functional grounded reasoning acquired from 20 image-question-answer triplets.
  • Vision-Language Reasoning Cross-Modal Correlation metric: GPT-4o-based Set-of-Mark evaluation of coherence between generated bounding boxes and co-occurring text.
  • Demonstrated unification of pre-existing but disconnected grounding and reasoning abilities in base MLLMs.

Results

  • Qwen2.5-VL-3B GRIT vs. baselines (GPT-as-judge ACC / Grounding IoU):
  • VSR: 72.9 ACC / 0.325 GIoU vs. 59.7 / 0.216 (few-shot SFT), 49.5 / 0.00 (direct query)
  • TallyQA: 47.8 / 0.447 vs. 44.5 / 0.284 (few-shot SFT)
  • GQA: 62.8 / 0.485 vs. 64.6 / 0.475 (few-shot SFT)
  • MME: 89.3 / 0.398 vs. 68.3 / 0.391 (few-shot SFT)
  • MathVista: 48.2 ACC vs. 17.4 (few-shot SFT), 43.0 (direct query)
  • OVDEval: 82.0 ACC / 56.0 GIoU vs. 62.3 / 7.8 (few-shot SFT)
  • InternVL3-2B GRIT achieves similar gains: VSR 64.9 ACC / 0.495 GIoU; GQA 63.2 / 0.457.
  • GRIT-trained models outperform Zero-shot ICL and Few-shot SFT on the cross-modal correlation metric; human-written chains remain a higher ceiling.
  • Attention analysis: presence of bounding boxes in <think> segments increases average visual token attention during subsequent <rethink> generation.
  • Scaling (20→500→7000 samples): in-domain accuracy improves substantially; out-of-domain gains are more modest, with diminishing returns.

Limitations

  • Evaluated only on small models (2B–3B parameters); scaling to larger MLLMs is left to future work.
  • Out-of-domain generalization is limited; data diversity matters more than volume for generalization.
  • Bounding box semantic accuracy is not directly rewarded—the model may generate syntactically valid but semantically misplaced boxes.
  • No pixel feedback from generated regions during inference; the model must rely entirely on internal representation of the coordinates.
  • RL with verifiable rewards biases existing reasoning patterns rather than instilling fundamentally new capabilities; pre-training quality remains a ceiling.
  • Human cross-modal correlation remains notably higher than GRIT-trained models.

Relevance to Vision-Language Models

GRIT directly addresses a structural deficiency in current VLM reasoning: the disconnect between a model's grounding capability (locating image regions) and its chain-of-thought generation (reasoning about them). By framing grounded reasoning as a unified autoregressive task and using RL with lightweight format rewards, GRIT shows that tightly integrating visual evidence into reasoning chains is achievable with minimal data—a meaningful result for the broader effort to make VLM reasoning transparent and verifiable. The work also extends the DeepSeek-R1/GRPO paradigm to multimodal settings in a way that preserves the base model's generalist abilities, which is a key desideratum for practical VLM deployment.