RegionReasoner: Region-Grounded Multi-Round Visual Reasoning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Cees Snoek; iterative region-grounded multi-round visual reasoning beyond single-pass VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RegionReasoner is a reinforcement-learning framework that extends single-turn structured VLM reasoning to multi-round dialogues by requiring explicit bounding-box citation inside <think> blocks and enforcing global–local semantic consistency across turns. The authors also introduce RegionDial-Bench, a multi-round detection and segmentation benchmark built from RefCOCO+/g. RegionReasoner-7B outperforms strong single-turn and multi-round baselines, with gains widening at later dialogue turns.

Problem

Existing VLMs either do single-turn reasoning (VisionReasoner) or multi-round interaction without explicit reasoning traces (SegLLM). Naively extending single-turn methods to multi-round settings causes two failure modes: (1) reference propagation is brittle because prior regions are never explicitly cited, making coordinate hallucinations hard to detect; (2) semantic drift accumulates across turns when there is no mechanism to align global scene descriptions with local region evidence.

Method

RegionReasoner structures each dialogue turn as a four-block auto-regressive output: <scene> (global caption \(s_t\)), <focus> (caption restricted to the referenced region \(f_t\)), <think> (reasoning trace \(h_t\) that must verbatim cite required bounding boxes \(B_t^{\text{ref}}\)), and <answer> (JSON-encoded detection box or segmentation point). Constrained decoding enforces tag schema and JSON validity.

Two novel RL rewards are added on top of base geometric rewards inherited from VisionReasoner:

1. Reference citation reward \(R_{\text{ref}}(t)\): rewards fraction of \(B_t^{\text{ref}}\) mentioned in \(h_t\) and penalizes hallucinated coordinates (\(\eta = 0.5\) scale-down), clipped to \([0, 2]\).

\[R_{\text{ref}}(t) = \lambda\,\text{kw}(h_t) + \mu\,\frac{|S(h_t)\cap B_t^{\text{ref}}|}{\max(|S(h_t)|,1)}\]

1. Global–local consistency reward \(R_{\text{cons}}(t)\): asymmetric keyword overlap \(\text{Ov}(X,Y)=|K(X)\cap K(Y)|/\max(|K(X)|,1)\) aligns keywords extracted from \(s_t\) and \(f_t\) with \(h_t\), plus a spatial/localization lexicon prior \(\ell(h_t)\).

\[R_{\text{cons}}(t) = w_s\,\text{Ov}(s_t,h_t) + w_f\,\mathbb{1}[B_t^{\text{ref}}\neq\emptyset]\,\text{Ov}(f_t,h_t) + w_\ell\,\ell(h_t)\]

The episode return sums \(R(t)=R_{\text{base}}(t)+\alpha R_{\text{ref}}(t)+\beta R_{\text{cons}}(t)\) over turns, and policy optimization uses GRPO with GAE advantages over multi-turn rollouts.

RegionDial-Bench is built from RefCOCO+ (715 images, 2355 turns) and RefCOCOg (1580 images, 4405 turns) by consolidating same-image expressions into dialogues and rewriting later turns to reference prior predicted boxes, so early-turn errors propagate.

Key Contributions

• RegionDial-Bench: first multi-round benchmark covering both referring detection (AP50) and referring segmentation (gIoU) with per-turn evaluation and error propagation in test splits.
• Reference-grounded thinking: explicit bbox citation requirement in <think> with a citation+hallucination penalty reward, enabling automatic verification of evidence use.
• Global–local consistency reward: lightweight keyword-overlap signal aligning global scene and local region captions with the reasoning trace, reducing semantic drift across turns.
• RegionReasoner-7B: Qwen2.5-VL-7B backbone fine-tuned with GRPO, no task-specific heads; detection and segmentation unified through JSON <answer>.

Results

Multi-round detection (7-round, AP50 avg): - RegionReasoner-7B vs. VisionReasoner-7B: 80.7 vs. 74.8 on RefCOCO+; 78.2 vs. 73.6 on RefCOCOg - vs. Seg-Zero-7B: +7.6 (RefCOCO+), +7.1 (RefCOCOg) - Late-turn margins over VisionReasoner-7B on RefCOCO+: R5 +5.6, R6 +11.8, R7 +17.7

Multi-round segmentation (7-round, gIoU avg): - RegionReasoner-7B vs. VisionReasoner-7B: 69.6 vs. 64.3 on RefCOCO+; 66.5 vs. 59.9 on RefCOCOg - vs. SegLLM-7B: +8.9 (RefCOCO+), +9.8 (RefCOCOg)

Ablation (detection, RefCOCO+ multi-round): base-only 74.8 → +Ref-cite 78.9 → +Consist. 80.2 → +Logic prior 80.7; single-round performance is essentially flat (87.9→87.7), confirming rewards are multi-round-specific.

Limitations

• RegionDial-Bench is automatically constructed from RefCOCO+/g with rule-based spatial templates, limiting linguistic and scene diversity.
• Segmentation uses sparse point outputs rather than dense predicted masks, which may underestimate true mask quality.
• Reward functions rely on lightweight heuristics (keyword lemmatization, spatial lexicons) that may miss semantic mismatches not captured by surface overlap.
• Evaluation is confined to the authors' own benchmark; no results on established multi-round benchmarks (e.g., SegLLM's original splits) are reported.
• Training requires 4×H100 GPUs (~10 hours), and multi-round rollout generation adds compute overhead over single-turn RL.

Relevance to Vision-Language Models

RegionReasoner demonstrates that RL reward shaping can be targeted specifically at intermediate reasoning steps—not just final outputs—to stabilize multi-round VLM grounding, a direction distinct from most prior RL-for-VLM work that operates single-turn. The reference-citation mechanism is a concrete operationalization of chain-of-thought verifiability for spatial tasks, offering a template for reducing hallucination in structured VLM outputs. For researchers tracking VLMs, the paper's treatment of semantic drift under iterative context is directly relevant to long-horizon multimodal agents and dialogue-grounded assistants. RegionDial-Bench also fills a gap between single-turn grounding benchmarks and full embodied-interaction scenarios.