Skip to content

H-GRPO: Permutation-Invariant Reinforcement Learning for Grounded Visual Reasoning

🕒 Published (v1): 2026-06-29 07:51 UTC · Source: Arxiv · link

Why this paper was selected

GRPO + grounded VLM reasoning; directly relevant to RL-based VLM improvement

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

H-GRPO trains VLMs to decompose visual reasoning into sub-questions, each grounded in a spatial evidence bounding box, then uses a Hungarian matching-based reward that is permutation-invariant across different reasoning orders. The approach outperforms vanilla GRPO and prior methods on both answer accuracy and interpretability, especially for smaller VLMs and spatial reasoning tasks.

Problem

VLMs achieve high accuracy on benchmarks while remaining "black boxes" prone to hallucination and superficial shortcuts. Existing RL-based reasoning methods (Vision-R1, R1-VL, Reason-RFT) supervise textual reasoning with outcome or step-wise rewards but do not ground each step in localized visual evidence. Grounding methods like Visual-RFT and ViGoRL treat grounding as auxiliary rather than integral to reasoning. The core gap is that no prior work formulates reasoning as decomposable, explicitly grounded sub-steps with a reward that is invariant to valid but differently-ordered reasoning chains.

Method

  • Grounded reasoning decomposition: Each intermediate reasoning step is a triplet \(\tau_i = \langle q_i, a_i, b_i \rangle\) — a sub-question, a sub-answer, and an evidence bounding box \(b_i = [x_{\text{min}}, y_{\text{min}}, x_{\text{max}}, y_{\text{max}}]\). The model also outputs a reasoning path \(r\) and final answer \(a_{\text{final}}\).
  • H-GRPO reward: Three components — format reward (\(R_{\text{format}}\) enforces valid triplets/reasoning-path/final-answer tags), answer reward (\(R_{\text{answer}} = 1\) if final answer is correct), and a Hungarian matching reward \(R_{\text{HS}} = \max(0, S_{\text{HS}} - \gamma)\).
  • Hungarian matching score \(S_{\text{HS}}\): Builds a similarity matrix \(S_{ij} = \frac{1}{4}[E(\hat{b}_i, b^*_j) + \text{sim}_q(\hat{q}_i, q^*_j) + \text{sim}_a(\hat{a}_i, a^*_j) + \text{IoU}(\hat{b}_i, b^*_j)]\) between predicted and reference triplets, then finds the optimal bipartite matching via \(X^* = \arg\max_X \sum_i \sum_j S_{ij} x_{ij}\) under one-to-one constraints. The final answer reward is gated: \(R = \alpha R_{\text{format}} + \beta R_{\text{answer}} \cdot R_{\text{HS}}\), so the model only gets the correctness reward when grounded reasoning quality exceeds threshold \(\gamma\).
  • Training: Two-stage — supervised fine-tuning on the synthetic GVRS dataset (10k samples from Visual7W, Visual-CoT, A-OKVQA, ERQA, generated via GPT-4o/Gemini-3/Qwen3.5-OmniPlus with human + SAM3 verification), then GRPO-based RL with \(G=8\) rollouts per sample. Backbones: Qwen2.5-VL-3B and SmolVLM-2.2B.

Key Contributions

  • Grounded reasoning decomposition protocol representing each reasoning step as a sub-question, sub-answer, and spatial evidence bounding box
  • Scalable data synthesis pipeline for step-wise grounded reasoning traces without extensive manual annotation
  • H-GRPO, a permutation-invariant RL framework using Hungarian matching over semantic and spatial alignment
  • Evaluation protocol jointly measuring answer correctness, grounding fidelity, step-wise reasoning alignment, and visual evidence dependence

Results

  • In-domain (A-OKVQA, Visual7W): H-GRPO with SmolVLM-2.2B achieves 73.4% / 77.2%, the best for that backbone (vs. SFT 72.5%/74.6%, vanilla GRPO 71.7%/74.7%). With Qwen2.5-VL-3B: 82.8%/83.9% (best on Visual7W; GRPO achieves 83.4% on A-OKVQA).
  • Out-of-distribution: Best among compared methods on RealWorldQA (60.3%), RoboSpatial (70.2%), and MMStar (52.4%). Largest gain on RoboSpatial, showing spatial reasoning benefits most from explicit grounding per step.
  • Interpretability (LLM-as-a-Judge): H-GRPO scores 4.73/5 overall interpretability, 4.92 clarity, 4.94 relevance — vs. R1-VL (4.25), ViGoRL (3.81), Reason-RFT (3.02).
  • Ablation: SFT provides strong behavioral prior; vanilla GRPO gives inconsistent gains (e.g., SmolVLM A-OKVQA drops from 72.5% to 71.7%); H-GRPO yields the most consistent improvements, especially for smaller models and localization-sensitive tasks.

Limitations

  • Moderate improvement on knowledge-intensive benchmarks (MMMU: 45.2% vs. ViGoRL's 47.6%)
  • Requires reference reasoning traces for reward computation — no reference-free grounding reward
  • GVRS synthetic data quality depends on LLM generation pipeline; coverage is limited to four source datasets
  • Smaller backbones benefit more; stronger models (Qwen2.5-VL-3B) gain less from H-GRPO over SFT+GRPO alone
  • Hungarian matching adds computational overhead compared to standard sequence-level rewards

Relevance to Vision-Language Models

This work directly addresses the faithfulness gap in VLM reasoning by enforcing step-wise visual grounding via RL, bridging the line of process-reward models (R1-VL's StepGRPO, Vision-R1) with spatial localization. The permutation-invariant reward design is a practical contribution for any VLM pipeline where multiple valid reasoning trajectories exist. Most importantly, it demonstrates that explicit per-step grounding improves OOD spatial generalization — a critical capability gap in current VLMs — not just benchmark accuracy.