Skip to content

Improving Vision-language Models with Perception-centric Process Reward Models

🕒 Published (v1): 2026-04-27 15:08 UTC · Source: Arxiv · Venue: CVPR · link

Why this paper was selected

Process-level reward models for VLM RLVR; finer than outcome-only supervision

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Perceval is a perception-centric process reward model (PRM) that identifies hallucinatory token spans in VLM reasoning chains by checking image-grounded claims against visual evidence. It replaces GRPO's coarse sequence-level advantage with token-level penalties on detected hallucinations, and enables iterative truncation-based test-time refinement. Perception-targeted training generalizes surprisingly to math and chart reasoning.

Problem

RLVR methods like GRPO assign a single scalar reward to an entire reasoning chain, creating a hard credit-assignment problem: perceptual errors (hallucinated objects, wrong spatial relations, attribute mismatches) mid-chain receive the same gradient signal as correct tokens, so the model cannot localize or correct localized perception failures. Existing step-level PRMs require expensive manual annotation and lack a natural automatic verification signal for intermediate steps.

Method

Perceval (PRM training). Given a tuple \(\langle v, q, o \rangle\), Perceval extracts all image-grounded claims from the response, verifies each against the image, and outputs a structured Python list of erroneous verbatim spans inside <answer> tags. Training follows a 4-stage pipeline: (1) query selection from visual-search and referring-expression datasets; (2) rollout generation via Qwen2.5-VL to produce realistic hallucinations; (3) automated annotation with Gemini-2.5-Pro following the error-finding schema; (4) SFT on the resulting corpus.

Token-level advantage rescaling. Hallucinatory spans are located in response \(o_i\) by exact string match, yielding a binary mask \(M_i\). The sequence-level GRPO advantage \(\hat{A}_i\) is modulated token-by-token:

\[\hat{A}'_{i,t} := \hat{A}_i - \alpha \cdot m_{i,t} \cdot |\hat{A}_i|\]

where \(\alpha \in [0,1]\) controls penalty strength. Correct tokens (\(m_{i,t}=0\)) keep the original advantage; hallucinated tokens are downweighted (\(\hat{A}_i > 0 \Rightarrow \hat{A}'_{i,t} = \hat{A}_i(1-\alpha)\)) or further penalized (\(\hat{A}_i < 0 \Rightarrow \hat{A}'_{i,t} = \hat{A}_i(1+\alpha)\)). PRM intervention is applied only on perception-related training data; mathematical reasoning data uses plain GRPO.

Test-time scaling. Two iterative loops repeat up to \(k\) times: Truncate-then-Regenerate prunes the rationale at the first detected erroneous span and lets the policy regenerate from the clean prefix; Truncate-Thinking-then-Regenerate appends a short Perceval-generated reflection hint before regeneration.

Key Contributions

  • Perceval: an interpretable, perception-centric PRM that produces structured, span-level hallucination annotations without human step-level labeling.
  • Token-level advantage rescaling integrated into GRPO, enabling fine-grained credit assignment within each response.
  • Demonstration that perception-focused RL supervision transfers to math and chart reasoning without explicit training on those domains.
  • A test-time iterative truncation–regeneration strategy that outperforms majority voting on visual search benchmarks.

Results

  • 3B model vs. GRPO baseline: V_all 80.10 → 83.25 (+3.15), V_pos 86.95 → 90.43 (+3.48), MathVision 23.36 → 26.32 (+2.96), ChartQA 83.32 → 86.48 (+3.16), RealWorldQA 62.1 → 64.9 (+2.8).
  • 7B model vs. GRPO baseline: V_all 84.29 → 86.39 (+2.10), V_pos 82.89 → 86.84 (+3.95), MathVision 27.96 → 30.92 (+2.96), BLINK 53.55 → 54.49 (+0.94), MMStar 62.0 → 63.8.
  • 7B vs. tool-augmented baselines: Perceval-trained 7B matches or exceeds DeepEyes (tool-assisted) on V*_all (86.39 vs. 87.43) using intrinsic grounding alone.
  • Test-time scaling (3B, \(k=16\)): Truncate achieves V_all 89.53 vs. major voting 85.86; both PRM strategies consistently outperform major voting on V across \(k \in \{4,8,16\}\).
  • Optimal penalty strength: \(\alpha=0.1\) is best on all evaluated benchmarks; \(\alpha=0.3\) degrades performance.

Limitations

  • Perceval is only applied on perception-related training data; its effectiveness as a universal process reward on pure logical reasoning steps is not established.
  • The Truncate-Thinking strategy underperforms Truncate-only, attributed to insufficient reflective data in training, suggesting the approach is sensitive to policy training distribution.
  • Majority voting converges early on hard subsets (V*_pos), and the PRM loop shows diminishing gains at \(k=16\), suggesting a latency–accuracy ceiling.
  • Hallucination detection relies on exact string match to locate spans, which may fail under paraphrasing or tokenization artifacts.
  • Perceval itself is trained with Gemini-2.5-Pro–annotated data; annotation quality is not independently verified at scale.

Relevance to Vision-Language Models

This work directly addresses a known bottleneck in RLVR for VLMs — sparse, sequence-level rewards that cannot localize perceptual failures in multi-step reasoning chains — by introducing the first automatically-trainable, perception-centric PRM. The finding that perception-focused token-level supervision transfers to math and chart reasoning is significant, suggesting that grounding accuracy is a shared prerequisite across reasoning domains, not a domain-specific skill. The test-time truncation loop is a lightweight alternative to sampling-heavy test-time compute strategies and is directly applicable to any RLVR-trained VLM. Researchers tracking VLMs should note the competitive performance against tool-augmented models (DeepEyes, Pixel-Reasoner) from intrinsic capability improvements alone, pointing toward perception-centric training as a scalable paradigm.