Perceive-to-Reason: Decoupling Perception and Reasoning for Fine-Grained Visual Reasoning

🕒 Published (v1): 2026-07-01 17:24 UTC · Source: Arxiv · link

Why this paper was selected

Decouples perception/reasoning for fine-grained VQA — addresses critical VLM failure mode

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

P2R (Perceive-to-Reason) explicitly decomposes fine-grained visual reasoning into two sequential stages: a Perceiver that localizes question-relevant bounding boxes in high-resolution images, followed by a Reasoner that answers using the annotated image and cropped evidence. To train this decoupled formulation with only final-answer supervision, the authors introduce PRA-GRPO, a role-aware reinforcement learning strategy that alternates between optimizing each role while freezing the other. Built on Qwen3-VL-Instruct-2B/4B/8B, P2R achieves state-of-the-art results on fine-grained visual reasoning benchmarks.

Problem

Fine-grained visual reasoning over high-resolution images requires both localizing subtle cues and reasoning over them, yet current VLMs conflate these two sub-tasks. "Thinking with Images" methods (e.g., DeepEyes) interleave exploration and reasoning into a single noisy context, while visual search methods (e.g., ZoomEye) externalize perception into complex test-time pipelines that are hard to optimize. Neither formulation enables clean credit assignment: when the final answer is wrong, it is unclear whether the fault lies in localization or reasoning. A diagnostic experiment confirms that perception is the dominant bottleneck — Qwen3-VL-Instruct-4B jumps from 81.7% to 90.6% on V-Star when given oracle bounding boxes.

Method

Two-stage inference. Given an image-question pair \((I, Q)\), a single VLM with shared parameters \(\theta\) operates in two roles:

1. Perceiver \(\pi_p(\cdot;\theta)\): given a perception-oriented prompt \(\tilde{Q}_p = T_p(Q)\), predicts a set of bounding boxes \(B = \{B_k\}_{k=1}^K\) where each \(B_k = (x_1, y_1, x_2, y_2)\). The boxes are used to produce an annotated image \(I_a = \text{annotate}(I, B)\) and cropped patches \(I_c = \text{crop}(I, B)\).
2. Reasoner \(\pi_r(\cdot;\theta)\): given \((I_a, I_c, Q)\), generates the final answer \(Y\).

PRA-GRPO training. Standard GRPO on a unified answer-only objective entangles the two roles, making learning signals ambiguous. PRA-GRPO instead alternates between a perception phase (Perceiver optimized, Reasoner frozen) and a reasoning phase (Reasoner optimized, Perceiver frozen). In each phase, \(G = 8\) rollouts are sampled from the active role; the complementary role produces the final answer used for reward. A binary reward \(r_i = \mathbf{1}[Y_i = Y]\) and group-relative advantage

\[A_i = \frac{r_i - \text{mean}(\{r_j\}_{j=1}^G)}{\text{std}(\{r_j\}_{j=1}^G) + \epsilon}\]

drive the standard GRPO clipped-ratio objective with a KL penalty (\(\beta = 0.01\)). No ground-truth bounding box annotations are required.

Training data. 10K examples sampled from DeepEyes (3K), VisualProbe (3K), and ZwZ (4K). Training uses 4 H100 GPUs with one epoch per alternating phase.

Key Contributions

• P2R framework: a unified two-stage perceive-then-reason pipeline that makes evidence localization an explicit intermediate step rather than an implicit byproduct of generation.
• PRA-GRPO: role-aware alternating RL that converts final-answer correctness into an attributable per-role training signal without ground-truth box supervision.
• Empirical finding: P→R alternation order (Perceiver first) outperforms R→P (93.2% vs. 90.6% on V-Star), confirming that accurate localization is prerequisite to strong reasoning.
• Cross-role transfer under shared parameters: Perceiver-only training also improves Reasoner behavior and vice versa, validated by mixing checkpoints in Table 3.
• Training efficiency: PRA-GRPO scales faster per iteration than text-only GRPO (fitted slope 0.77 vs. 0.43 on MME-RealWorld-Lite).

Results

• V-Star: P2R-4B 93.2% (+11.5% vs. Qwen3-VL-4B baseline 81.7%); P2R-8B 93.7% (+9.9%); P2R-2B 84.3% (+9.4%).
• HR-Bench-4K: P2R-4B 81.9% (+8.1%); P2R-8B 81.5% (+6.7%); P2R-2B 75.1% (+4.7%).
• HR-Bench-8K: P2R-4B 80.5% (+13.5%); P2R-8B 82.6% (+12.5%); P2R-2B 74.8% (+10.2%).
• Average across three benchmarks: P2R-2B +8.1%, P2R-4B +11.0%, P2R-8B +9.7% over respective Qwen3-VL-Instruct backbones; P2R-8B achieves best average among all open-source models.
• MME-RealWorld-Lite (general multimodal): P2R-8B best overall (+7.0% vs. Qwen3-VL-8B); P2R-4B +7.1%; broad gains across OCR, remote sensing, document, and anomaly detection sub-tasks.
• Grounding generalization (ReasonSeg): P2R-4B improves Acc@0.5 by +0.5% (test) / +1.8% (val) over Qwen3-VL-4B with zero grounding-specific training data.
• P2R inference alone (no RL training) lifts Qwen3-VL-4B from 81.7% to 89.0% on V-Star; combining P2R inference with PRA-GRPO reaches 93.2%.
• Outperforms ZoomEye-7B (90.6% V-Star), DeepEyes-7B (90.1%), and PixelReasoner-7B (80.6%) despite using a 4B backbone for P2R-4B.

Limitations

• Relies on the Perceiver producing valid bounding boxes; when the Perceiver outputs empty boxes (as shown in the qualitative example), the pipeline can still fail — the fallback behavior is not formally characterized.
• Gains on HR-Bench-4K FSP (fine spatial perception) at the 2B scale are minimal (+0.3%), suggesting small models remain perception-limited even after training.
• Training data is assembled from three existing datasets without novel annotation; coverage of certain visual domains (e.g., medical, satellite at extreme resolution) is not assessed.
• Grounding task generalization gains are modest (+1.1% on ReasonSeg) compared to reasoning benchmark gains, indicating partial transfer to localization-primary tasks.
• All experiments use Qwen3-VL-Instruct as the backbone; generalization to other VLM families is not demonstrated.

Relevance to Vision-Language Models

P2R directly addresses a structural weakness in how VLMs handle high-resolution fine-grained tasks: by forcing the model to commit to an explicit intermediate localization step, the perceive-to-reason formulation avoids the noisy, entangled chain-of-thought that plagues "Thinking with Images" methods and the pipeline complexity of visual search approaches. PRA-GRPO is a practically important contribution because it learns better perception without any bounding-box annotation — only final-answer rewards — which is more scalable than supervised grounding approaches. For researchers tracking VLMs, this work exemplifies the growing trend of using structured RL (building on GRPO) to impose task-specific decompositions, and the oracle experiment provides a clean quantitative argument for why perception, not reasoning capacity, is the current ceiling for small-to-medium-scale VLMs on fine-grained benchmarks.

🔗 Related in this collection

• DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning