MedVR: Annotation-Free Medical Visual Reasoning via Agentic Reinforcement Learning¶
🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link
Why this paper was selected
ICLR 2026; annotation-free medical VLM reasoning via agentic reinforcement learning
Ask a follow-up
Open an assistant pre-loaded with this paper's context.
TL;DR¶
MedVR is a reinforcement learning framework that enables medical VLMs to interleave textual chain-of-thought with executable image-manipulation tools (e.g., zoom-in) during inference, without requiring any human annotation of intermediate reasoning steps. Two novel mechanisms — Entropy-guided Visual Regrounding (EVR) and Consensus-based Credit Assignment (CCA) — together create a fully self-supervised curriculum for visual grounding. MedVR achieves state-of-the-art results on six public medical VQA benchmarks.
Problem¶
Existing RLVR-trained medical VLMs reason exclusively in the textual domain, failing to ground inferences in visual evidence. This causes visual hallucinations and language shortcuts — the model answers from textual priors while ignoring image content. Adapting general-domain interactive visual reasoning approaches (e.g., DeepEyes, Pixel-Reasoner) to medicine is blocked by the scarcity and cost of fine-grained grounding annotations (bounding boxes for pathological findings) needed for cold-start supervision.
Method¶
MedVR casts training as policy optimization over a VLM agent (Qwen2.5-VL-7B backbone) using GRPO. The agent's action space combines token generation for chain-of-thought and invocation of a Zoom-in tool that crops a specified bounding box, encodes the crop as tokens, and feeds it back into context.
Entropy-guided Visual Regrounding (EVR): During rollout generation, token-level entropy \(H_t = -\sum_j p_{t,j} \log p_{t,j}\) is monitored. The entropy increase relative to a baseline, \(\Delta H_\text{tool} = H_\text{tool} - H_\text{base}\), flags spatial uncertainty while the model specifies tool coordinates. A branching probability \(P = P_\text{base} + \gamma \Delta H_\text{tool}\) governs adaptive forking: when \(P\) triggers, the current generative state is cloned and a new parallel trajectory explores a different bounding-box choice from the same context. Half the rollout budget is reserved for these targeted branches.
Consensus-based Credit Assignment (CCA): From the ensemble of \(M\) trajectories, the subset \(\mathcal{T}^+\) of correct-answer trajectories is identified. Each trajectory's union of zoom bounding boxes is rasterized to a binary mask \(M_i\); these are aggregated into a consensus heatmap \(C = \sum_{T_i \in \mathcal{T}^+} M_i\), then binarized by majority vote: \(\hat{M}(u,v) = \mathbf{1}(C(u,v) > |\mathcal{T}^+|/2)\). A tool reward is then assigned as: $\(R_\text{tool}(T_j) = \begin{cases} 1.0 & \text{if } \text{IoU}(M_j, \hat{M}) > \eta \\ 0.5 & \text{otherwise} \end{cases}\)$ The full reward is \(R(T) = R_\text{acc}(T) + R_\text{format}(T) + \mathbf{1}(R_\text{acc} > 0) \cdot R_\text{tool}(T)\), so tool reward is only granted on correct trajectories.
Key Contributions¶
- First end-to-end RL framework achieving annotation-free visual reasoning for medical VLMs, eliminating intermediate grounding supervision.
- EVR: uncertainty-driven adaptive branching that focuses exploration budget at high-entropy (spatially ambiguous) decoding steps.
- CCA: self-generated pseudo-labels from cross-trajectory consensus, providing IoU-based grounding rewards without human bounding-box annotations.
- SOTA results across both in-domain and OOD splits of six medical VQA benchmarks spanning multiple-choice and free-text formats.
Results¶
- Overall: MedVR-7B achieves 59.2 avg on General-Domain VQA and 74.0 avg on Modality-Specific VQA, outperforming all baselines including much larger models.
- OMVQA (in-domain, multiple-choice): 96.8 vs. best baseline Lingshu-7B at 84.2.
- PMC-VQA (OOD, multiple-choice): 54.3, on par with the best baseline (Lingshu-7B: 54.3).
- MedXpertQA (OOD, multiple-choice): 26.4 vs. Lingshu-7B 26.5 (competitive), vs. InternVL3-14B 23.1.
- VQA-RAD (in-domain, free-text): 74.4 vs. Lingshu-7B 67.9.
- SLAKE (in-domain, free-text): 85.3 vs. Lingshu-7B 83.1.
- PathVQA (OOD, free-text): 62.3 vs. Lingshu-7B 61.9, substantially above all other baselines.
- Ablations confirm both EVR and CCA are individually necessary; removing either degrades performance.
Limitations¶
- Only one visual tool (Zoom-in/crop) is implemented; richer operations (windowing adjustment, measurement, registration) used by clinicians are absent.
- Trained on Qwen2.5-VL-7B; scalability and behavior at larger model sizes are not evaluated.
- The consensus pseudo-label in CCA depends on having enough diverse correct trajectories per batch; very hard or rare cases may lack sufficient \(|\mathcal{T}^+|\) to form reliable consensus.
- OOD generalization on MedXpertQA (26.4%) is only marginally above baselines, suggesting difficulty on complex multi-step clinical reasoning without in-domain training data.
- No evaluation of whether the zoomed regions correspond to clinically meaningful anatomy beyond aggregated VQA accuracy.
Relevance to Vision-Language Models¶
MedVR demonstrates that RLVR — the same paradigm behind DeepSeek-R1 and general VLM reasoning advances — can be extended to agentic, tool-augmented visual reasoning without the bottleneck of grounded annotation, a critical step for VLMs in data-scarce expert domains. The EVR+CCA self-supervision loop is a domain-agnostic design pattern applicable wherever intermediate-step annotations are expensive, making it relevant to fine-grained visual grounding research beyond medicine. It also highlights a concrete failure mode of pure text-chain-of-thought VLMs (visual hallucination under language-prior shortcuts) and a principled RL-based remedy. For VLM researchers, the paper bridges active perception (iterative tool use to resolve uncertainty) with reward-shaping without privileged spatial labels.