Skip to content

MedVR: Annotation-Free Medical Visual Reasoning via Agentic Reinforcement Learning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; MedVR: annotation-free agentic RL for grounded medical visual reasoning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MedVR is a reinforcement learning framework that enables medical Vision-Language Models to perform annotation-free visual reasoning by interleaving textual chain-of-thought with image-manipulation tool calls (zoom-in). Two novel mechanisms—Entropy-guided Visual Regrounding (EVR) and Consensus-based Credit Assignment (CCA)—provide self-generated supervision without any human-labeled intermediate steps. MedVR sets state-of-the-art on six medical VQA benchmarks using a 7B-parameter backbone.

Problem

Existing medical VLMs fine-tuned with RLVR reason purely in the textual domain, which causes visual hallucinations and prevents fine-grained grounding of clinical findings (e.g., lesion localization, tissue density comparison). Adapting general-purpose visual reasoning frameworks to medicine requires bounding-box annotations for intermediate visual steps—annotations that are prohibitively scarce and expensive in clinical settings. MedVR addresses the lack of any intermediate visual supervision for teaching a model to actively inspect images during reasoning.

Method

MedVR casts training as policy optimization over a VLM agent (Qwen2.5-VL-7B backbone) whose action space includes both token generation and invocation of a Zoom-in tool that crops a region of interest and re-encodes it into the context. The objective is:

\[\max_\theta \, \mathbb{E}_{(Q,I)\sim\mathcal{D},\, T\sim\pi_\theta} \left[ R(T) - \beta D_{\mathrm{KL}}[\pi_\theta(\cdot|Q,I) \| \pi_{\mathrm{ref}}(\cdot|Q,I)] \right]\]

The composite reward is \(R(T) = R_{\mathrm{acc}}(T) + R_{\mathrm{format}}(T) + \mathbf{1}[R_{\mathrm{acc}}>0]\cdot R_{\mathrm{tool}}(T)\), where \(R_{\mathrm{tool}}\) is only enabled on correct trajectories to avoid rewarding spurious tool use.

EVR (Entropy-guided Visual Regrounding): Token-level entropy \(H_t = -\sum_j p_{t,j}\log p_{t,j}\) is tracked during tool-coordinate generation. When the rolling-mean entropy increase \(\Delta H_{\mathrm{tool}} = H_{\mathrm{tool}} - H_{\mathrm{base}}\) exceeds a threshold, an adaptive branching event forks the trajectory with probability \(P = P_{\mathrm{base}} + \gamma\Delta H_{\mathrm{tool}}\), generating parallel rollouts exploring different zoom coordinates from the same high-uncertainty state.

CCA (Consensus-based Credit Assignment): From the set of successful trajectories \(T^+\), each trajectory's bounding-box footprint is rasterized into a binary mask \(M_i\). A consensus heatmap \(C = \sum_{T_i\in T^+} M_i\) is binarized via majority vote: \(\hat{M}(u,v) = \mathbf{1}(C(u,v) > |T^+|/2)\). The tool reward is then tiered: \(R_{\mathrm{tool}}=1.0\) if \(\mathrm{IoU}(M_j, \hat{M})>\eta\), else \(0.5\). This "wisdom of the crowd" over rollouts acts as a self-generated pseudo-label for visual grounding.

Policy optimization uses GRPO on 32 H20 GPUs for 64 iterations, with 16 trajectories per prompt and up to 6 tool calls per trajectory.

Key Contributions

  • First end-to-end RL framework for annotation-free visual reasoning in medical VLMs, integrating tool-augmented image manipulation into the reasoning loop.
  • EVR: uncertainty-driven adaptive branching that allocates exploration budget to high-entropy coordinate generation steps without any external signal.
  • CCA: consensus-mask distillation from successful rollouts that produces fine-grained pseudo-labels for intermediate visual actions without human annotation.
  • State-of-the-art results on six public medical VQA benchmarks (both in-domain and OOD), outperforming both general-purpose VLMs (InternVL3-14B, Qwen2.5-VL-32B) and domain-specific models (Lingshu-7B).

Results

  • OMVQA (in-domain, multi-choice): MedVR-7B = 96.8 vs. Lingshu-7B = 84.2, InternVL3-14B = 81.9.
  • PMC-VQA (OOD, multi-choice): MedVR = 54.3 vs. Lingshu-7B = 54.3 (tied best), InternVL3-14B = 54.1.
  • MedXpertQA (OOD, multi-choice): MedVR = 26.4 vs. Lingshu-7B = 26.5 (near-tied best).
  • General-domain VQA average: MedVR = 59.2 vs. Lingshu-7B = 55.0, InternVL3-14B = 53.0.
  • VQA-RAD (in-domain, free-text): MedVR = 74.4 vs. MedGemma-4B = 72.5.
  • SLAKE (in-domain, free-text): MedVR = 85.3 vs. Lingshu-7B = 83.1.
  • PathVQA (OOD, free-text): MedVR = 62.3 vs. Lingshu-7B = 61.9.
  • Modality-specific VQA average: MedVR = 74.0 vs. Lingshu-7B = 70.3, MedGemma-4B = 65.9.

Limitations

  • Only the Zoom-in operation is implemented; richer tool sets (windowing, measurement, annotation overlay) that clinicians use are absent.
  • The consensus pseudo-label quality degrades when successful trajectories are rare (sparse \(T^+\)), which may occur on very hard cases or early in training.
  • Evaluation is on multiple-choice and open-ended VQA only; radiology report generation and other structured clinical outputs are not assessed.
  • Backbone is a 7B model; scaling behavior and whether EVR/CCA remain stable at larger scales is unexplored.
  • Domain coverage is limited to the training modalities; zero-shot transfer to imaging modalities not seen during RL fine-tuning is not characterized.

Relevance to Agentic AI / LLM Agents

MedVR is a direct instantiation of the tool-augmented reasoning agent paradigm—the model emits structured tool calls, receives grounded observations, and updates its chain-of-thought before acting again—applied to a high-stakes domain where intermediate supervision is unavailable. The EVR+CCA mechanism is a domain-agnostic technique for bootstrapping fine-grained credit assignment in agentic loops without human-labeled trajectories, directly addressing the core RLVR challenge of sparse reward over long reasoning chains. This connects to broader work on RLVR (DeepSeek-R1, DeepEyes, Pixel-Reasoner) but uniquely solves the cold-start annotation bottleneck, making it relevant to any agentic application where intermediate-step labels are scarce. The conditional tool reward (enabled only on correct trajectories) is a practical design pattern that prevents reward hacking in tool-use agents.