VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

RL finetuning teaches VLMs to reason with multimodal tools

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VTool-R1 is the first reinforcement learning finetuning (RFT) framework that trains vision-language models (VLMs) to generate multimodal chains of thought by interleaving text with intermediate visual reasoning steps produced via Python-based image-editing tools. Using outcome-based rewards only (no process supervision), the framework teaches models when and how to invoke visual tools, not merely whether to use them. Experiments on chart and table reasoning show substantial gains over both prompted tool-use baselines and direct-inference baselines.

Problem

Prior RFT-based reasoning approaches for VLMs (e.g., R1-VL) remain purely text-driven: they encode the input image once and generate reasoning chains entirely in text, never producing or consuming intermediate visual artifacts. Inference-time methods like Visual Sketchpad and Refocus can incorporate visual steps but depend on powerful closed models (GPT-4o) and provide no training signal, causing them to fail with smaller open-source VLMs. The gap is: no training framework existed that teaches a VLM to autonomously decide to call visual tools and reason over the resulting edited images during RFT rollout.

Method

VTool-R1 extends GRPO-based RFT to a two-stage iterative rollout. In stage 1, the VLM policy \(\pi_\theta\) receives the original image \(I\) and text prompt \(x\), generates an optional Python tool-call \(y'\), which is executed in an external Python environment to produce an edited image \(I' = T(y', I)\). In stage 2, the model reasons over both images:

\[y \sim \pi_\theta(\cdot \mid I \oplus I', x)\]

where \(\oplus\) denotes dual-image concatenation. If no tool is invoked, inference collapses to \(y \sim \pi_\theta(\cdot \mid I, x)\). Training optimizes only the final response \(y\) (not the intermediate tool-invoking output \(y'\)) via the standard GRPO objective with normalized group-relative advantages \(\hat{A}_{i,t} = \frac{r_i - \text{mean}(r)}{\text{std}(r)}\) and KL penalty against a reference policy. Rewards are outcome-based: an LLM judge scores exact-match correctness of the final answer (binary 0/1). Process-based rewards (penalizing/rewarding tool calls) are explicitly rejected due to reward hacking. The visual toolset (from Refocus) includes highlight/mask/draw operations on columns, rows, or chart bars. Base models are Qwen2.5-VL at 3B, 7B, and 32B scales.

Key Contributions

• First RFT framework that trains VLMs to generate multimodal chains of thought with intermediate visual reasoning steps interleaved in the response.
• Outcome-based-only reward design that elicits strategic, adaptive tool use without process supervision.
• Empirical demonstration that small open-source VLMs (3B, 7B) can learn meaningful visual tool use via RL, closing the gap to GPT-4o inference-time tool use.
• Analysis showing tool-call frequency is non-monotonic during training: models initially overtool, then learn selective, context-aware invocation.

Results

• VTool-R1 7B vs. Qwen2.5-VL 7B (pure run): Chart Split 80.7 vs. 76.2 (+4.5 pp); Table Split 71.7 vs. 41.1 (+30.6 pp).
• VTool-R1 7B vs. prompted tool-use (no RFT) 7B: Chart 80.7 vs. 53.4; Table 71.7 vs. 41.1 — shows that RFT is essential; prompting alone fails to elicit reliable tool use.
• VTool-R1 3B vs. Qwen2.5-VL 3B (pure run): Chart 64.0 vs. 51.8; Table 57.9 vs. 41.3.
• VTool-R1 7B vs. R1-VL 7B (RL-trained text-only baseline): Chart 80.7 vs. 63.8; Table 71.7 vs. 45.4.
• VTool-R1 7B vs. GPT-4o (inference-only tool use): Chart 80.7 vs. 82.9 (near parity); Table 71.7 vs. 75.7.
• VTool-R1 32B: Chart 86.7; Table 84.5 — approaching or exceeding GPT-4o on both splits.
• Qwen2.5-VL 32B pure run already achieves 88.0/86.2, unexpectedly strong, suggesting prior VQA post-training; VTool-R1 32B at 86.7/84.5 is marginally below pure run on Chart but competitive overall.

Limitations

• Single-turn tool use only (at most one tool call per query); multi-turn iterative visual editing is left to future work, requiring stronger multi-image VLMs.
• Visual toolset is task-specific (chart/table attention masks) and not general-purpose; broader tool diversity is not validated.
• Reward function relies on an LLM judge rather than a deterministic rule, introducing noise in the reward signal.
• The 32B VTool-R1 result is marginally below the Qwen2.5-VL 32B pure run on chart tasks, suggesting that for very large models with strong priors, tool use may not always add value.
• Training is computationally expensive (up to 8× H200 GPUs for 32B), limiting reproducibility.
• Dataset scope is narrow (structured visual reasoning: charts and tables only); generalization to other visual domains is unverified.

Relevance to Agentic AI / LLM Agents

VTool-R1 directly advances the agentic paradigm for multimodal models: rather than treating tool use as a static inference-time heuristic, it learns when to invoke tools as part of an RL-trained policy, closely mirroring the decision-making loop of an autonomous agent. The two-stage rollout (plan → act → observe → reason) is a concrete instantiation of the perception-action cycle central to agentic AI. The outcome-only reward design generalizes the DeepSeek-R1/GRPO approach to tool-augmented, multi-modal settings — a pattern directly applicable to any agent architecture that needs to decide dynamically whether to call an external capability. The finding that process rewards cause hacking (models game tool-invocation criteria without improving reasoning) is an important negative result for agent training practitioners designing reward functions.