VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

VLMs learn multimodal tool use via RL; key agentic VLM capability

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

VTool-R1 is the first reinforcement learning finetuning (RFT) framework that trains vision-language models (VLMs) to generate multimodal chains of thought by interleaving textual reasoning with intermediate visual steps produced via Python-based image editing tools. Using only outcome-based (correctness) rewards—no process supervision—the framework teaches VLMs when and how to invoke tools, yielding substantial accuracy gains on structured visual reasoning tasks. The key insight is that pure text-driven CoT in VLMs is insufficient; visual reasoning steps act as a complementary modality for grounding and attention.

Problem¶

Prior RFT approaches for VLMs (R1-style) only generate textual chains of thought conditioned on a fixed input image; no intermediate visual reasoning occurs during response generation. Inference-time methods like Visual Sketchpad that do produce visual steps require powerful closed-source models (GPT-4o) and fail with smaller open-source VLMs. There is no training mechanism that teaches a VLM to autonomously decide when and how to use visual tools as part of its reasoning process.

Method¶

VTool-R1 augments the GRPO RFT objective to accommodate a two-stage iterative rollout. Given a multimodal query \((I, x)\), the policy \(\pi_\theta\) first generates a response \(y' \sim \pi_\theta(\cdot | I, x)\) that may include a Python-like tool call invoking one of several image editing functions (highlight/mask/draw on table rows/columns or chart bars). The tool is executed externally to produce an edited image \(I' = T(y', I)\); the VLM then reasons over the concatenated dual-image input \(I \oplus I'\) to produce the final answer:

\[y \sim \pi_\theta(\cdot \mid I \oplus T(y', I),\, x)\]

Only the final response \(y\) is optimized (not the intermediate \(y'\)), so the model must learn through outcome signals alone whether invoking a tool helps. The reward is binary: an LLM judge scores 1 if the final answer matches ground truth. Process-based rewards (penalizing/rewarding tool calls directly) are explicitly rejected due to reward hacking. Training uses GRPO with group size 5, KL coefficient \(\beta = 10^{-2}\), on Qwen2.5-VL at 3B/7B/32B scale via the VeRL infrastructure.

The visual toolset (from Refocus) applies selective-attention operations: highlight/mask/draw bounding boxes on rows, columns, or bars. The model is limited to at most one tool invocation per query.

Key Contributions¶

First RFT framework enabling VLMs to generate multimodal chains of thought with interleaved intermediate visual steps (images + text in the reasoning trace).
Demonstrates that outcome-based rewards alone are sufficient to induce strategic visual tool use—no process supervision or ground-truth tool-call labels needed.
Shows that open-source VLMs (3B, 7B) trained with VTool-R1 acquire tool-use capability that previously required GPT-4o.
Documents non-monotonic tool-call frequency during training: models over-invoke tools early, then learn selectivity, correlating with accuracy gains.
Open-sources code and training infrastructure for multi-turn multimodal reasoning research.

Results¶

VTool-R1 7B vs. Qwen2.5-VL 7B (no RFT, direct) — Chart: 80.7 vs. 76.2 (+4.5 pp); Table: 71.7 vs. 41.1 (+30.6 pp).
VTool-R1 7B vs. Qwen2.5-VL 7B (prompted tool use, no RFT) — Chart: 80.7 vs. 53.4; Table: 71.7 vs. 41.1 — confirming that prompting alone cannot elicit meaningful tool use.
VTool-R1 3B vs. Qwen2.5-VL 3B (direct) — Chart: 64.0 vs. 51.8; Table: 57.9 vs. 41.3.
VTool-R1 7B vs. R1-VL 7B (RL-tuned academic baseline) — Chart: 80.7 vs. 63.8; Table: 71.7 vs. 45.4.
VTool-R1 7B vs. GPT-4o (prompted tool use) — Chart: 80.7 vs. 82.9; Table: 71.7 vs. 75.7 — competitive with a commercial upper bound.
VTool-R1 32B achieves 88.0 on charts and 86.2 on tables (vs. GPT-4o 82.9 / 77.0), surpassing GPT-4o on both splits.
Qwen2.5-VL 32B direct inference is already strong (85.0 chart / 76.0 table), so 32B RFT gains are more modest but still positive.

Limitations¶

Tool use is limited to one invocation per query; multi-turn iterative tool use is left for future work due to demanding multi-image input requirements.
The visual toolset is domain-specific (table/chart attention operations); generalization to arbitrary visual tools or open-ended visual reasoning has not been demonstrated.
Training on 32B requires 8× H200 GPUs (141 GB VRAM each), limiting accessibility.
The LLM-based judge reward is not strictly rule-based, introducing potential reward noise for free-form answers.
Experiments are confined to structured image reasoning (charts and tables); results on other VQA domains or more diverse visual tasks are not reported.
Concurrent works (Deepeyes, OpenThink-IMG) are acknowledged but only partially compared against.

Relevance to Vision-Language Models¶

VTool-R1 addresses a critical gap in the VLM post-training literature: existing RFT methods for VLMs keep visual reasoning frozen at the input encoding stage, while VTool-R1 is the first to inject trainable visual reasoning steps into the generation process itself. This shifts the paradigm from VLMs that see images to VLMs that think with images, directly relevant to researchers studying multimodal chain-of-thought, agentic VLMs, and visual grounding. The result that outcome-only rewards suffice to teach strategic visual tool use—without any annotated reasoning traces—is significant for scalable VLM alignment. It also highlights an unresolved tension: large, instruction-tuned VLMs (Qwen2.5-VL 32B) already achieve high direct-answer accuracy on these benchmarks, suggesting that the marginal value of visual CoT may be task- and scale-dependent and warrants further investigation.