Skip to content

ToolSciVer: Multimodal Scientific Claim Verification with Visual Tool Augmented Reinforcement Learning

🕒 Published (v1): 2026-07-17 17:11 UTC · Source: Arxiv · link

Why this paper was selected

VL tool-use + RL for multimodal scientific verification; novel reward signal design

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ToolSciVer is the first tool-augmented reinforcement learning framework for Multimodal Scientific Claim Verification (MSCV), equipping VLMs with three type-aware visual tools (Table Focus, Chart Parse, Region Zoom) trained via GRPO with a composite reward that simultaneously optimizes answer correctness, format, length, and tool-use efficiency. It consistently outperforms both prompting-based and RL-based tool-use baselines across five open-source VLMs on the SciVER and MuSciClaims benchmarks.

Problem

Existing VLMs—including frontier proprietary models (GPT, Claude, Gemini)—fail at MSCV because they cannot reliably localize decisive visual evidence in tables, charts, and multi-panel figures, nor aggregate it across modalities for grounded multi-step reasoning. General-domain visual tool systems (VTool-R1, OpenThinkIMG) use generic image-editing operations (crop, zoom, OCR) that are not designed for claim-conditioned evidence localization over structured scientific visuals. Prompt-only tool access is brittle, frequently underperforming zero-tool CoT on SciVER.

Method

ToolSciVer frames MSCV as an evidence acquisition problem: a VLM iteratively calls type-specific visual tools, accumulates observations into context, and produces a final binary verdict (supported/refuted).

Tool suite (three tools, one per evidence type): - Table Focus: OCR converts a table image to CSV; focus_row/focus_column return only the targeted row or column, suppressing irrelevant context. - Chart Parse: Calls a stronger VLM to convert a chart image into structured JSON-like textual observations (axes, values, trends). - Region Zoom: Crops and upscales a model-selected bounding box (coordinates normalized to \([0, 1000]\)) from dense or multi-panel figures.

GRPO training with a composite reward: $\(R(\tau) = s_{\text{eff}}(\tau)\bigl(r_{\text{ans}}(\tau) + \lambda_{\text{fmt}} r_{\text{fmt}}(\tau)\bigr) + \lambda_{\text{len}} r_{\text{len}}(\tau) + \lambda_{\text{tool}} r_{\text{tool}}(\tau)\)$ with \(\lambda_{\text{fmt}} = \lambda_{\text{len}} = \lambda_{\text{tool}} = 0.1\).

The key component is \(s_{\text{eff}}(\tau)\), an OTC-style tool-efficiency coefficient that estimates the group-local optimal tool budget \(n^*(x)\) as the minimum tool calls among strict-correct rollouts in the GRPO group. It rewards matching this minimum while penalizing both unnecessary tool use (when direct answer suffices) and missing required tool use: $\(s_{\text{eff}}(\tau) = \begin{cases} 1, & n^*=0, m=0 \\ \cos\!\frac{m\pi}{2m+c_{\max}}, & n^*=0, m>0 \\ 0, & n^*>0, m=0 \\ \sin\!\frac{m\pi}{m+n^*}, & n^*>0, m>0 \end{cases}\)$

A tool-interaction penalty \(r_{\text{tool}}\) provides dense negative feedback for malformed tool turns (\(-0.20\) per first, \(-0.10\) each additional) and execution failures (\(-0.30\)/\(-0.15\)), with an extra \(-0.40\) if the rollout terminates without a valid answer after any tool error.

Key Contributions

  • First visual tool-augmented framework specifically designed for MSCV, with tools matched to the three dominant scientific evidence types (table, chart, multi-panel figure).
  • Type-aware tool suite that returns structured, claim-facing observations rather than raw OCR, bounding boxes, or edited images.
  • GRPO reward design incorporating a group-adaptive tool-efficiency coefficient (\(s_{\text{eff}}\)) that dynamically calibrates the expected tool budget per instance.
  • Comprehensive evaluation across five open-source VLMs from three families (Qwen, InternVL, Gemma) on two MSCV benchmarks, with diagnostic analyses of evidence acquisition rate, tool selection accuracy, efficiency, and reasoning reliability.

Results

  • Main benchmarks (Table 2): ToolSciVer improves over non-tool CoT across all five backbones on both SciVER and MuSciClaims; e.g., Qwen3.5-4B overall on SciVER: CoT baseline vs. ToolSciVer = 81.67% (Ours beats VTool-R1 at 79.28% and OpenThinkIMG at 78.09%).
  • Prompt-only tool access degrades performance: Tool Prompt underperforms CoT on SciVER for 4 of 5 backbones, confirming that untrained tool access is insufficient.
  • REAR (Relevant Evidence Acquisition Rate, Table 3): ToolSciVer raises average REAR from 83.00% (CoT) and 83.33% (Tool Prompt) to 90.00%, with the largest gain on multi-panel figures (69% → 79%).
  • Type-aware tool selection (Table 4): On the table subset, ToolSciVer achieves \(\Delta\)Acc = +16.15 pp over CoT with tool success rate 97.39% and \(\rho_{\text{tab}}\) = 89.25%; chart subset yields +11.51 pp with \(\rho_{\text{chart}}\) = 74.24%.
  • OTC reward ablation (Table 5, Qwen3.5-4B): Adding \(s_{\text{eff}}\) raises SciVER accuracy from 76.96% → 81.20% and MuSciClaims from 73.47% → 78.61%, while halving tool calls (1.05 → 0.74 per example) and cutting average tokens from 2760 → 1362.
  • Proprietary baselines (non-tool CoT, external reference): Claude Sonnet 4.6 leads on MuSciClaims (Overall 79.41%) while GPT-5.4 is substantially weaker on SciVER (Overall 62.90%), underscoring that frontier models do not uniformly solve MSCV.

Limitations

  • Evaluation is restricted to binary (supported/refuted) verification; multi-class or evidence-ranking formulations are not studied.
  • The Chart Parse tool depends on a separate, stronger VLM as an oracle, introducing a black-box dependency not subject to the same training regime.
  • Multi-panel figure handling (Region Zoom) shows the smallest gains: tool success rate remains only 50.77% even after training, and \(\Delta\)Acc = +7.19 pp—the weakest among the three evidence types.
  • The REAR annotation uses GPT-5.5-assisted labeling, introducing potential annotation noise.
  • The evaluation suite covers only two English MSCV benchmarks (SciVER, MuSciClaims); generalization to other scientific domains, languages, or document layouts is untested.

Relevance to Vision-Language Models

ToolSciVer directly advances the capability of open-source VLMs on a task where even frontier closed-source models struggle, demonstrating that domain-specific, type-aware tool augmentation is a more effective strategy than generic visual editing tools for structured scientific understanding. The GRPO-based training paradigm with a multi-component reward—particularly the OTC tool-efficiency coefficient—offers a reusable template for teaching VLMs selective and efficient external tool use without behavioral collapse into over-calling. This work complements the broader line of tool-augmented VLM research (VTool-R1, OpenThinkIMG, Visual Sketchpad) by showing that task-alignment of both tools and reward signals matters as much as the RL training method itself. For researchers tracking VLMs, the evidence-acquisition framing—decomposing verification into observation-stage evidence extraction followed by reasoning—is a transferable design principle for other document-understanding and multimodal reasoning tasks.