MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents

🕒 Published (v1): 2026-07-13 17:13 UTC · Source: Arxiv · link

Why this paper was selected

ECCV 2026; 500+-tool stateful benchmark for visual tool-calling agents; strong eval coverage

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MM-ToolSandBox is a benchmark and stateful execution framework for evaluating multimodal agents that must translate progressively arriving visual inputs into executable tool calls across 511 tools and 16 application domains. Evaluating 12 models reveals that even the best frontier system achieves below 50% success rate, with 53% of failures in capable models attributable to incorrect visual information extraction rather than planning failures. A scale-dependent crossover emerges: smaller models fail at planning, larger models fail at visual precision.

Problem

Existing tool-use agent benchmarks are predominantly text-centric, operate with small tool spaces (15–100 tools), and either treat images as static prefixes or restrict evaluation to single-turn interactions. No prior framework supports multi-turn, multi-image, large-tool-space evaluation of agents that must ground dynamically arriving visual artifacts into executable actions while handling realistic conversational dynamics (goal revisions, error corrections, state mutations).

Method

MM-ToolSandBox extends the ToolSandbox stateful environment (Lu et al., 2025) with:

1. Visual artifact system: images are treated as first-class entities stored in a shared database with stable identifiers, passed via typed tool calls (send_message_with_image) rather than as untracked context-window attachments.
2. Dual execution modes: structured Tool-Use (provider API schema-constrained calls) and Code-Execution (Python in a sandboxed persistent interpreter with three-layer safety guardrails: static import checks, runtime file/syscall restrictions, and wall-clock/memory limits).
3. Tool management: a search_tool meta-tool retrieves relevant tools on demand from 511 native Python functions; an LRU policy bounds the agent's active working set.
4. Benchmark axes: scenarios are structured along three orthogonal dimensions — seven information flow types (aggregate, compare, filter, compute, validate, lookup chain, cross-reference), four challenge types (goal change, error correction, state mutation, cooperative baseline), and four image arrival patterns (upfront, progressive, late, mixed).
5. Automated scenario generation: an information-flow-guided planning pipeline with multi-stage quality filtering produces visually grounded scenarios, reducing annotation to targeted human review; 258 scenarios (1,284 unique images) are human-verified.
6. UI mode subset: 50 scenarios require agents to act as dynamic UI designers using the A2UI protocol, rendering interactive screens through which a simulated user makes decisions via button clicks.
7. Evaluation: state-based entity diff verification (Hungarian matching with type-aware column similarity) combined with rubric-based LLM judges (GPT-5.4 user simulator, Claude 4.5 Sonnet agent judge scoring five criteria, UI judge). Agent SR requires all five criteria to pass; Entity F1 gives partial credit.

Key Contributions

• A stateful, multi-turn, multi-image tool-calling benchmark with 511 tools across 16 domains, 258 human-verified nominal scenarios, and 50 interactive UI scenarios.
• A taxonomy of seven information flow types grounded in data-processing primitives that impose structurally distinct reasoning demands.
• An automated scenario generation pipeline that reduces full manual construction to targeted human review.
• A systematic evaluation of 12 models (open-weight 4B to frontier proprietary) under a fixed harness isolating model capability from scaffolding.
• Empirical discovery of a planning-to-precision failure crossover as a function of model scale.

Results

• Best frontier model achieves below 50% Agent SR on the full benchmark.
• 53% of failures in capable (larger) models stem from incorrect visual information extraction despite otherwise correct task workflows.
• Smaller models fail primarily at planning (what to do); larger models fail primarily at visual precision (what they see) — a crossover effect with scale.
• Agent judge achieves 88% agreement with human annotation (Cohen's \(\kappa = 0.748\), substantial agreement) with a conservative bias.
• Agent SR and User SR correlate strongly (\(r = 0.81\)), confirming user simulation quality does not confound model rankings.
• UI judge raw agreement: 77% with lower \(\kappa\) driven by rarity of user-simulation failures.
• Multi-image working memory identified as a key bottleneck in per-failure analysis.
• (Detailed per-model numbers are in sections truncated from the provided text.)

Limitations

• Benchmark size is relatively modest (258 nominal + 50 UI scenarios), which may limit statistical resolution for fine-grained model comparisons.
• The text is truncated before the scenario generation pipeline details and full results tables, so per-model breakdowns and ablations are not available here.
• GPT-5.4 as the fixed user simulator may introduce systematic biases in conversational dynamics that favor or disfavor particular model families.
• Evaluation relies on LLM judges for trajectory quality, introducing model-in-the-loop variance despite strong human-agreement numbers.
• The benchmark is currently limited to simulated application environments; real-world deployment complexity (latency, API errors, ambiguous permissions) is not captured.
• UI mode is restricted to 50 scenarios, limiting conclusions about interactive UI agent behavior.

Relevance to Agentic AI / LLM Agents

MM-ToolSandBox directly addresses a critical gap in agentic evaluation by introducing the first multi-turn, multi-image, large-tool-space benchmark with stateful environment verification, filling a niche between text-only tool-use benchmarks and GUI navigation benchmarks. The finding that visual precision — not planning — is the primary bottleneck for frontier-scale agents reframes the research agenda: improving perception-to-action grounding is at least as important as improving reasoning or planning capabilities for capable models. The planning-to-precision crossover with scale is a concrete, actionable insight for practitioners choosing which capability to prioritize given their model tier. The dual execution mode design (structured Tool-Use vs. Code-Execution) also provides infrastructure for studying how interface choice affects agentic performance, relevant to the growing debate around tool-calling APIs vs. code-as-action paradigms.