Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Deep multimodal reasoning benchmark for sequential vision-centric agentic tasks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Agent-X is a large-scale benchmark of 828 manually verified, real-world multimodal agentic tasks spanning six environments, designed to evaluate vision-centric agents on multi-step tool-augmented reasoning rather than final-answer accuracy alone. It introduces a three-mode evaluation framework (step-by-step, deep reasoning, outcome) with nine fine-grained metrics. Even the best-performing frontier LMMs (GPT, Gemini, Qwen families) achieve less than 50% full-chain success, exposing fundamental gaps in multimodal agentic reasoning.

Problem¶

Existing agentic benchmarks evaluate agents with fully synthetic or single-turn queries, restrict inputs to static images or text, and measure only final-answer accuracy—ignoring the quality of intermediate reasoning steps. No prior benchmark simultaneously covers diverse real-world visual modalities (images, video, multi-image), tool-augmented reasoning, and interpretable step-level evaluation across multiple agentic environments. Tasks in prior work (GTA, m&m's, GAIA) often include explicit tool hints or instructions, preventing genuine autonomous planning.

Method¶

Agent-X defines each task as a structured tuple \(S_i = (V_i, Q_i, T_i, R_i, A_i, J_i)\), where \(V_i\) is the multimodal visual context, \(Q_i\) is a naturalistic query with no explicit tool or step cues, \(T_i \subseteq T_c\) is the tool subset drawn from a curated library of 14 executable tools, \(R_i = \{(t_j, a_j, r_j)\}_{j=1}^m\) is the ground-truth reasoning trace (tool, arguments, output per step), \(A_i\) is the final answer, and \(J_i\) is a natural-language justification.

Benchmark construction follows a semi-automated pipeline: an LMM generates three candidate queries per visual input from 1,021 raw inputs; human annotators select and refine the best query (yielding 828 validated tasks after filtering); the LMM then produces an initial reasoning trace, which annotators rigorously correct for logical consistency, tool appropriateness, and factual accuracy. Queries explicitly avoid referencing tools or steps to enforce autonomous planning.

Evaluation uses three modes with nine metrics: step-by-step (Grounding Score \(G_s\), Tool Precision \(T_p\), Tool Accuracy \(T_a\)), deep reasoning (Faithfulness \(F_{acc}\), Context Score \(C_s\), Factual Precision \(F_p\), Semantic Accuracy \(S_{acc}\)), and outcome (Goal Accuracy \(G_{acc}\), Goal Accuracy with image generation \(G^*_{acc}\), Toolset Accuracy \(T_{acc}\) as F1 over tool selection).

Key Contributions¶

Agent-X benchmark: 828 real-world, manually verified agentic tasks with authentic image/video/multi-image inputs, spanning six environments (general visual reasoning, web browsing, security & surveillance, autonomous driving, sports, math reasoning).
Semi-automated construction pipeline combining LMM-generated queries/traces with human refinement (~50 annotator-hours each, ~800K API tokens), producing structured annotation tuples with full reasoning traces and justifications.
Three-mode, nine-metric evaluation framework that separately assesses step-level grounding, multi-step logical coherence, and final task outcomes.
Empirical evaluation of 10 mainstream LMMs revealing sub-50% full-chain success rates, surfacing specific bottlenecks in tool selection, visual grounding, and reasoning faithfulness.

Results¶

All evaluated models (including GPT, Gemini, and Qwen families) achieve less than 50% full-chain success on Agent-X tasks.
828 tasks involve 716 image inputs and 112 video inputs; tasks average 3.4 reasoning steps and span 14 unique tools with 2,807 total tool calls.
Tasks distribute across queries requiring 2 tools (239 tasks), 3 tools (389), 4 tools (130), and 5 tools (24).
Key failure modes identified: inadequate tool selection, insufficient visual grounding (low \(G_s\)), reasoning hallucination (low \(F_{acc}\)), and poor sequential coherence across steps.
(Note: per-model numerical breakdowns are present in the full paper but were not included in the provided excerpt.)

Limitations¶

Benchmark size is 828 tasks—large for its class but small relative to single-modality benchmarks; coverage of all real-world agentic scenarios is necessarily incomplete.
The 14-tool library is curated and fixed; agents cannot use arbitrary APIs, limiting generalization to open-tool settings.
Generative queries (\(A_i = \emptyset\)) are excluded from \(G_{acc}\) computation because LMMs cannot produce visual outputs, leaving image/video generation evaluation under-assessed.
Semi-automated construction still relies on LMM-generated traces before human refinement, potentially introducing systematic biases from the generating model.
Evaluation of web-search tasks requires real-time query validity; temporal drift may degrade benchmark reliability over time.

Relevance to Agentic AI / LLM Agents¶

Agent-X directly addresses a central open problem in agentic LLM research: the lack of rigorous, step-level evaluation of tool-augmented multimodal reasoning in realistic settings. By exposing that frontier models fail more than half of multi-step vision tasks, it provides a concrete diagnostic distinguishing genuine multi-step planning from single-step or pattern-matched answers. The fine-grained metrics (\(G_s\), \(F_{acc}\), \(T_p\), \(T_{acc}\)) offer a vocabulary for analyzing where in the agentic loop failures occur—perception, tool selection, or reasoning coherence—which is actionable for designing improved agent architectures. The benchmark's design principle of withholding tool names from queries directly tests autonomous planning, making it a critical reference point for evaluating future agent systems that aim to operate with minimal scaffolding.