Skip to content

ChartAgent: A Multimodal Agent for Visually Grounded Reasoning in Complex Chart Question Answering

🕒 Published (v1): 2025-10-06 06:05 UTC · Source: Arxiv · Venue: ACL 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ChartAgent is a ReAct-style multimodal agent harness that performs visually grounded reasoning on chart images by iteratively decomposing queries into visual subtasks and executing a library of chart-specialized perception tools (segmentation, axis localization, bar-height estimation, etc.). An LLM orchestrator routes queries—sending annotated charts directly to the base MLLM and triggering the full tool loop only for unannotated or numerically intensive cases. It achieves state-of-the-art accuracy on ChartBench (+16.07% absolute over the prior best) and ChartX (+2.83% over GPT-4o).

Problem

Multimodal LLMs degrade sharply on unannotated charts—charts lacking explicit numerical labels or tick-value annotations—where the model cannot exploit textual shortcuts and must instead estimate values (bar heights, pie-slice areas, legend–color associations) directly from pixel-level evidence. Existing approaches (fine-tuned chart MLLMs, CoT prompting, DePlot-style table conversion) all bypass this visual grounding problem rather than solving it.

Method

ChartAgent wraps a base MLLM (GPT-4o by default) inside a structured pipeline with four interlocking components:

  1. Orchestrator & Smart Routing: Extracts chart metadata (type, annotation status, axis ticks, legend) and routes: annotated charts / qualitative queries → direct MLLM; unannotated / numeric queries → ReAct loop.
  2. ReAct Loop (up to 15 iterations): Each step produces Thought (MLLM derives the next visual sub-goal \(g_t\)), Action (selects and executes a tool \(a_t^{\text{chart-tool}}\) on the image), and Observation (MLLM visually inspects the returned annotated image \(o_{t+1}\) and updates state \(s_{t+1} = (s_t, g_t, a_t, o_{t+1})\)).
  3. Chart Tool Library: Python functions covering 40+ chart types, organized as (a) universal tools—annotate_legend, axis_localizer, segment_and_mark—and (b) chart-specific tools—compute_bar_height, compute_segment_area, get_edgepoints. Tools internally use SAM, Semantic SAM, Tesseract, and EasyOCR and return both numeric outputs and interpretable visualizations.
  4. Visual Self-Verification & Adaptive Use: At each observation step, the agent inspects the returned visualization; if unsatisfactory (wrong segmentation, mismatched colors, negative heights), it adjusts parameters or switches tools in \(t+1\). If tool-based reasoning consistently fails, the system falls back to the base MLLM.

Chart-type-specific 1–2-shot ICL examples (full ReAct trajectories) are retrieved based on the orchestrator's detected chart type. The framework is built on AutoGen.

Key Contributions

  • First tool-augmented agentic framework demonstrating visually grounded reasoning for Chart VQA, operating directly in the chart's spatial domain rather than via text conversion.
  • Modular chart tool library spanning 40+ chart types with interpretable intermediate visualizations enabling visual self-verification and adaptive tool use.
  • State-of-the-art on ChartBench and ChartX, surpassing 30+ baselines including concurrent GPT-o3/o4-mini/GPT-5.
  • Plug-and-play design: validated across GPT-4o-mini, Claude 3 Haiku, Phi-3 Vision, Qwen2-VL, DeepSeek-VL2, LLaMA-3.2—performance improves for all tested backbones.
  • In-depth ablations over chart type, visual complexity, reasoning complexity, and failure modes.

Results

  • ChartBench overall: 71.39% (+16.07% absolute over Phi-3 Vision, 2nd best among all models)
  • ChartBench unannotated: 60.81% (+17.31% over Qwen2-VL)
  • ChartBench numeric QA: 70.91% (+15.02% over Phi-3 Vision)
  • ChartX overall: 59.69% (+2.83% over GPT-4o)
  • ChartX unannotated: 44.16% (top)
  • ChartX numeric QA: 55.93% (top)
  • Concurrent-works leakage-controlled dataset: +10.48% over GPT-5 (2nd best); 5.72-point reduction in mean absolute error vs. GPT-o3
  • On annotated charts, performance matches GPT-4o (routing preserves accuracy without extra cost)

Limitations

  • Tool library covers 40+ chart types but exotic or highly stylized charts outside this taxonomy are unsupported.
  • Iteration cap of 15 steps is heuristic; complex charts may require more and still fall back to the base MLLM.
  • Orchestrator misclassification of annotation status or chart type propagates errors into ICL retrieval and tool selection.
  • Computational overhead of the full ReAct loop (SAM, OCR, multi-turn MLLM calls) is substantially higher than single-pass inference; the paper does not report latency or cost figures.
  • Core backbone is proprietary (GPT-4o); open-weight backbone experiments show ChartAgent helps but absolute accuracy is lower.
  • The 5% numeric tolerance evaluation may mask cases where visual estimation errors compound across multi-step reasoning chains.

Relevance to Harnesses / Meta-Harnesses

ChartAgent is a concrete, domain-specialized instantiation of a tool-orchestration harness: an orchestrator meta-layer that classifies the incoming task, retrieves context (chart type → ICL examples), and then drives a ReAct loop over a library of stateless, composable tools—exactly the pattern that general meta-harnesses aim to generalize. The visual self-verification loop—where the agent inspects intermediate tool outputs and adaptively re-invokes tools with adjusted parameters before committing—is a domain-specific realization of the self-repair / reflection pattern central to robust agentic harness design. The plug-and-play backbone claim (same harness, swap the MLLM) is directly relevant to harness designers evaluating how tightly orchestration logic should couple to a specific model. Building this harness on AutoGen also provides a worked example of layering a domain-specific orchestration protocol on top of a general multi-agent framework.