Skip to content

Navigating the Mirage: A Dual-Path Agentic Framework for Robust Misleading Chart Question Answering

🕒 Published (v1): 2026-03-30 15:32 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ChartCynics is an agentic dual-path framework that decouples visual perception (via ROI-cropping diagnostic agents) from numerical extraction (via OCR serialization) to defend Vision-Language Models against misleading charts. A two-stage optimizer—Oracle-Informed SFT followed by Deception-Aware GRPO with asymmetric trap penalties—trains a Summarizer agent to arbitrate cross-modal conflicts using a five-step Detective Chain-of-Thought (D-CoT). On two MQA benchmarks the system reaches 74.43% and 64.55% accuracy, a ~29 pp absolute gain over the Qwen3-VL-8B backbone.

Problem

State-of-the-art VLMs score below 50% on Misleading ChartQA because they process charts holistically: they are hijacked by deceptive visual heuristics (inverted axes, truncated baselines) and cannot reconcile the contradiction between \(V_{trend}\) (what the chart looks like) and \(D_{rel}\) (what the numbers actually say). Pure OCR pipelines recover numbers but lose spatial context, causing catastrophic entity misalignment.

Method

ChartCynics composes three specialized agents in a fixed pipeline:

  1. Diagnostic Vision Path — a Diagnostic Agent performs a Blind Test: it receives only the full image and high-resolution ROI crops (title, legend, x-axis, y-axis) extracted by a graphic-element detector, never the question \(Q\) or options \(O\). It outputs a structured report \(R_{diag} = \{\text{Diagnosis}, \text{Action Directive}\}\) (e.g., "Y-axis is inverted; ignore visual slope"). A Reasoning Agent then anchors its CoT unconditionally to \(R_{diag}\) before considering \(Q\).

  2. OCR-Driven Data Path — an OCR parser serializes all textual/numeric entities into a Markdown table \(M_{ocr}\). Calibration Directives propagate trust levels (High Trust for explicit data labels; Low Trust when only axis ticks are present) to downstream reasoning.

  3. Agentic Summarizer — fuses \(R_{diag}\) and \(M_{ocr}\) via a five-step D-CoT: (1) Perception Audit, (2) Numerical Anchoring, (3) Deception Mapping against taxonomy \(\mathcal{T}\), (4) Sufficiency & Integrity Check, (5) Adversarial Trap Rejection (must prove why \(a_{trap}\) is a mathematical fallacy).

The Summarizer is optimized in two stages: - Oracle-Informed SFT: Qwen3-VL-32B generates visually-grounded reasoning traces using ground-truth CSV + trap labels but constrained to visible elements only; the 8B model imitates these traces. - Deception-Aware GRPO: multi-component reward $\(R_{total} = w_1 R_{fact} + w_2 R_{contra} + w_3 R_{logic} + w_4 R_{fmt} + R_{shaping}\)$ where \(R_{shaping}\) applies asymmetric penalties: correct answer = \(+1.0\), trap answer = \(-2.0\), training the policy to prioritize trap avoidance.

Key Contributions

  • ChartCynics: a training-free dual-path inference pipeline decoupling structural diagnosis from OCR-based numerical extraction via specialized agents.
  • Five-step Detective Chain-of-Thought (D-CoT) operationalizing "skeptical auditor" reasoning with explicit trap rejection.
  • Two-stage optimization: Oracle-Informed SFT for reasoning distillation + Deception-Aware GRPO with asymmetric reward shaping for adversarial alignment.
  • Validation that skeptical augmentation does not cause over-skepticism on benign charts—ChartCynics improves standard ChartQA accuracy over specialized chart-parsing SOTA.

Results

  • 74.43% accuracy on Misleading ChartQA benchmark vs. 45.57% for Qwen3-VL-8B baseline (~28.86 pp absolute gain).
  • 64.55% on a second (mixed standard + misleading) benchmark.
  • SFT alone contributes +22.95 pp over baseline; GRPO adds further improvement on complex cross-modal contradictions.
  • Outperforms state-of-the-art proprietary models on both benchmarks.
  • On benign standard charts, ChartCynics improves over specialized chart-parsing SOTA (no over-skepticism penalty).

Limitations

  • Relies on an external graphic-element detector to localize ROI bounding boxes; performance degrades if detection fails on unusual chart layouts.
  • OCR path still susceptible to hallucination when data labels are absent; trust-level heuristics are hand-coded, not learned.
  • SFT data generated by a 32B teacher under access to ground-truth CSV—real-world deployment has no CSV oracle, and the gap between teacher-generated traces and truly blind inference is not ablated.
  • Evaluation confined to two benchmarks; generalization to non-English charts or domain-specific visualizations (e.g., medical, financial) is untested.
  • Asymmetric GRPO penalty weights (\(-2.0\) vs. \(+1.0\)) appear empirically chosen without sensitivity analysis reported.

Relevance to Harnesses / Meta-Harnesses

ChartCynics is a concrete instantiation of a multi-agent harness with fixed orchestration topology: it composes three agents (Diagnostic, OCR/Reasoning, Summarizer) in a defined pipeline where each agent's output strictly conditions the next, analogous to tool-chaining harnesses that route intermediate results between specialized modules. The two-stage SFT + GRPO training of the Summarizer agent demonstrates how a meta-harness can expose a sub-agent's reasoning trajectory to harness-level reward signals (including structural format gates and adversarial shaping), making the optimization of individual agents aware of the full pipeline objective. The pattern of decoupled agents with structured inter-agent communication contracts (\(R_{diag}\), \(M_{ocr}\), Calibration Directives) is directly transferable to harness design where reliable handoffs between agents are required.