Navigating the Mirage: A Dual-Path Agentic Framework for Robust Misleading Chart Question Answering¶

🕒 Published (v1): 2026-07-14 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

Dual-path agentic VLM framework for misleading chart QA; concrete scaffold pattern to borrow

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

ChartCynics is an agentic dual-path framework that decouples visual perception from OCR-based numerical extraction to robustly answer questions about deliberately deceptive charts. A trained Agentic Summarizer resolves cross-modal conflicts via a five-step Detective Chain-of-Thought (D-CoT), boosted by SFT and deception-aware GRPO. On two benchmarks it reaches 74.43% and 64.55% accuracy, a ~29 percentage-point absolute gain over its Qwen3-VL-8B backbone.

Problem¶

State-of-the-art VLMs fail on Misleading Chart Question Answering (MQA)—charts whose visual encoding (inverted axes, truncated baselines, inappropriate ordering) contradicts the underlying data. Models process charts holistically and exhibit "cognitive sycophancy," trusting visual trend $V_\text{trend}$ over numerical truth $D_\text{rel}$. Pure OCR pipelines provide numbers but lose spatial context, causing catastrophic entity misalignment (axis ticks confused with data labels). No existing method robustly arbitrates the inherent cross-modal conflict.

Method¶

Dual-path architecture (training-free inference): - Vision Path – A Diagnostic Agent receives only the chart image and high-resolution ROI crops $C_\text{roi} = \{\text{title, legend, x-axis, y-axis}\}$, withheld from the question/options to prevent self-prompt pollution. It outputs a Diagnostic Report $R_\text{diag} = \{\text{Diagnosis, Action Directive}\}$ naming structural anomalies (e.g., "y-axis inverted"). A Reasoning Agent then grounds its inference exclusively on $R_\text{diag}$. - Data Path – An advanced multimodal OCR module serialises all textual/numerical entities into structured Markdown $M_\text{ocr}$, accompanied by Calibration Directives encoding a dual-tier trust principle (High Trust for explicit data labels; Low Trust for axis-only inference). - Agentic Summarizer – Fuses $R_\text{diag}$ and $M_\text{ocr}$ via a five-step D-CoT: (1) Perception Audit, (2) Numerical Anchoring, (3) Deception Mapping against taxonomy $\mathcal{T}$, (4) Sufficiency & Integrity Check, (5) Adversarial Trap Rejection. The objective is: $$a^* = \arg\max_{a \in O} P(a \mid P_v(I), P_d(I), Q, \mathcal{T})$$

Two-stage optimization of the Summarizer: - Oracle-Informed SFT: Qwen3-VL-32B generates visually-grounded reasoning traces (using ground-truth CSV + trap labels but constrained to visible chart elements only), which are distilled into the 8B target model. - Deception-Aware GRPO: Multi-dimensional reward $R_\text{total} = w_1 R_\text{fact} + w_2 R_\text{contra} + w_3 R_\text{logic} + w_4 R_\text{fmt} + R_\text{shaping}$, with asymmetric shaping that awards +1.0 for a correct answer but penalises trap selection with $-2.0$.

Key Contributions¶

ChartCynics dual-path agentic framework with a "Blind Test" Diagnostic Agent that enforces question-blind ROI inspection to eliminate confirmation bias.
A five-step Detective Chain-of-Thought (D-CoT) with hierarchical evidence-weighting rules for cross-modal conflict arbitration.
Two-stage training protocol: Oracle-Informed SFT for structured reasoning distillation + Deception-Aware GRPO with asymmetric trap-penalty reward shaping.
Empirical demonstration that agentic specialisation grants an 8B open-source model robustness exceeding proprietary SOTA while avoiding over-skepticism on benign charts.

Results¶

74.43% accuracy on Misleading ChartQA benchmark vs. 45.57% for Qwen3-VL-8B backbone (+28.86 pp absolute).
64.55% accuracy on a second (mixed standard + misleading) benchmark.
SFT alone contributes +22.95 pp over baseline; GRPO adds further gain on complex cross-modal cases.
Outperforms state-of-the-art proprietary models on both benchmarks.
On standard (non-misleading) charts, ChartCynics exceeds specialised chart-parsing SOTA—no over-skepticism penalty observed.

Limitations¶

Relies on an external graphic-element detection module for ROI bounding-box localisation; errors here propagate to both paths.
OCR parsing can hallucinate or misread labels when they are absent or overlap; the framework mitigates but does not eliminate this via the Low Trust tier.
SFT training data is generated by a 32B oracle with access to ground-truth CSV—creating a pipeline dependency on privileged data that may not generalise to novel deception types not in taxonomy $\mathcal{T}$.
Evaluation is limited to two benchmarks; broader generalisation across chart types (maps, flow diagrams, infographics) is untested.
The asymmetric GRPO reward ($-2.0$ for trap selection) introduces a hyperparameter whose sensitivity is not ablated in the provided text.

Relevance to Agentic AI / LLM Agents¶

ChartCynics exemplifies the agentic design pattern of decomposing a single hard perception-reasoning task into specialised sub-agents with deliberately restricted information access (the Diagnostic Agent's Blind Test), directly addressing confirmation bias—a known failure mode in monolithic LLM reasoning. The two-stage SFT → GRPO training pipeline, where RL fine-tunes a behaviorally-cloned policy against adversarial reward shaping, is a canonical method for aligning agentic reasoning under distribution shift and directly informs reward-design choices for multi-step tool-using agents. The D-CoT "skeptical auditor" paradigm—where an agent must explicitly falsify a high-confidence but deceptive hypothesis before committing to an answer—is transferable to any agentic setting involving untrustworthy inputs (web retrieval, sensor data, adversarial tools). The result that a small specialised agentic system outperforms large proprietary models reinforces the architectural argument for task-specific agent scaffolding over scaling alone.