Navigating the Mirage: A Dual-Path Agentic Framework for Robust Misleading Chart Question Answering¶

🕒 Published (v1): 2026-07-14 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

ChartCynics is an agentic dual-path framework that defends VLMs against misleading charts by decoupling structural visual perception from OCR-based numerical extraction, then fusing them via a five-step Detective Chain-of-Thought (D-CoT). A two-stage training protocol—Oracle-Informed SFT followed by Deception-Aware GRPO with asymmetric trap penalties—teaches the Summarizer module to actively arbitrate cross-modal conflicts rather than capitulate to deceptive visual heuristics. On two benchmarks, it lifts Qwen3-VL-8B from ~45% to ~74% accuracy, surpassing proprietary SOTA models.

Problem¶

State-of-the-art VLMs fail catastrophically on Misleading Chart QA (MQA)—scoring below 50% on benchmarks like Misleading ChartQA—because they process charts holistically and succumb to "cognitive sycophancy": deceptive encodings (inverted axes, truncated baselines, inappropriate ordering) hijack the visual encoder's attention, causing the model to endorse the trap answer $a_{\text{trap}}$ derived from the perceived visual trend $V_{\text{trend}}$ rather than the true numerical relationship $D_{\text{rel}}$. Purely OCR-based pipelines suffer the complementary failure of entity misalignment without spatial context.

Method¶

Dual-Path Architecture. Given chart image $I$, question $Q$, and candidate options $O$, ChartCynics models the answer as: $$a^* = \arg\max_{a \in O} P(a \mid P_v(I),\, P_d(I),\, Q,\, T)$$ where $T$ is an expert Misleading Taxonomy, $P_v$ is the Vision Path output, and $P_d$ is the OCR Data Path output.

Vision Path (Diagnostic-Augmented). A graphic-element detector localizes bounding boxes of title, legend, x-axis, and y-axis, producing high-resolution ROI crops $C_{\text{roi}}$. A Diagnostic Agent performs a blind test—it sees only $I$ and the crops, never $Q$ or $O$—generating a diagnostic report $R_{\text{diag}} = \{\text{Diagnosis, Action Directive}\}$ to prevent self-prompt pollution. A separate Reasoning Agent anchors its CoT to $R_{\text{diag}}$ via an unconditional trust policy.

OCR Data Path. An OCR module serializes all textual/numerical entities into a structured Markdown $M_{\text{ocr}}$. Dynamic Trust tiers (High Trust for explicit data labels; Low Trust when only axis ticks are present) and structural integrity checks (reversed time series, incomplete-period deceptions) are passed as Calibration Directives to downstream reasoning.

Agentic Fusion (D-CoT). An Agentic Summarizer applies a five-step Detective Chain-of-Thought: (1) Perception Audit—reiterating Action Directives; (2) Numerical Anchoring—reconstructing $D_{\text{rel}}$ from $M_{\text{ocr}}$; (3) Deception Mapping—classifying the manipulation subtype via $T$; (4) Sufficiency & Integrity Check—allowing "Cannot be Inferred" to avoid hallucination; (5) Adversarial Trap Rejection—explicitly proving why $a_{\text{trap}}$ is a mathematical fallacy.

Two-Stage Optimization. - Oracle-Informed SFT: A larger teacher (Qwen3-VL-32B) generates reasoning chains using ground-truth CSV and trap labels but constrained to only reference visible chart elements, producing <Visual_Heuristic>, <OCR_Validation>, <Ambiguity_Resolution> traces. - Deception-Aware GRPO: A composite reward $R_{\text{total}} = w_1 R_{\text{fact}} + w_2 R_{\text{contra}} + w_3 R_{\text{logic}} + w_4 R_{\text{fmt}} + R_{\text{shaping}}$ where $R_{\text{fact}}$ uses Spearman rank correlation (rewarding trend rather than absolute values), $R_{\text{contra}}$ checks semantic overlap with expert-annotated contradiction explanations, $R_{\text{fmt}}$ is a binary gatekeeper on XML-tagged structure, and $R_{\text{shaping}}$ applies asymmetric penalties: correct answer $= +1.0$, trap answer $= -2.0$.

Key Contributions¶

ChartCynics: a training-free inference pipeline that decouples diagnostic visual perception (ROI cropping + blind-test Diagnostic Agent) from OCR-based numerical extraction, resolving cross-modal conflicts via an Agentic Summarizer.
A formalization of Misleading ChartQA as inconsistency-aware fusion under a Misleading Taxonomy $T$, with formal definitions of trap answers and the deception function $f_{\text{deceptive}}$.
Oracle-Informed SFT that bridges the information gap between a data-privileged teacher and a pixel-only student model.
Deception-Aware GRPO with asymmetric reward shaping ($-2.0$ trap penalty) that trains adversarial skepticism without a dedicated Critic network.
Empirical validation that "skeptical" augmentation does not induce over-skepticism: ChartCynics also improves accuracy on standard, non-misleading charts relative to chart-parsing SOTA baselines.

Results¶

ChartCynics achieves 74.43% on the primary misleading chart benchmark and 64.55% on a second benchmark, representing an absolute $\approx +29$ percentage points over the Qwen3-VL-8B backbone (baseline: 45.57%).
SFT alone provides a +22.95 pp boost over the backbone; GRPO adds further gains to reach the full 74.43%.
ChartCynics outperforms state-of-the-art proprietary models on the MQA benchmarks (specific proprietary model scores not enumerated in the provided text excerpt).
On a mixed standard-and-misleading benchmark, ChartCynics improves basic chart comprehension over specialized chart-parsing SOTA models, demonstrating no over-skepticism regression.

Limitations¶

The architecture relies on a graphic-element detection module for ROI extraction; failure modes of this detector (missed bounding boxes, incorrect component classification) are not systematically evaluated.
OCR extraction can still hallucinate when labels are absent or partially occluded; the Dynamic Trust mechanism mitigates but does not eliminate this.
GRPO training uses oracle CSV ground-truth during reward computation ($R_{\text{fact}}$ via Spearman correlation), which assumes such structured data is available during fine-tuning—not always realistic for real-world deployment.
The five-step D-CoT and multi-agent workflow substantially increase inference latency and API/compute cost compared to single-pass VLM baselines (not quantified in the provided excerpt).
Evaluation is limited to two benchmarks; generalization to non-English charts, domain-specific visualizations (medical imaging, geographic maps), or novel deception types outside taxonomy $T$ is not demonstrated.

Relevance to Vision-Language Models¶

This work directly probes a critical robustness failure mode of modern VLMs: their susceptibility to adversarial visual encodings in structured data charts, where holistic visual encoders are systematically deceived. The agentic decomposition strategy—separating perception from verification and using RL-based alignment (GRPO) to override pre-trained visual priors—offers a transferable template for other VLM robustness challenges (e.g., image-text inconsistency detection, multimodal misinformation). The Oracle-Informed SFT approach, which bridges a data-privileged teacher and a pixel-only student, addresses a general knowledge-distillation bottleneck in chart/document VLMs. The asymmetric reward shaping in GRPO ($-2.0$ for trap answers) contributes to the emerging literature on targeted RL alignment for multimodal reasoning, complementing work on MCTS-guided VLM training and reflective CoT methods.