Skip to content

ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understanding

🕒 Published (v1): 2026-03-28 00:45 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; million-scale high-quality chart dataset for geometry-numerical-language reasoning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ChartNet is a 1.5-million-sample open-source multimodal dataset for chart understanding, built via a code-guided synthetic generation pipeline that produces fully aligned 5-tuples (image, plotting code, CSV table, natural-language summary, CoT QA). Fine-tuning on ChartNet yields consistent, large gains across all chart understanding tasks for models ranging from 256M to 7B parameters, with the best fine-tuned model outperforming GPT-4o and models an order of magnitude larger.

Problem

Existing chart understanding datasets are limited in scale (at most ~680K samples), narrow in chart-type coverage (typically 1–6 types), single-task in scope, and incomplete in modality coverage—most lack plotting code, grounding annotations, reasoning traces, or human verification simultaneously. This data scarcity is the primary bottleneck preventing open-source VLMs from matching proprietary systems on complex chart reasoning tasks that require joint visual perception, structured numerical extraction, and language generation.

Method

ChartNet uses a code-guided synthesis pipeline with five stages:

  1. Chart-to-Code Reconstruction: A VLM (pixtral-large-instruct-2411) converts 150K seed images (from TinyChart) into executable Python plotting code.
  2. Code-Guided Chart Augmentation: An LLM (gpt-oss-120b) iteratively rewrites the code to vary chart type, plotting library, data values, and style, scaling each seed to an arbitrary number of variations across 24 chart types and 6 plotting libraries (matplotlib, pygal, etc.).
  3. Chart Rendering: All generated scripts are executed; ~77% succeed.
  4. Quality Filtering: A VLM classifies rendered images for visual defects (overlapping text, cropped labels, obscured features); ~36.5% of images are discarded, reducing human-detectable issues from 14.9% to 5.9%.
  5. Code-Guided Attribute Generation: A VLM extracts CSV tables and generates natural-language summaries grounded in both image and code context.

CoT QA pairs are generated via the Vision-R1 framework: pixtral-large-instruct-2411 generates complex reasoning questions; a four-step Pseudo-CoT (Summary→Caption→Reasoning→Conclusion) is constructed via separate model calls; modality bridging converts visual content to text; gpt-oss-120b produces final <think>/<answer>-tagged reasoning traces.

Specialized subsets add: 96K human-annotated synthetic charts, 30K real-world charts (World Bank, Pew Research, etc.), geometry-derived grounding QA pairs (bounding boxes from plotting code elements, entropy-filtered), and 7K safety preference pairs for adversarial/jailbreak scenarios.

Key Contributions

  • A code-guided synthetic data pipeline that treats plotting code as a structured intermediate, enabling controllable, scalable chart generation across 24 types and 6 libraries with automated quality filtering.
  • ChartNet: the largest open-source chart dataset (1.5M tuples), providing complete 5-tuple modality alignment (image + code + CSV + text + CoT QA) plus grounding, safety, human-annotated, and real-world subsets—the only dataset to cover all these simultaneously (Table 1).
  • Empirical validation showing consistent, scale-invariant fine-tuning gains across all five models tested (256M–7B), including ultra-compact models gaining full chart-reconstruction capability from zero baseline and a 7B model surpassing GPT-4o.

Results

ChartNet evaluation set (Table 2) — fine-tuning gains per model: - SmolVLM-256M: gains of +14.9 pp Exec., +56.8 pp Code-D, +74.9 pp Code-S, +77.5 pp Img., +33.4 pp Chart Summarization - Granite-Vision-2B: Exec. 63.4→90.4 (+27.0), Img. 60.0→83.9 (+23.0), Chart Data Extraction 53.8→70.3 (+16.5) - Qwen2.5-VL-3B: Chart Reconstruction Code-D +10.6, Chart Summarization +9.5 - LLaVA-7B: Code-D 27.0→69.4 (+42.4), Exec. 45.3→83.9 (+38.6), Chart QA 55.1→70.3 (+15.2)

Off-the-shelf models vs. ChartNet fine-tuned (Table 3): GPT-4o achieves 95.9 Exec. / 48.8 Code-D / 77.2 Code-S / 88.2 Img.; fine-tuned Granite-Vision-2B at 90.4 Exec. matches or exceeds many larger open-source models (Qwen2-VL-72B: 83.1 Exec.).

Public benchmarks (Table 4): - Granite-Vision-2B-ChartNet: ChartCap METEOR 6.4→30.1 (+23.7), ChartMimic-v2 direct 30.84→58.42 (+27.6) - SmolVLM-256M-ChartNet: ChartMimic-v2 direct 0→12.67 from zero baseline

Limitations

  • Synthetic distribution shift: despite real-world and human-annotated subsets, the 1.5M core samples are synthetic and may not capture the full stylistic diversity of production charts.
  • Seed dependency: pipeline is initialized from TinyChart's 150K images; diversity of the augmented corpus inherits any biases in that seed set.
  • Execution rate ceiling: ~23% of generated scripts fail to execute, and ~36.5% of rendered images are discarded, indicating pipeline inefficiency and potential systematic gaps in chart-type coverage.
  • Evaluation relies on GPT-4o as an automated judge for most tasks (except CoT QA), introducing judge-model bias and cost.
  • Largest model tested is only 7B parameters; behavior at frontier scale (≥13B) is not studied.
  • Grounding QA pairs are template-based, which may limit linguistic diversity of grounding supervision.

Relevance to Vision-Language Models

ChartNet directly addresses the data bottleneck limiting VLM chart-understanding capabilities, establishing that code-space augmentation—rather than image-space manipulation—is an effective strategy for generating diverse, semantically aligned multimodal training data at scale. The complete 5-tuple alignment (image, code, table, text, CoT QA) provides a richer supervisory signal than any prior chart dataset, enabling tight visual–numerical–linguistic coupling that current VLMs lack. The scale-invariant fine-tuning gains (from 256M to 7B) suggest the dataset is complementary to architectural improvements and can serve as a drop-in training resource for any VLM regardless of size. The safety subset is a notable addition, providing chart-domain preference data for RLHF/DPO alignment—an underexplored area in document-understanding VLMs.