Skip to content

ChartGalaxy: A Dataset for Infographic Chart Understanding and Generation

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; ChartGalaxy: large-scale infographic chart understanding and generation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ChartGalaxy is a million-scale dataset of 1.76M infographic charts (synthetic + real) paired with data tables, constructed via an inductive pipeline that codifies 75 chart types, 440 variations, and 68 layout templates extracted from real designs. Fine-tuning LVLMs on it yields gains of +23–27% on infographic chart VQA and demonstrates state-of-the-art infographic chart generation, substantially outperforming GPT-Image-1 in fidelity.

Problem

Existing chart datasets overwhelmingly target plain charts and fail to capture the visual/structural complexity of infographic charts—those combining imagery, icons, and multi-element layouts with data. LVLMs trained on plain-chart corpora systematically underperform on infographic comprehension, while automated infographic generation suffers from low fidelity, limited diversity, and weak cross-modal coherence. No large-scale, structured infographic chart dataset with paired tabular data previously existed.

Method

Dataset construction follows a two-stage pipeline: 1. Real collection: 61,833 infographic charts scraped from 18 chart-rich websites (Visual Capitalist, Statista, etc.) and search engines under CC licenses; duplicates removed via perceptual hashing + CLIP similarity; tabular data extracted via a multi-step LVLM + human-in-the-loop pipeline. 2. Synthetic generation via inductive structuring: - Chart types/variations: 75 types and 440 visual-style variations implemented in D3.js. - Layout templates: 55 templates manually annotated from 1,500 real charts, then expanded to 68 via a detection model (InternImage + DINO fine-tuned on synthetic charts) that flags low-similarity layouts (measured by LTSim), followed by \(k\)-means clustering (\(k=50\)) and manual curation. - Chart creation: (a) data tables curated from VizNet, UN data, etc. plus Gemini-2.0-Flash synthetics; (b) text generated with retrieval-augmented prompting via Sentence-BERT; (c) images retrieved from a 681K-icon repository; (d) layout optimized by maximizing ink ratio subject to template spatial constraints and minimum inter-element distance \(d(\partial e_i, \partial e_j) \geq p\), solved by grid search.

Applications demonstrated: - Understanding: 443K QA pairs across text-based, visual-element-based, and visual-understanding question types used to fine-tune InternVL3-8B and Qwen2.5-VL-7B. - Code generation: 500-chart benchmark evaluating Direct Mimic (D3.js code generation from image) via SVG/PNG similarity. - Chart generation: Example-based generation compared against GPT-Image-1 via user study on fidelity, aesthetics, and creativity.

Key Contributions

  • Inductive pipeline that automatically extracts and expands layout templates from real infographic charts using a bootstrapped detection model.
  • ChartGalaxy dataset: 1,701,356 synthetic + 61,833 real infographic charts, each paired with tabular data; 75 chart types, 440 variations, 68 layout templates.
  • Infographic chart understanding instruction dataset (443K QA pairs, 7 question types) and an independent human-verified evaluation set (2,176 charts, 4,975 QA pairs).
  • Infographic code generation benchmark (500 charts, ground-truth PNG/SVG/table triplets).
  • Example-based infographic chart generation method evaluated against GPT-Image-1.

Results

Understanding (fine-tuning with ChartGalaxy): - InternVL3-8B: +3.80% on InfographicVQA, +5.98% on ChartQAPro; +26.87% overall on independent eval set - Qwen2.5-VL-7B: +4.44% on InfographicVQA, +3.59% on ChartQAPro; +23.85% overall on independent eval set - Largest gains on visual understanding sub-tasks: style detection +60.49% (InternVL3), visual encoding analysis +40.78% (both models) - Closed-source LVLM baselines on ChartGalaxy benchmark (no fine-tuning): Gemini-2.5-Pro 85.21, GPT-4.1 80.00, Claude-3.7-Sonnet 79.91, Qwen2.5-VL-72B 61.29

Chart generation (user study, 5-point scale): - Ours vs. GPT-Image-1: Fidelity 4.63 vs. 2.10; Aesthetics 4.14 vs. 2.90; Creativity 3.95 vs. 2.65

Limitations

  • Synthetic charts are generated programmatically from templates and may not fully replicate the organic irregularities of human-designed infographics.
  • Layout templates were initialized from only 1,500 manually annotated charts from two sources (Statista, Visual Capitalist), potentially biasing style coverage.
  • The code generation benchmark is limited to 500 charts and only evaluates D3.js output; other rendering targets are not assessed.
  • The VQA evaluation set covers only synthetic charts (bounding-box annotations required for visual-element questions), leaving real-chart evaluation of visual-element reasoning incomplete.
  • Tabular data extraction from real charts relies on LVLMs, which can introduce noise despite human verification.

Relevance to Vision-Language Models

ChartGalaxy directly addresses a documented blind spot in LVLM pretraining: infographic-style visual reasoning requiring simultaneous interpretation of imagery, icons, layout, and embedded data. The +23–27% absolute gains from fine-tuning 7–8B models signal that the gap between plain-chart and infographic-chart capability is large and tractable with targeted data. The dataset's paired chart-table structure and structured QA taxonomy (distinguishing text-based, visual-element, and visual-understanding reasoning) provide a fine-grained diagnostic for where VLMs fail on complex mixed-modality documents. The code generation benchmark further probes structural understanding—whether models can reconstruct visual programs from infographic images—complementing existing chart-QA benchmarks like ChartQAPro and InfographicVQA.