Zebra-CoT: A Dataset for Interleaved Vision-Language Reasoning¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; Zebra-CoT large dataset for interleaved vision-language reasoning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Zebra-CoT is a large-scale interleaved text–image reasoning dataset of 182,384 traces across 18 domains, designed to train VLMs to natively generate visual chain-of-thought (visual CoT) — intermediate reasoning images alongside text — rather than relying on external tools or text-only reasoning. Fine-tuning Anole-7B on it improves in-distribution accuracy from 4.2% to 16.9% and yields up to +13.1% gains on external VLM benchmarks, while fine-tuning Bagel-7B unlocks native interleaved image generation suitable for RL.

Problem¶

Current VLMs reason almost exclusively in text, discarding the visual modality during chain-of-thought. Prior approaches to visual CoT rely on agentic pipelines with external tools (visual programming), which are brittle and non-native. The few existing interleaved datasets are either web-scraped noise without logical reasoning structure or restricted to narrow domains (e.g., VisualCoT is visual search only). This leaves no diverse, high-quality supervised dataset to train models for innate, multi-domain visual CoT, and without such data reinforcement learning from weak visual CoT signals is infeasible.

Method¶

The authors curate a two-pronged pipeline. For real-world domains (geometry, physics, chemistry, chess, VQA, robotics), they collect openly licensed data and use Gemini-2.5 Pro to denoise, parse, and logically enrich traces into the interleaved format: <question> → <text-reasoning-1> → <visual-reasoning-1> → ... → <answer>. For synthetic domains (graph algorithms, jigsaw, mazes, 3D multi-hop object counting), they use programmatic image generation and templated text filled in by frontier VLMs. Four macro-categories are covered: (a) scientific reasoning (geometry, physics, chemistry, algorithms, graph problems), (b) 2D visual reasoning (visual search with bounding-box/zoom CoT, jigsaw), (c) 3D visual reasoning (ALFRED-reformulated embodied planning, RoboMIND robot planning, CLEVR-inspired multi-hop 3D object counting), and (d) visual logic and strategic games (mazes, chess, checkers, Connect Four, Tetris, ciphers, ARC-AGI, RPM). Reasoning images per trace range from 1 to 8+, with the distribution varying by category. They fine-tune Anole-7B (a model capable of interleaved image–text generation) and Bagel-7B (a high-quality VLM not originally capable of interleaved generation) on this data.

Key Contributions¶

182,384 interleaved text–image reasoning traces across 18 subcategories and 4 macro-categories, released as the first diverse open-source visual CoT training dataset.
Scaffolding analysis showing frontier models (GPT-5: 41.98%, Claude Sonnet 4: 27.61%, Gemini 2.5 Pro: 24.93% zero-shot) improve dramatically to an average of 56.70% (+25.19 pts) when given two multimodal CoT steps, isolating the value of visual reasoning steps.
Anole-Zebra-CoT-7B: fine-tuning raises in-distribution accuracy from 4.2% → 16.9% and achieves average +4.9% on 7 external benchmarks, with a peak of +13.1% on a visual logic benchmark.
Bagel-Zebra-CoT-7B: fine-tuning unlocks native interleaved image generation in a model that previously could not produce reasoning images, enabling future RL training.
Ablation confirming visual CoT steps are essential: text-only CoT traces yield substantially smaller gains and sometimes degrade performance because textual steps reference missing images.

Results¶

Zero-shot frontier model accuracy on Zebra-CoT tasks: GPT-5 41.98%, Claude Sonnet 4 27.61%, Gemini 2.5 Pro 24.93%; average 31.51%.
Scaffolding (1 multimodal step): average rises to 47.99% (+16.48 pts across three models).
Scaffolding (2 multimodal steps): average rises to 56.70% (+25.19 pts); maze tasks jump from 52.59% → 96.36% (+43.77 pts); graph reasoning from 3.92% → 22.03%.
Anole-7B fine-tuned (in-distribution test): 4.2% → 16.9% (+12.7 pts).
Anole-Zebra-CoT-7B on 7 external VLM benchmarks: average +4.9%, max +13.1% on a visual logic benchmark.
Bagel-7B fine-tuned: qualitatively generates coherent interleaved visual reasoning (no aggregate benchmark number reported for Bagel in the excerpt).

Limitations¶

Fine-tuning gains evaluated on only one interleaved-capable base model (Anole-7B) with quantitative external benchmarks; Bagel results are qualitative only in the provided text.
Scaffolding experiments select a subset of tasks (those amenable to QA evaluation), excluding robotic planning and embodied CoT, potentially overstating or underrepresenting gains across the full dataset.
Visual CoT traces rely heavily on Gemini-2.5 Pro for enrichment; quality and logical coherence are thus bounded by that model's capabilities and potential hallucinations.
Frontier model evaluation uses zero-shot prompting without chain-of-thought elicitation, which may underestimate baseline capabilities.
No reported RL training results; Bagel fine-tuning suitability for RL is argued qualitatively, not demonstrated.

Relevance to Vision-Language Models¶

Zebra-CoT directly addresses one of the most fundamental open problems in VLM research: enabling models to reason natively through the visual modality rather than collapsing everything to text. By providing the first large-scale, multi-domain supervised dataset of interleaved text–image reasoning traces, it fills the data gap that has blocked both supervised fine-tuning and RL-based approaches to visual CoT. The scaffolding experiments quantify precisely how much frontier VLMs are bottlenecked by their inability to generate intermediate visual representations, establishing a clear benchmark for progress. For researchers tracking VLMs, this work is a prerequisite resource for the emerging line of innate visual reasoning models and complements efforts on multimodal RL (e.g., R1-style training extended to image generation).