DataComp-VLM: Improved Open Datasets for Vision-Language Models

🕒 Published (v1): 2026-06-26 19:11 UTC · Source: Arxiv · link

Why this paper was selected

DataComp series has proven impact; open VLM dataset benchmark is foundational for the whole field

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DCVLM is the first systematic benchmark for controlled data-centric experiments in autoregressive VLM training, built from 160 datasets (6T multimodal tokens) across four data types. The central finding is that data mixing, not filtering, drives performance: instruction-heavy mixtures scale significantly better than caption-heavy ones, with the gap widening at larger model sizes and token budgets. The resulting DCVLM-BASELINE dataset achieves 63.6% on a 33-task core suite with an 8B model (200B tokens), outperforming FineVision by +5.4pp.

Problem

VLM progress has focused overwhelmingly on architectures and training recipes, treating data curation as a second-class citizen. The community lacks a systematic benchmark for evaluating curation strategies (filtering, mixing, formatting, sampling) in the modern VLM setting. Several factors make this harder than for text-only or contrastive models: (1) modern VLMs aggregate multiple data types (image-caption pairs, interleaved multimodal documents, text-only corpora, instruction-tuning data) that have already undergone different degrees of upstream curation; (2) existing open training datasets operate at millions of samples, far below the trillions of tokens used by SOTA models; (3) the interaction between data types, model scale, and training budget creates a massive design space; and (4) VLM evaluation lacks standardization, making fair cross-dataset comparisons difficult.

Method

DCVLM fixes model architecture and training recipe, allowing participants to vary only data curation. The architecture mirrors InternVL3: an InternViT-300M vision encoder, 2-layer MLP projector, and Qwen2.5-Base LLM, with AnyRes dynamic tiling (448×448 tiles, 256 visual tokens each after pixel shuffling).

• Data pool: 160 publicly available datasets across four types: (1) image-caption pairs (e.g., DataComp-1B, ReLAION-2B, ShareGPT-4o), (2) multimodal interleaved documents (e.g., MINT-1T, WanJuan, OmniCorpus), (3) text-only data (e.g., FLAN, SlimOrca, Numina-Math-1.5), and (4) multimodal instruction-tuning data (categorized into 8 capabilities following FineVision). Total: 6T multimodal tokens.
• Scales: four tiers — small (1B model, 6.25B tokens), medium (2B, 25B), large (4B, 100B), x-large (8B, 200B). Each step is an 8× compute increase (2× model size, 4× tokens). Pool-to-training-token ratio fixed at 30×.
• Evaluation: 52 benchmarks across 9 domains, filtered for stability and monotonicity into three nested tiers: Validation (13 benchmarks), Core (33), Extended (52). Safety, Hallucination, and Reasoning are deferred to Extended.
• Key experiments: 60+ filter configurations (CLIP-score, text-quality classifiers, multimodal filters, perplexity-based); mixing experiments varying the image-caption \(\leftrightarrow\) instruction-tuning ratio across 3 model sizes × 3 token budgets; controls for backbone choice and SFT transfer.

Key Contributions

• First benchmark designed for systematic, controlled data curation research in autoregressive VLMs.
• Mixing over filtering: In the modern data landscape where datasets already have upstream curation, additional quality filtering provides diminishing (often negative) returns. The dominant performance lever is data mixing.
• Scale-dependent mixture rankings: Instruction-heavy mixtures (10% image-caption, 70% instruction-tuning) start as the worst at smallest scales but become the best at larger scales — small-scale proxy experiments would select the wrong mixture.
• Pretraining performance predicts post-SFT performance with near-perfect fidelity (Pearson \(r = 0.99\) across 54 SFT runs), validating pretraining-only metrics for data curation research.
• DCVLM-BASELINE, a new state-of-the-art open VLM training dataset, along with full infrastructure release (data pool, evaluation suite, model checkpoints at 4 scales).

Results

• Filtering experiments (60+ configurations at small and medium scale): best result was +0.8pp (SigLIP-2 global filtering on image-text pairs); most filters leave baseline unchanged or hurt. Performance degrades as upstream pre-filtering in the pool increases (gain drops from +2.4pp at 25% pre-filtered to +0.6pp at 100% pre-filtered).
• Mixing scaling grid (3 model sizes × 3 token budgets): Instruction-heavy mixture shows a steeper scaling slope — worst at 1B×6.25B but best at 2B×25B and 4B×25B. Caption-heavy mixture dominates at small scale but underperforms as compute increases.
• DCVLM-BASELINE (x-large: 8B model, 200B tokens): 63.6% on Core set (33 tasks), outperforming FineVision by +5.4pp. A 4B model trained for 100B tokens on DCVLM-BASELINE beats an 8B model trained on FineVision for 200B tokens — a 4× compute reduction.
• Instruction-tuning repeatability: Each doubling of data repetitions costs ~0.5–1.0pp. Even at 4× repetitions, instruction-heavy (49.8%) matches caption-heavy with unique data (50.3%).
• Control experiments: Pearson \(r = 0.99\) between pretraining and post-SFT performance; data mixture rankings preserved when switching from Qwen2.5-Base to Qwen2.5-Instruct (\(r = 0.97\)).

Limitations

• The data pool consists of already-curated public datasets (not raw web crawls), so the finding that filtering provides diminishing returns may not generalize to training from unfiltered web data.
• Instruction-tuning datasets are orders of magnitude smaller than caption corpora, requiring data repetitions at high instruction proportions; although performance degrades gracefully, this is a practical constraint.
• All experiments use a single architecture family (InternVL-style with InternViT-300M and Qwen2.5) — results may not transfer to other VLM designs (e.g., different vision encoders or LLM backbones).
• Maximum scale tested is 8B / 200B tokens; findings at even larger scales (which SOTA proprietary models use) may differ.
• Mixing experiments vary only the image-caption \(\leftrightarrow\) instruction-tuning axis, with text-only (15%) and multimodal documents (5%) held fixed.

Relevance to Vision-Language Models

DCVLM provides the first controlled benchmark for understanding what makes VLM training data effective, directly challenging the assumption (carried over from text-only pretraining) that aggressive quality filtering is the primary lever — for already-curated VLM pools, mixing ratios dominate. The finding that mixture rankings reverse with scale is a critical practical warning: small-scale proxy experiments (1B models, 6.25B tokens) would select exactly the wrong mixture, underscoring the need for scale-aware data curation in VLM development.