Skip to content

Should VLMs be Pre-trained with Image Data?

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper systematically investigates when and how to introduce image data during VLM pre-training, training ~300 models across scales, data ratios, and checkpoint timings. The central finding is that inserting image-text data at 80% through text-only pre-training—rather than after full LLM pre-training—yields better vision-language performance without sacrificing text quality. At 1B scale, this timing choice provides a 2% average improvement across 6 diverse tasks.

Problem

The dominant VLM training paradigm fully pre-trains an LLM on text, then adds vision in a separate stage (LLaVA, Flamingo, PaliGemma, etc.). Whether integrating images earlier in pre-training is beneficial, and precisely when/how to do so, is empirically undocumented. The two-step pipeline's cost in downstream performance—versus earlier image introduction—is unknown.

Method

The authors train a 1.4B decoder-only transformer (DCLM-1B backbone, SigLIP 400M frozen encoder, two-layer MLP projection) using a three-stage process: (1) partial text-only pre-training to a chosen checkpoint (0%, 20%, 40%, 60%, 80%, or 100% of 4.3T tokens), (2) image-text continued pre-training for 28B tokens by resuming the existing cosine LR schedule (rather than re-warming), mixing DCLM-Baseline text with DataCompDR-1B captions at varying ratios, and (3) LLaVA instruction fine-tuning for 1–4 epochs. They ablate each axis independently: checkpoint timing, image-text ratio, instruction data in pre-training, and fine-tuning duration. A "stable score" aggregate metric (accuracy minus random baseline, averaged across tasks) is used to compare configurations.

Key Contributions

  • Demonstrates that introducing images at 80% through text pre-training outperforms the standard fully-pretrained + re-warmup approach by ~2% average across 6 VL tasks at 1B scale.
  • Shows that continued LR schedule (no re-warmup) when adding images is critical; re-warming the fully pre-trained model degrades performance.
  • Identifies an optimal image-text ratio of 10–20% visual tokens at 1B scale during the image-text pre-training phase; ratios outside this range hurt both modalities.
  • Finds that training from scratch with mixed image-text data (without prior text pre-training) consistently harms both vision and text performance at 28B tokens.
  • Establishes that instruction fine-tuning data should be excluded from the image-text pre-training phase—including it hurts VL performance despite slight text gains.
  • Shows fine-tuning for 2–4 epochs is optimal; beyond 4 epochs, both vision and text performance degrade (overfitting).

Results

  • Timing: 80% text pre-training checkpoint + image-text cooldown → ~2% average improvement over 100% pre-trained + re-warmup baseline on 6 VL tasks (1B model).
  • Image-text ratio: 10% images during cooldown is near-optimal for 1B; too few (<1%) or too many (100%) images both degrade downstream performance.
  • Training from scratch: All image fractions tested when starting from random initialization yield worse vision and text performance than the partial-text-pretraining strategy at 28B tokens.
  • Instruction data in pre-training: Any mixing of instruction tuning data into the pre-training image mix reduces VL stable score; best VL results come from 0% instruction data in pre-training.
  • Fine-tuning epochs: Vision performance peaks at 4 epochs; text performance degrades monotonically with more epochs; 2 epochs is the practical sweet spot for retaining text ability.
  • Reported on VQA-v2, ARC-easy, and a multi-benchmark "stable score" suite (0-shot throughout).

Limitations

  • Experiments capped at 1.4B parameters; the optimal image-text ratio and timing may shift at larger scales (authors note ratio trends differ at 79M vs. 1B).
  • Only single-image caption data used (no interleaved multi-image sequences like OBELICS/MINT-1T); findings may not generalize to interleaved pre-training.
  • Image encoder (SigLIP 400M) is kept frozen throughout; end-to-end training dynamics are not explored at scale.
  • The 28B-token image-text pre-training budget is near Chinchilla-optimal for 1.4B but may be insufficient for training-from-scratch conclusions to hold at scale.
  • Fine-tuning uses only LLaVA-1.5 mix; generalization to other instruction sets is not evaluated.

Relevance to Vision-Language Models

This paper directly challenges the implicit assumption shared by LLaVA, Flamingo, PaliGemma, DeepSeek-VL, and most production VLMs—that text pre-training should be completed before vision is introduced. The empirical finding that the LR schedule continuity (resuming vs. re-warming) matters as much as timing has immediate implications for practitioners choosing training recipes. For researchers tracking VLMs, the image-text ratio sweet spot (10–20% at 1B) and the strict separation of caption pre-training from instruction fine-tuning are actionable design principles that complement concurrent work on connector architecture (LLaVA, Prismatic) and data quality (DataComp-DR, Recap-DataComp). The study fills a documented gap in open-source VLM training transparency, as natively multimodal models (Pixtral, Gemini, Fuyu) have not disclosed their pre-training strategies.