Skip to content

Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper systematically investigates how synthetic captions (short, descriptive, dense) interact with web-crawled AltText across three multimodal foundation models—CLIP, MLLMs, and diffusion models. The central finding is that a hybrid of synthetic captions and AltText consistently outperforms either alone, because AltText provides vocabulary diversity while synthetic captions improve image-text alignment. Each model class has a distinct preferred caption format.

Problem

It remains unclear whether high-quality synthetic captions can fully replace web-crawled AltText in multimodal pre-training, and which caption format (short vs. descriptive vs. dense) is optimal for each model class. Prior work either used synthetic captions without systematic ablation against AltText or relied on costly GPT-4V at scale. The role of caption diversity vs. alignment quality in pre-training is not well understood.

Method

The authors build a two-stage, controllable captioning pipeline on top of a 3B MM1 MLLM: - Stage 1: Fine-tune on a 1M human-annotated dataset (Stage-1-1M) with strict prompts and few-shot formatting to produce a hallucination-reduced captioner that follows format instructions. - Stage 2: Further fine-tune on expert-annotated dense descriptions (Stage-2-HA) to enable rich, scene-level captioning.

The pipeline generates four caption formats: Short Synthetic Caption (SSC, ~10–15 tokens), Descriptive Synthetic Caption (DSC, ~40–60 tokens, fits CLIP's 77-token encoder), Dense Synthetic Caption (DSC+, >100 tokens), and AltText Fusion Caption (AFC, DSC enriched with cleaned AltText). These are applied to VeCap-300M and pre-training experiments are run across CLIP (ViT-B/16), MM1 (1.2B), and Stable Diffusion 3, with systematic ratio ablations between synthetic and AltText data.

Key Contributions

  • A controllable, scalable, cost-effective MLLM-based captioning pipeline generating four caption formats without relying on GPT-4V.
  • Empirical demonstration that AltText and synthetic captions are complementary: AltText provides entity/concept diversity, synthetic captions provide alignment.
  • Per-model captioning recipes: CLIP prefers SSC + AltText (~40–50% AltText ratio); MLLM pre-training prefers DSC + AltText (66/33 ratio); MLLM post-SFT prefers DSC+ alone; diffusion models prefer DSC + AltText (~50% DSC optimal on GenEval).
  • Introduction of Average Number of Assertions (ANA) as a metric for caption richness beyond token length.

Results

  • CLIP retrieval (COCO I→T R@1): SSC + AltText: 66.67 vs. AltText alone: 54.24; All Synthetic + AltText: 70.12.
  • CLIP zero-shot ImageNet: DSC alone drops to 27.3% vs. AltText 65.7%; SSC + AltText matches AltText (66.63%) but boosts retrieval significantly.
  • CLIP linear probing: DSC (75.72%) and SSC (79.94%) are competitive with AltText (78.34%), showing richer representations than zero-shot suggests.
  • MM1 1.2B pre-training (0-shot): DSC + AltText achieves 37.09 vs. AltText baseline 34.80; optimal ratio (66% synthetic / 33% AltText) gives 37.02.
  • MM1 SFT average across 9 benchmarks: DSC+ (58.7%) > AltText (57.6%) > LLaVA captions (56.5%).
  • Diffusion GenEval average: DSC + AltText: 67.4 vs. AltText alone: 62.3; SSC + AltText: 65.5.
  • Diffusion DSG Average: DSC + AltText: 74.2 vs. AltText: 72.4.

Limitations

  • All experiments use VeCap-300M as the pre-training dataset; generalizability to other web-crawled corpora is not validated.
  • CLIP experiments are limited to ViT-B/16; scaling behavior to larger vision encoders is not studied.
  • MM1 experiments are on the 1.2B model only; larger MLLMs may have different format preferences.
  • The captioner itself is built on a 3B MM1 variant; quality ceiling relative to GPT-4V is not rigorously quantified.
  • DSC+ is the best format for MLLM SFT but also carries the highest hallucination rate among caption types—this tension is noted but not fully resolved.
  • The diversity advantage of AltText is measured via entity counts, which is a proxy for broader concept coverage, not a causal analysis.

Relevance to Vision-Language Models

This paper directly addresses a foundational data question for VLMs: how to construct optimal image-text training corpora without expensive proprietary captioners. The finding that no single caption type dominates across CLIP, MLLMs, and diffusion models has practical implications for practitioners choosing pre-training recipes. The complementary roles of AltText (diversity) and synthetic captions (alignment) explain why prior VLM works—LaCLIP, ShareGPT4V, MM1, DALL-E 3—all retained some web-crawled text even with high-quality rewrites. The model-specific format preferences (short for contrastive learning, dense for generative VLMs) provide actionable guidance for scaling future multimodal pre-training.