Skip to content

MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MM1.5 is a family of multimodal LLMs (1B–30B, dense and MoE) from Apple that takes a data-centric approach to post-pre-training, systematically ablating data mixtures across three training stages to improve text-rich understanding, visual referring/grounding, and multi-image reasoning. The core finding is that careful data curation—not architectural changes—drives substantial capability gains over the MM1 baseline. Even 1B and 3B models achieve competitive performance when trained with the optimized recipe.

Problem

Building high-performing MLLMs is highly empirical, yet the literature offers limited insight into how different SFT data categories interact and trade off against each other, how continual pre-training with OCR data should be configured, and how dynamic high-resolution image splitting should be designed. Most open-source models also lack native visual referring and grounding, relying instead on workarounds like set-of-mark prompting.

Method

MM1.5 retains the MM1 architecture (CLIP encoder + CAbstractor connector + LLM backbone) and introduces a three-stage training pipeline:

  1. Large-scale pre-training with 2B image-text pairs, 600M interleaved documents, and 2T text-only tokens. Pre-training data ratio adjusted from 45:45:10 to 50:10:40 (image-text : interleaved : text-only), heavily downweighting interleaved data.

  2. Continual pre-training at high resolution (1344×1344) on 45M OCR documents (PDFA, IDL, Renderedtext, DocStruct-4M). Ablations show this stage requires ≥1344×1344; lower resolutions (378×378) can underperform no continual pre-training at all. Synthetic captions did not conclusively outperform OCR-only data.

  3. Supervised fine-tuning (SFT) with a curated mixture: 80% single-image (37.2% text-rich, 22.5% refer&ground, 11.3% general, 5.6% math, 2.3% code, 1.1% science), 10% multi-image, 10% text-only. Mixing ratios are determined via per-category ablations using a MMBase score (average of general, text-rich, knowledge benchmarks). Key ratios: α_rg=2.0, α_math=0.5, α_code=0.2, α_science=0.1 relative to the general category.

Dynamic image splitting uses (nmin, nmax)=(4,9) during training, supports up to 4MP (~2016Ă—2016), with an overview image placed after sub-images and index-based position indicators. Inference can be extended to (4,16) without retraining.

MoE variants (1B-MoE, 3B-MoE) use 64 experts with dense-to-MoE replacement every two layers, top-2 gating, load-balance loss (0.01), and router z-loss (0.001).

Key Contributions

  • MM1.5 model family: 1B, 3B, 7B, 30B dense models plus 1B-MoE and 3B-MoE variants.
  • Systematic per-category SFT ablations revealing capability trade-offs (e.g., refer&ground data slightly hurts general/text-rich; multi-image data hurts MMBase but boosts multi-image benchmarks).
  • Identification of high-resolution (1344Ă—1344) continual pre-training with OCR data as essential for text-rich performance.
  • Dynamic AnyRes image splitting analysis: larger nmax significantly improves DocVQA/InfoVQA; overview image after sub-images improves decoder attention.
  • Native visual referring and grounding without requiring set-of-mark prompts.
  • Demonstration that MoE at 3B can surpass dense 7B on several capability dimensions.

Results

  • MM1.5-30B vs MM1-30B: MathVista 55.6 vs 39.4; DocVQA 91.4 vs 75.8; InfoVQA 67.3 vs 47.3; TextVQA 79.2 vs 73.5; RealWorldQA 69.0 vs 59.4.
  • MM1.5-1B: DocVQA 81.0 vs SPHINX-Tiny 53.0; InfoVQA 50.5 vs 26.3; outperforms LLaVA-OneVision-0.5B on ScienceQA (82.1 vs 67.2) and DocVQA (81.0 vs 70.0).
  • MM1.5-3B: DocVQA 87.7 vs MiniCPM-V 2.0 71.9; MathVista 44.4 vs 38.7; exceeds Phi-3-Vision (4.2B) on DocVQA (87.7 vs 83.3) and in-context learning.
  • MM1.5-3B-MoE: Surpasses MM1.5-7B on knowledge (42.9 vs 41.8 MMMU), general (73.3 vs 73.4 SEED-I), grounding (86.0 vs 86.9 RefCOCO avg), and multi-image (59.6 vs 56.0 NLVR2).
  • Visual grounding: MM1.5-3B outperforms Ferret-7B and matches Ferret-13B (specialists) on RefCOCO/LVIS grounding metrics.
  • In-context learning (VL-ICL): MM1.5-30B 77.6 vs MM1-30B 52.1.

Limitations

  • Like most MLLMs, MM1.5 may generate harmful or counterfactual outputs; no dedicated safety alignment reported.
  • CLIP encoder and LLM backbone are in-house Apple models, limiting full reproducibility.
  • Grounding performance is highly sensitive to image grid configuration changes, complicating inference-time resolution scaling.
  • Synthetic caption benefit over OCR-only data is inconclusive; self-training captions showed "some promise" but are deferred to future work.
  • Interleaved pre-training was heavily downweighted based on empirical tuning; the theoretical basis for the 50:10:40 ratio is not fully explained.

Relevance to Vision-Language Models

MM1.5 provides one of the most rigorous published ablation studies on MLLM training data composition, offering concrete guidance on how SFT data categories interact—findings directly applicable to any VLM training pipeline. The native integration of visual referring and grounding without architectural modification advances the capability frontier for open-weight models, where this feature has been largely absent. The demonstration that MoE scaling at 3B can match dense 7B on multiple benchmarks is directly relevant to efficient VLM deployment. The continual pre-training findings establish OCR-heavy high-resolution data as a necessary bridge stage between pre-training and SFT for text-rich tasks.