Skip to content

MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Engine

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MAVIS introduces a fully automated data engine for generating large-scale mathematical visual instruction-tuning data (558K diagram-caption pairs + 834K problem-solution pairs with CoT) and a four-stage training pipeline that sequentially improves math-diagram encoding, vision-language alignment, instruction following, and chain-of-thought reasoning. The resulting 7B model, MAVIS-7B, outperforms all open-source MLLMs on visual math benchmarks and even surpasses GPT-4V on several tasks.

Problem

Existing MLLMs fail at visual mathematical reasoning due to three compounding gaps: (1) general-domain CLIP encoders do not attend to the abstract, monochromatic symbols and shapes in math diagrams; (2) vision-language alignment is trained on natural images, leaving LLMs unable to interpret math-diagram embeddings; and (3) CoT reasoning degrades when diagrams are present rather than text-only, because models lean on textual shortcuts instead of visual information. Large-scale, accurately annotated visual math datasets with step-by-step rationales are prohibitively expensive to produce manually.

Method

Automatic data engine (rule-based, no GPT API at generation time): - Diagram generation: Python/Matplotlib renders plane geometry (iterative shape extension), analytic geometry (multi-figure Cartesian plots), and function graphs (7 function families with parameterised equations). - MAVIS-Caption (558K pairs): captions assembled by filling GPT-4-authored templates with diagram attributes—no API call per sample. - MAVIS-Instruct (834K problems): problems and CoT rationales derived by backward deduction from target attributes (geometry) or GPT-4-generated reasoning templates (functions); supplemented by GPT-4V augmentation of 4K manually collected problems (→83K) and two existing datasets Geometry-3K + GeoQA+ (→80K).

Four-stage training pipeline: 1. CLIP-Math: Contrastive fine-tuning of CLIP ViT-L on MAVIS-Caption (frozen text encoder, trainable vision encoder) to specialise for math-diagram representations. 2. Diagram-language alignment: CLIP-Math frozen; train a two-layer MLP projection and LoRA-adapted Mammoth2-7B LLM on diagram-captioning with MAVIS-Caption. 3. Instruction tuning: Unfreeze projection + LoRA; train on MAVIS-Instruct with text-lite, visual-cue-forcing problem formats. 4. DPO preference alignment: Generate CoT on 582K engine problems; filter wrong answers (88K negatives) and pair against annotated positives; fine-tune LoRA only.

Key Contributions

  • Fully automated, GPT-API-free data engine covering plane geometry, analytic geometry, and functions at scale.
  • MAVIS-Caption: 558K diagram-caption pairs for math-specific visual encoding and alignment.
  • MAVIS-Instruct: 834K CoT-annotated visual math problems with minimised textual redundancy.
  • CLIP-Math: a domain-adapted vision encoder with improved attention to math-diagram features.
  • Progressive four-stage training (encode → align → instruct → DPO) systematically addressing each identified deficiency.
  • MAVIS-7B: a 7B open-source MLLM achieving state-of-the-art on multiple visual math benchmarks.

Results

  • MathVerse (testmini, CoT-E, all versions): MAVIS-7B 35.2 vs. next-best open-source InternLM-XComposer2-7B 25.9 (+9.3%) and LLaVA-NeXT-110B 28.3 (+6.9%).
  • Surpasses G-LLaVA-7B by +19.5% and Math-LLaVA-13B by +11.1% on MathVerse CoT-E.
  • MMMU-Math: MAVIS-7B outperforms closed-source Qwen-VL-Max by +6.1%.
  • MathVision: surpasses Qwen-VL-Max by +3.6%.
  • We-Math (S1/S2/S3): outperforms Qwen-VL-Max by ~+10% across subsets; DPO gain increases from S1 to S3 (harder multi-step problems benefit more).
  • MathVista (GPS/ALG/GEO): exceeds GPT-4V on all three categories.
  • GeoQA: 68.3% vs. prior best G-LLaVA 66.4%; FunctionQA: 50.0%.
  • DPO stage improves CoT-E more than raw accuracy, confirming its targeted benefit on reasoning quality.

Limitations

  • MAVIS-Instruct still uses GPT-4V for augmenting manually collected and existing-dataset problems (252K of 834K), so the pipeline is not entirely API-free.
  • Coverage is restricted to three diagram types (plane geometry, analytic geometry, functions); real-world math includes statistics, 3D geometry, commutative diagrams, etc.
  • Text-lite problem format is designed to force visual attention, but evaluation is still on predefined benchmarks that may not reflect open-ended mathematical reasoning.
  • DPO training uses final answer correctness as the signal for negative mining, which may not catch subtly wrong but answer-correct reasoning chains.
  • Vocabulary size of 149 in MAVIS-Caption suggests limited linguistic diversity in captions, potentially narrowing generalisation.

Relevance to Vision-Language Models

MAVIS directly addresses a documented failure mode of standard VLMs—poor visual encoding of domain-specific, non-photographic content—by proposing a targeted encoder fine-tuning strategy (CLIP-Math) alongside a scalable, annotation-free data pipeline. The four-stage curriculum (encoder specialisation → cross-modal alignment → instruction tuning → preference optimisation) is a reusable template for adapting general-purpose VLMs to structured visual domains beyond mathematics. The paper also demonstrates that minimising textual redundancy in training prompts is an effective inductive bias to prevent VLMs from bypassing visual reasoning, a finding with broader implications for multimodal training design. For researchers tracking VLMs, this work establishes a practical benchmark: a 7B specialist model trained on synthetically generated data outperforms 110B general-purpose models on math-visual tasks.