Skip to content

MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MIA-Bench is a 400-item benchmark that evaluates multimodal LLMs on strict adherence to compositional, layered instructions paired with images—a dimension left largely unmeasured by existing VQA and open-ended chat benchmarks. Evaluation of 29 MLLMs reveals wide performance gaps, and targeted supervised fine-tuning on synthetically generated diverse instruction-tuning (DIT) data yields ~10-point improvements without catastrophic forgetting.

Problem

Existing MLLM benchmarks either use fixed-form VQA (short/multiple-choice answers, easy to score but not reflective of real use) or open-ended free-form chat (loosely defined instruction compliance). Neither precisely measures a model's ability to satisfy multiple simultaneous, explicit, compositional constraints embedded in a single prompt—e.g., length limits, grammatical requirements, genre, perspective, OCR, and description simultaneously. Instruction-adherence benchmarks like IFEval exist for text-only LLMs but not for multimodal models.

Method

Each of the 400 image-prompt pairs is manually crafted to contain multiple sub-instructions drawn from 8 categories: description, length limit, mention, genre, grammar, math, perspective, OCR. Instructions span 5 complexity tiers (basic → complex). Three design principles are enforced: correctness (human-answerable), no answer leakage, and image-dependence (cannot be solved without the visual input). Scoring is performed by GPT-4o as judge: each sub-instruction receives a weight (summing to 10 per prompt), and compliance with each is scored independently; the final score is the average percentage across all 400 prompts. For SFT, 5,000 image-prompt pairs are generated from COCO 2017 train images using GPT-4v (generating 5 difficulty-tiered instructions per image), with GPT-4v also generating the ground-truth responses (90% manual verification pass rate). LLaVA-NeXT-13b is then fine-tuned for 1 epoch on this diverse instruction-tuning (DIT) data.

Key Contributions

  • MIA-Bench: 400 image-prompt pairs with compositional, layered instructions across 8 sub-instruction categories and 5 complexity levels, targeting strict instruction adherence.
  • Comprehensive evaluation of 29 MLLMs (closed- and open-source), revealing large performance gaps and ranking discrepancies relative to existing benchmarks.
  • Analysis showing that MLLM instruction-adherence capability does not simply correlate with underlying LLM backbone IFEval scores, nor with meta-rankings on existing multimodal benchmarks.
  • Synthetic DIT data construction pipeline (5,000 pairs) and SFT recipe that significantly improves instruction adherence.
  • Public release of benchmark data and evaluation code.

Results

  • Best overall: GPT-4o scores 88.58; GPT-4v 86.11; Claude-3-Opus 84.50.
  • Best open-source: LLaVA-NeXT-110b (79.84), MiniCPM-Llama3-V2.5 (76.27), Phi-3-Vision (76.02), InternVL-Chat-V1.5 (75.42).
  • Weakest models: Fuyu-8b (24.52), Kosmos-2 (26.06).
  • Ranking discrepancy: InternVL-Chat-V1.5 holds the highest meta-ranking across MME/MMMU/MMBench/MMVet/HallusionBench/MathVista among 5 tested MLLMs, yet ranks last (5th) on MIA-Bench; Claude-3-Opus (lowest meta-rank among the five) ranks 2nd on MIA-Bench.
  • SFT results (LLaVA-NeXT-13b): DIT fine-tuning raises MIA-Bench score from 69.16 → 78.85 (+9.69 points); minor regressions on MMBench (70.6 → 68.6), TextVQA (64.26 → 63.20), VQAv2 (82.80 → 82.58), POPE (87.7 → 86.9).
  • Judge stability: Rankings are consistent across gpt-4o-mini, gpt-4o-2024-05-13, gpt-4o-2024-11-20, and chatgpt-4o-latest as judge models.

Limitations

  • The benchmark covers only 400 samples; the diversity of real-world instructions is effectively unbounded.
  • Evaluation relies on LLM-as-judge (GPT-4o), introducing potential model-specific bias and ~1% score variance across runs (though cross-judge experiments partially mitigate this concern).
  • SFT experiments are limited to LLaVA-NeXT-13b as the backbone; broader applicability to other architectures is undemonstrated.
  • The five complexity tiers used during data collection are not reported per-category in results, limiting analysis of where models fail by complexity.
  • Alignment-based training alternatives (RLHF, DPO) are mentioned as future work but not explored.

Relevance to Vision-Language Models

MIA-Bench directly addresses a blind spot in MLLM evaluation: the ability to follow precise, multi-constraint instructions grounded in visual context—a core competency for real assistant deployments but largely invisible to VQA and open-ended chat benchmarks. The ranking reversals it surfaces (strong VQA models failing strict instruction adherence, and vice versa) suggest that instruction following is a relatively orthogonal capability axis that current training pipelines underemphasize. The DIT-based SFT recipe provides a practical, low-cost path for improving adherence in open-source models, and the benchmark's GPT-4o-as-judge protocol extends the evaluation paradigm from IFEval (text-only) into the multimodal regime.