Skip to content

Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Multimodal Item Response Theory enables rigorous VLM reasoning benchmark design

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

M3IRT (Multidimensional Multimodal Item Response Theory) extends classical IRT by decomposing both VLM ability and benchmark item difficulty into image-only, text-only, and cross-modal latent components. This enables principled identification of shortcut questions (solvable with a single modality) and construction of compact, high-quality benchmark subsets. Across 24 VLMs on three benchmarks, M3IRT reconstructs full model rankings using only 10% of items while halving the inclusion rate of low-quality questions compared to baselines.

Problem

Current VLM benchmarks (MMMU, MathVista, SEED-Bench) are contaminated with shortcut questions answerable from a single modality alone—image or text—yielding unreliable model rankings and inflating evaluation cost. Classical IRT and MIRT are modality-agnostic: they represent ability and difficulty as a single (or unstructured) scalar/vector, so they cannot distinguish whether a model's correct response reflects true cross-modal integration or single-modality shortcuts.

Method

Modality decomposition. Each VLM is evaluated under all four input formats \(s = (s^{\text{image}}, s^{\text{text}}) \in \{0,1\}^2\) (both withheld, image-only, text-only, both present). IRT parameters are additively decomposed:

\[\theta_i(s) = \theta_i^{\text{base}} + s^{\text{image}}\theta_i^{\text{image}} + s^{\text{text}}\theta_i^{\text{text}} + s^{\text{image}}s^{\text{text}}\theta_i^{\text{cross}}\]
\[b_j(s) = b_j^{\text{base}} - s^{\text{image}}b_j^{\text{image}} - s^{\text{text}}b_j^{\text{text}} - s^{\text{image}}s^{\text{text}}b_j^{\text{cross}}\]

M2IRT applies this decomposition to the scalar 2PL IRT logistic model. M3IRT extends MIRT: ability and discrimination/difficulty are 4-vectors \(\theta_i, a_j, b_j \in \mathbb{R}^4\), with format indicator vector \(s = [1, -s^{\text{image}}, -s^{\text{text}}, -s^{\text{image}}s^{\text{text}}]^\top\). The response probability is:

\[\hat{P}(r_{i,j,s}=1) = \sigma\!\left(a_j^\top \operatorname{diag}(s)\,\theta_i - s^\top b_j\right)\]

Parameters are estimated by minibatch SGD (Adam, lr=0.01) on binary cross-entropy loss over the response tensor \(R' \in \{0,1\}^{M \times N \times |S|}\); partial observability is supported (tensor completion). Cross-modal difficulty \(b_j^{\text{cross}}\) ranks questions by degree of required multimodal integration. Compact subsets are selected via Computerized Adaptive Testing (CAT) using D-optimality on the cumulative Fisher information matrix.

Key Contributions

  • M3IRT/M2IRT frameworks that factorize both model ability and item difficulty into image-only, text-only, and cross-modal latent components, extending IRT and MIRT respectively.
  • A practical benchmark refinement pipeline: cross-modal difficulty scores identify genuine cross-modal items; CAT-based subset selection constructs compact, high-quality evaluation sets.
  • Demonstration that M3IRT is robust to up to 50% shortcut/unsolvable question contamination (semi-synthetic benchmarks via image–text swapping).
  • Interpretable ability profiles for 24 VLMs, revealing that even highly-ranked models can rely primarily on text understanding (\(\theta^{\text{text}}\)) with limited cross-modal integration (\(\theta^{\text{cross}}\)).

Results

  • 10% subset suffices: M3IRT/M2IRT reconstruct original model rankings (Spearman's \(\rho\)) on all three benchmarks—MMMU, MathVista, SEED-Bench—using only 10% of items, outperforming IRT, MIRT, TinyBenchmarks, FlashEval, and Random selection.
  • Low-quality question rate halved: The proportion \(\gamma\) of low-quality questions in extracted subsets under M3IRT/M2IRT is less than half that of existing methods across all benchmarks and subset sizes.
  • Robustness at 50% contamination: Even when half of all benchmark items are artificially replaced with shortcuts or unsolvable questions, M3IRT preserves reliable ranking fidelity where baselines degrade.
  • Ability decomposition insight (MMMU): The top-performing model shows high \(\theta^{\text{cross}}\); the 2nd and 3rd best show high \(\theta^{\text{text}}\) but limited \(\theta^{\text{cross}}\), revealing text-reliance.
  • MathVista pattern: Most VLMs exhibit high \(\theta^{\text{text}}\) and moderate \(\theta^{\text{image}}\), consistent with the benchmark's text-heavy mathematical phrasing.
  • Evaluated on 24 VLMs including GPT-4.1, Gemini-2.0-flash, Claude-3.7-sonnet, Qwen-2.5-VL, Llama-3.2, and Pixtral series.

Limitations

  • Semi-synthetic contamination is generated only by swapping images or texts across questions; more realistic corruption (LLM-modified options, adversarial image noise) is excluded due to experimental complexity.
  • M2IRT requires each model to be queried under all four modality ablation formats, multiplying inference cost 4Ă— during the calibration phase.
  • Cross-benchmark generalization of estimated parameters is not validated; item parameters are benchmark-specific and must be re-estimated for new benchmarks.
  • Only vision-language is experimentally evaluated; extension to audio, action, or higher-modality settings is described but untested.

Relevance to Vision-Language Models

This work directly targets a core weakness in VLM evaluation methodology: the widespread presence of unimodal shortcuts that make benchmarks uninformative about cross-modal reasoning. M3IRT provides a principled, data-driven tool for auditing and compressing existing VLM benchmarks without requiring human re-annotation. The decomposed ability profiles (\(\theta^{\text{cross}}\) vs \(\theta^{\text{text}}\)) offer a more diagnostically useful characterization of VLM strengths than aggregate accuracy, revealing that strong benchmark performance does not always imply strong cross-modal integration. For researchers designing or selecting benchmarks to probe multimodal reasoning specifically, M3IRT offers a systematic filter to surface genuinely cross-modal questions and reduce spurious ranking differences driven by text-only shortcuts.