Test-Time Matching: Unlocking Compositional Reasoning in Multimodal Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; test-time matching unlocks compositional reasoning in frontier VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Compositional reasoning benchmarks have been systematically underestimating VLM capability because the standard GroupScore metric imposes unnecessarily strict per-pair constraints. This paper introduces GroupMatch, a fairer metric based on optimal bijective matching, and Test-Time Matching (TTM), an iterative self-training algorithm that exploits matching-induced pseudo-labels to further boost performance at test time without any external supervision. Together, these enable SigLIP-B16 to surpass GPT-4.1 on MMVP-VLM and GPT-4.1 to exceed estimated human performance on Winoground for the first time.

Problem

Standard compositional reasoning benchmarks (Winoground, MMVP-VLM, ColorSwap) use GroupScore, which requires every diagonal entry \(s_{ii}\) to be simultaneously the largest in its row and column — a condition that fails with probability \(1 - (k{-}1)!/(2k{-}1)!\) even for a perfectly correct model under random tie-breaking. This makes GroupScore pathologically strict: for \(k=2\) it reduces random-chance accuracy from \(1/2\) (the correct baseline) to \(1/6\), causing frontier models to appear at or below random guessing and masking their true compositional reasoning ability.

Method

GroupMatch. For a \(k \times k\) group with similarity matrix \(s \in \mathbb{R}^{k \times k}\), GroupMatch equals 1 iff the ground-truth bijection \(\pi^\star\) maximizes total similarity: \(\sum_i s_{i,\pi^\star(i)} > \sum_i s_{i,\pi(i)}\) for all \(\pi \neq \pi^\star\). Under random scores, \(P(\text{GroupMatch}=1) = 1/k!\) vs.\ \(P(\text{GroupScore}=1) = (k{-}1)!/(2k{-}1)!\), so GroupMatch is strictly more lenient and corresponds to the natural decision-theoretic criterion.

SimpleMatch. GroupMatch-correct predictions are translated to GroupScore-correct ones by overfitting the model's similarity scores to the induced matching \(\pi^\star\) at test time — a zero-supervision step that requires no new data.

Test-Time Matching (TTM, Algorithm 1). TTM iterates for \(T\) rounds. At round \(t\), the current model \(f_{t-1}\) computes a matching \(\pi_{f_{t-1}}(G) = \arg\max_\pi \sum_u s_{u,\pi(u)}\) for each group \(G\). A pseudo-label \((G, \pi_{f_{t-1}}(G))\) is retained only if the matching margin \(\Delta(G; f_{t-1}) = s(\pi_{f_{t-1}}(G)) - \max_{\pi \neq \pi_{f_{t-1}}(G)} s(\pi)\) exceeds a threshold \(\tau_t\). The model is then fine-tuned on the retained set \(S_t\). A decaying threshold schedule (\(\tau_{t+1} < \tau_t\)) starts high (selecting ~15–30% of groups with high precision) and decays to near-zero (covering >90% of the test set), balancing false positives early and false negatives late.

Global variant. For datasets without group structure, the entire dataset is treated as one large assignment problem solved via the Hungarian algorithm. Per-pair thresholding replaces group-level margin thresholding.

Key Contributions

• GroupMatch metric: proves GroupScore systematically undercounts correctness; provides a metric whose random-chance baseline is \(1/k!\) (the natural expectation for \(k\)-way matching).
• SimpleMatch: a zero-cost overfitting step converting GroupMatch accuracy to GroupScore accuracy, revealing large hidden capability without additional training data.
• TTM algorithm: iterative self-training with matching-induced pseudo-labels and a decaying confidence threshold; requires no external supervision and achieves new state-of-the-art on multiple benchmarks.
• Global TTM: extends TTM to unstructured datasets by formulating a single Hungarian-algorithm assignment across all images and captions.
• Comprehensive evaluation across 16 dataset variants spanning \(2 \times 2\), \(1 \times k\), and non-grouped settings.

Results

• Winoground (2×2): SigLIP-B16: Raw 10.25 → SimpleMatch 67.0 → TTM 72.5; GPT-4.1: Raw 69.75 → SimpleMatch 91.38 (first result surpassing estimated human performance of 85.5).
• MMVP-VLM (2×2): SigLIP-B16: Raw 22.96 → SimpleMatch 81.48 → TTM 89.44 (surpasses GPT-4.1's SimpleMatch of 88.52, new SOTA); prior SOTA was 70.7 (GPT-4o multi-agent).
• ColorSwap (2×2): SigLIP-B16: Raw 30.33 → SimpleMatch 88.0 → TTM 94.25; SigLIP-L16 TTM 96.08 (vs. prior SOTA 87.33/95.33 w/ finetuning); GPT-4.1 SimpleMatch 97.42.
• WhatsUp (1×k, no grouped structure): TTM yields up to 85.7% relative gain over raw baseline on challenging subsets.
• SugarCrepe (1×2): TTM delivers consistent gains; GroupScore and GroupMatch coincide here, so gains are purely from self-training.
• TTM error reductions on MMVP-VLM: 43.0% for SigLIP-B16, 27.7% for CLIP-B16.
• Improvements are consistent across all 16 dataset variants tested.

Limitations

• TTM requires test-time fine-tuning, adding computational cost \(O(T \cdot C_{f_t})\); not applicable in frozen-model deployment scenarios.
• GPT-4.1 requires log-probability outputs for accurate VQAScore computation; newer models (e.g., GPT-5 at submission time) did not support this, limiting applicability to some MLLMs.
• SimpleMatch/TTM benefit from group structure or global matching; performance gains on truly unstructured, single-instance queries are not addressed.
• The decaying threshold schedule introduces hyperparameters (\(\tau_1\), \(\tau_T\), \(T\)) that require tuning, albeit with practical rules-of-thumb (15–30% initial coverage, >90% final coverage).
• Evaluation focuses on discriminative image-caption matching; generalization to open-ended generative compositional reasoning tasks is not demonstrated.

Relevance to Vision-Language Models

This paper directly challenges the accepted narrative that VLMs fundamentally fail at compositional reasoning, showing instead that the failure was largely an evaluation artifact — a finding with broad implications for how the community benchmarks and interprets VLM capability. The GroupMatch metric and TTM framework are model-agnostic, applying to both contrastive VLMs (CLIP, SigLIP) and MLLMs (GPT-4.1), making them immediately relevant to any researcher evaluating or training VLMs on structured perception tasks. TTM's self-improving, supervision-free paradigm is closely related to test-time training and RLVR lines of work, but operates via combinatorial matching rather than reward signals, offering a complementary direction for test-time adaptation of VLMs. The results reopen questions about whether existing VLM architectures are already capable of fine-grained compositional understanding when evaluated appropriately.