Test-Time Training for Modality Order Consistency in Vision-Language Models¶

🕒 Published (v1): 2026-07-22 16:37 UTC · Source: Arxiv · link

Why this paper was selected

Gandelsman (Berkeley); TTT fixes systematic VLM modality-order bias; reproducible across 3 models

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Vision-language models (VLMs) exhibit a repeatable accuracy gap of 6–26 percentage points depending solely on whether the image or question appears first in the prompt. The authors design an asymmetric test-time training (TTT) procedure that uses the stronger image-first branch as a moving teacher to adapt the weaker question-first branch via KL divergence minimization. The method closes nearly all of the modality-order gap and slightly improves image-first accuracy as a side effect.

Problem¶

Current VLMs are sensitive to a semantically irrelevant prompt perturbation—swapping image and question order—producing structured, repeatable prediction differences of 6–26 pp across model families and benchmarks. This reveals an internal computational asymmetry whose causal origin and circuit-level location were previously uncharacterized, and no principled repair existed.

Method¶

For each test instance \(s = (x, q, A)\), two prompts are constructed: image-first (\(o_\text{IF}\)) and question-first (\(o_\text{QF}\)). The model's image-first distribution is treated as a moving teacher:

\[\bar{p}_\text{IF}^{(k)}(s) = \text{stopgrad}\!\left(p_\text{IF}(\theta^{(k)}; s)\right)\]

and the question-first distribution is adapted via asymmetric KL divergence:

\[\mathcal{L}_\text{align}(\theta^{(k)}; s) = D_\text{KL}\!\left(\bar{p}_\text{IF}^{(k)}(s) \,\|\, p_\text{QF}(\theta^{(k)}; s)\right)\]

Parameters are updated with SGD for \(K\) steps (3–5 depending on model), then reset to \(\theta_0\) before the next example—making adaptation label-free, instance-specific, and stateless across examples. Only language-model backbone parameters in a selected mid-layer window are updated; vision encoder parameters are frozen. Teacher is recomputed after each gradient step (moving teacher), which the authors show is critical. The layer window is chosen based on activation patching: replacing question-first final-token hidden states with image-first states at layers ~18–20 restores accuracy sharply, while patching earlier or later layers has little effect; TTT targets this same mid-network band.

Key Contributions¶

Identifies modality-order sensitivity as a structured, repeatable, semantically irrelevant failure mode in VLMs, with image-first outperforming question-first by 6–26 pp across three models and three benchmarks.
Uses activation patching to causally localize the failure to a narrow mid-layer band (around layers 18–20 in InternVL2-8B), showing it is not diffuse across depth.
Proposes asymmetric order-consistency TTT: an instance-level, label-free adaptation using a directional KL objective with a moving image-first teacher.
Shows the moving-teacher variant is essential: frozen-teacher achieves only +8.5 pp vs. +26.0 pp on InternVL2-8B/MMStar.
Demonstrates via NIoU analysis that TTT repairs MLP-sublayer alignment in late layers (strongly below-random overlap → strongly positive), while attention-head alignment changes little.

Results¶

InternVL2-8B / MMStar: QF baseline 35.5% → QF TTT 61.5% (+26.0 pp); IF baseline 61.6% → IF TTT 62.3% (+0.7 pp).
InternVL2-8B / RealWorldQA: QF 38.2% → 60.0% (+21.8 pp); IF 61.8% → 62.1% (+0.3 pp). Adapted QF exceeds IF baseline.
Qwen2.5-VL-7B / MMStar: QF 45.3% → 58.7% (+13.4 pp); IF 62.1% → 62.7% (+0.5 pp).
Qwen3-VL-8B / MMStar: QF 44.9% → 64.4% (+19.5 pp); IF 63.9% → 65.1% (+1.1 pp).
QF accuracy improves across all 9 model–dataset pairs; IF accuracy improves in 7/9 and is unchanged in 2/9—no degradation observed.
Loss ablation (InternVL2-8B/MMStar): Asymmetric KL and asymmetric CE both achieve +26.0/+25.9 pp QF gain; symmetric CE recovers only +9.8 pp and slightly hurts IF (−0.3 pp); hard pseudo-label yields +25.4 pp QF.
Layer-window ablation: All windows above layer 5 recover >25 pp of QF; layers 16–20 give peak QF (+26.2 pp); layers 18–22 give best combined IF/QF and are used as default.

Limitations¶

Evaluated only on multiple-choice VQA benchmarks (MMStar, RealWorldQA, AI2D); generalization to open-ended generation tasks is unverified.
TTT requires \(K\) forward+backward passes per test instance, increasing inference cost relative to single-pass evaluation.
The paper focuses on one semantically irrelevant perturbation; whether the circuit-level localization and TTT repair generalize to other prompt-format sensitivities is not established.
Mechanistic analysis (activation patching, NIoU) is conducted primarily on InternVL2-8B; Qwen3-VL results are in the appendix, and the finding that mid-layer localization generalizes is shown qualitatively but not quantified across all models.
The method assumes image-first is reliably the stronger ordering, which holds empirically but may not hold universally across tasks or future model architectures.

Relevance to Vision-Language Models¶

This paper directly targets a fundamental robustness failure in how VLMs integrate visual and linguistic tokens: the model's answer changes substantially based on prompt token order despite identical semantic content, revealing that cross-modal fusion is order-dependent at a circuit level. The activation-patching localization result—showing the failure concentrates in a narrow mid-network band—advances mechanistic understanding of VLM internals beyond coarse benchmark evaluation. For researchers tracking VLMs, the TTT approach offers a practical, label-free inference-time fix that is architecture-agnostic (tested across InternVL2 and Qwen families) and incurs no labeled data cost. The finding that symmetric consistency objectives substantially underperform asymmetric ones has implications for how consistency-based self-supervision should be designed for multimodal models.