MIRROR: Aligning Semantic Relations from Language to Image via Gromov--Wasserstein¶

🕒 Published (v1): 2026-06-28 15:39 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

Fundamental structural alignment gap in MLLMs; novel Gromov-Wasserstein approach

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

MIRROR introduces a geometric regularization loss (Semi-Inverse Gromov–Wasserstein) that transfers relational structure from language representations to visual representations in MLLMs. It exploits self-attention as intra-modal distance and cross-attention as the cross-modal coupling, and solves for the optimal visual geometry in closed form. No additional parameters or inference cost.

Problem¶

Projection-based adapters in MLLMs (e.g., LLaVA, Qwen-VL) align visual tokens at the identity level via next-token prediction, but never enforce that pairwise relational structure between concepts survives the cross-modal transfer. An LLM that can textually reason "Huskies differ from Malamutes in ear shape" may still confuse them visually; a model that predicts "leafy" from text priors alone overrides this with salient-but-misleading visual features. This "structural blind spot" is the gap MIRROR targets.

Method¶

Intra-modal geometry: Symmetrized self-attention matrices $A_t$ (text) and $A_v$ (vision) are converted to distance matrices via $D = -\log(A + \varepsilon)$. High mutual attention → small distance.
Cross-modal coupling: The text-to-vision cross-attention $C \in \mathbb{R}^{n_t \times n_v}$ at an intermediate LLM layer provides a data-dependent soft assignment between text and visual tokens.
Semi-Inverse GW problem: Holding $D^t$ (language geometry, the "teacher") and $C$ (coupling) fixed, solve for the visual distance matrix $\widehat{D}^v$ that minimizes the GW distortion: $$\widehat{D}^v = \argmin_{D^v} \sum_{i,j,k,\ell} |d_t(i,j) - d_v(k,\ell)|^2 C_{ik} C_{j\ell}.$$ Theorem 1: This has a unique closed-form solution $\widehat{D}^v = (C^\top D^t C) \oslash (b b^\top)$ where $b = C^\top \mathbf{1}_{n_t}$. Intuitively, each entry $\widehat{D}^v_{k\ell}$ is the expected text-space distance between the text tokens that attend to visual tokens $k$ and $\ell$.
SI-GW loss: $\mathcal{L}_{\text{SI-GW}} = \| D^v - \widehat{D}^v \|_F^2$, added to the standard LM loss with weight $\lambda = 0.002$. Computed in $O(n_t^2 n_v + n_v^2 n_t)$ vs. the naive $O(n_t^2 n_v^2)$.
Stable extraction strategies:
Layer decoupling: $A_t$ from LLM Layer 1 (pure text geometry), $A_v$ from final ViT layer, $C$ from LLM Layer 16 (balance of semantics and concentration).
Selective head aggregation: retain only the $k=8$ lowest-entropy cross-attention heads; near-uniform heads are discarded before averaging.
Non-semantic token filtering: soft, position-aware suppression of system prompts, <image> placeholders, and formatting tokens in $A_t$.

Key Contributions¶

Formalizes cross-modal relational alignment as a semi-inverse Gromov–Wasserstein problem, proving a unique closed-form solution.
Derives an efficient SI-GW loss ($O(n_t^2 n_v + n_v^2 n_t)$) suitable for long token sequences in decoder-only Transformers.
Introduces three targeted extraction strategies (layer, head, token level) that make SI-GW stable at scale without additional parameters.
Demonstrates consistent gains across four model configurations (LLaVA-1.5 and LLaVA-NeXT at 7B/13B) on both relational and general VLM tasks.

Results¶

GQA (overall): LLaVA-1.5-7B +0.77%, LLaVA-1.5-13B +0.34%, LLaVA-NeXT-7B +0.82%, LLaVA-NeXT-13B +0.62%.
GQA fine-grained categories: Largest gains in Global tasks (+1.04% to +3.16%), followed by Category (+0.63% to +2.44%) and Relation (+0.13% to +1.44%). The Global and Relation categories directly reflect improved relational consistency.
No inference overhead: MIRROR adds no parameters and no cost at test time; the regularization is active only during training.
Training on 8 NVIDIA RTX 6000 Ada GPUs for 1 epoch, following the original training protocol.

Limitations¶

Only evaluated on the LLaVA family (1.5 and NeXT at 7B/13B); generalizability to other architectures (e.g., Qwen-VL, InternVL) is not established.
Performance depends on hyperparameters ($\lambda$, layer selections, $k$ heads, $\alpha$) that may need retuning for different model scales or training budgets.
The quality of $\widehat{D}^v$ is bounded by the quality of the cross-attention coupling $C$ — if coupling is diffuse or noisy, the target geometry may be a poor supervisory signal.
Adds training-time computation (though reduced from quartic to sub-quartic via the closed form).
The relational priors transferred from language may themselves encode biases or spurious correlations present in the text training data.

Relevance to Vision-Language Models¶

MIRROR directly addresses a known failure mode of current MLLMs: the gap between language-level relational reasoning and visual-level application. By treating relational alignment as a geometric regularizer rather than adding new modules or data, it provides a lightweight, architecture-agnostic fix that improves both relational and general VLM benchmarks. This connects neatly to the Platonic Representation Hypothesis (representations converge across modalities) and opens a new axis for VLM training — aligning structure, not just content.