Cross-Modal Redundancy and the Geometry of Vision–Language Embeddings¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Geometric probe of VL joint embedding space via cross-modal redundancy

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This ICLR 2026 paper introduces the Iso-Energy Assumption — that truly shared cross-modal concepts must exhibit equal average activation energy across modalities — and operationalizes it with an Aligned Sparse Autoencoder (SAE-A) trained on VLM embeddings. Applied to six foundational dual-encoder models (CLIP, SigLIP variants), the framework reveals that a sparse set of bimodal dictionary atoms carries the entire cross-modal alignment signal, while unimodal atoms are modality-specific biases that fully explain the modality gap. Removing unimodal atoms collapses the gap without degrading retrieval performance.

Problem¶

Vision–language models achieve strong cross-modal alignment empirically, yet the internal geometric organization of their shared embedding space is poorly understood. Classical sparse autoencoders (SAEs) applied to VLMs produce concept dictionaries that segregate by modality, failing to identify which features underpin genuine cross-modal alignment versus modality-specific structure. Prior explanations of the modality gap (conical structure, contrastive training dynamics) do not provide concept-level mechanistic accounts, and attempts to remove the gap by projecting out canonical basis directions damage cross-modal retrieval performance.

Method¶

The paper models multimodal data via a shared generative process: a sparse latent concept vector $c$ is rendered through domain-specific generators $g^{(d)}$, and a VLM encoder $f$ maps observations to a shared embedding space. A sparse autoencoder $\varphi$ inverts this mapping, but the recovery is ill-posed without additional inductive bias.

Iso-Energy Assumption (Definition 2): A learned encoder $\psi = \varphi \circ f$ satisfies Iso-Energy if each atom's second moment is domain-invariant: $$\mathbb{E}_{X \in \mathcal{X}^{(d)}}\!\left[\psi(X)_k^2\right] = \mathbb{E}_{X \in \mathcal{X}^{(d')}}\!\left[\psi(X)_k^2\right], \quad \forall k, \; (d, d') \in \mathcal{D}^2.$$

This is operationalized as a soft regularizer on top of a Matching Pursuit SAE (which enforces $\ell_0$ sparsity via sequential residual updates). For $\ell_2$-normalized codes $Z^{(d)}, Z^{(d')} \in \mathbb{R}^{b \times K}$ from a paired batch, the training loss is: $$\mathcal{L}_{\text{SAE-A}} = \mathcal{L}_{\text{SAE}} + \beta \cdot \mathcal{L}_{\text{align}}, \quad \mathcal{L}_{\text{align}} = -\tfrac{1}{b}\,\text{Tr}\!\left(Z^{(d)} Z^{(d')\top}\right)$$ with $\beta \approx 10^{-4}$. The penalty encourages same-atom activations to be aligned (high cosine similarity) across paired samples, biasing recovery toward bimodal features without forcing them when none exist.

SAE and SAE-A are trained on 1M LAION embeddings with identical hyperparameters (expansion ratio 8, target $\ell_0 = 20$) for six models: CLIP ViT-B/32 and ViT-L/14, OpenCLIP, OpenCLIP-L, SigLIP, SigLIP2. Geometry is analyzed via four multimodality-sensitive metrics: probing accuracy ($p_{\text{acc}}$), functional alignment ratio ($\rho$), Functional and Distributional Agreement (FDA), and interventional robustness ($\delta_r$, change in retrieval recall when unimodal atoms are ablated).

Key Contributions¶

Iso-Energy Assumption: a domain-agnostic, instance-free inductive bias for identifying genuinely shared cross-modal concepts.
Aligned SAE-A: a Matching Pursuit SAE augmented with an energy-consistency penalty; validated on synthetic data with known ground truth before application to real VLMs.
Sanity check result: when Iso-Energy holds ($\tau_1 = 1$), standard SAE fails ($W = 0.396$, $\text{mma} = 0.29$) while SAE-A recovers the dictionary ($W = 0.184$, $\text{mma} = 0.52$); when it does not hold ($\tau_1 \neq 1$), both perform equally (neutral behavior).
Geometric decomposition: bimodal atoms span a compact cross-modal subspace $\Gamma$; unimodal atoms span modality-specific subspaces $\Omega_I \oplus \Omega_T$ and fully account for the modality gap.
Actionable interventions: ablating unimodal atoms collapses the modality gap; restricting semantic vector arithmetic to the bimodal subspace produces in-distribution edits and improved retrieval.

Results¶

Reconstruction parity: SAE-A matches SAE in MSE and $R^2$ across all six VLMs ($R^2$ ranges 0.742–0.885 for both), confirming the penalty does not harm fidelity.
Functional alignment $\rho$: increases by more than an order of magnitude in SAE-A vs. SAE for all models (e.g., CLIP: $0.327 \to 4.232$; SigLIP: $1.370 \to 2.182$).
FDA: doubles or triples in SAE-A (e.g., CLIP: $2.630 \to 4.559$; SigLIP: $8.831 \to 34.95$).
Interventional robustness $\delta_r$: near zero for SAE-A across all models (e.g., CLIP: $0.224 \to 0.125$; OpenCLIP-L: $0.037 \to -0.000$), confirming unimodal atoms can be ablated without retrieval loss.
Probing accuracy $p_{\text{acc}}$: modest improvement in SAE-A (e.g., CLIP: $0.847 \to 0.915$), indicating cleaner modality separation in the dictionary.

Limitations¶

Analysis is restricted to dual-encoder (contrastive) VLMs; it is unclear whether findings generalize to generative or decoder-based multimodal architectures.
The Iso-Energy Assumption requires paired cross-modal samples at training time for the regularizer; unpaired or asymmetric multimodal data is not addressed.
The controlled sanity checks use synthetic embeddings that mimic CLIP-like cosine statistics but may not capture all real-world distributional complexity.
$\beta$ selection relies on a log-sweep with a heuristic variance criterion; behavior under very high or very low $\beta$ is reported in appendices but not fully characterized in the main text.
The paper is truncated; any further stated limitations in the conclusion are not visible.

Relevance to Vision-Language Models¶

This work provides a principled, concept-level explanation of the modality gap — a phenomenon central to understanding why CLIP-style models work and how their representations are organized — which has previously only been described geometrically or through training dynamics. The Aligned SAE framework is directly applicable to auditing, editing, and improving any dual-encoder VLM by exposing its cross-modal versus modality-specific feature decomposition. The finding that a sparse bimodal subspace carries all contrastive alignment signal has immediate implications for retrieval-augmented systems, multimodal arithmetic, and bias analysis. More broadly, it contributes to the emerging line of mechanistic interpretability work for VLMs, complementing concurrent SAE-based analyses and providing a new handle on the Linear Representation Hypothesis in multimodal settings.