Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders¶

🕒 Published (v1): 2026-06-29 07:27 UTC · Source: Arxiv · link

Why this paper was selected

First sparse autoencoder analysis of cross-modal feature heterogeneity in VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper identifies and characterizes cross-modal feature heterogeneity — the phenomenon where the same semantic concept is encoded along different feature directions for image vs. text modalities in VLM joint embedding spaces. It proves that this directional mismatch is the root cause of modality split in multimodal sparse autoencoders (SAEs), and that existing alignment objectives (group-sparse, Iso-Energy) inevitably trade reconstruction quality for latent alignment. The authors propose training modality-specific SAEs and aligning corresponding latents post-hoc via the Hungarian algorithm, which improves reconstruction fidelity and downstream task performance.

Problem¶

Prior work applying SAEs to VLM joint embeddings observed modality split: the same concept activates different latent coordinates across image and text modalities. Existing remedies (e.g., group-sparse losses, Iso-Energy alignment) treat this as a latent-activation mismatch and force shared activations, implicitly assuming that corresponding features share identical directions across modalities — an assumption that is neither tested nor theoretically justified. When this assumption fails, alignment degrades reconstruction by collapsing geometrically distinct directions into a single coordinate.

Method¶

Formal definition: Cross-modal feature heterogeneity is defined as \(\phi_i \neq \psi_i\) for concept \(i\), where \(\phi_i, \psi_i \in \mathbb{R}^d\) are the image and text feature-direction vectors respectively.
Empirical measurement: On MS-COCO embeddings from CLIP, MetaCLIP, OpenCLIP, and SigLIP, the authors train separate SAEs on image and text embeddings, then measure the cosine distance \(d_{\text{cos}}(\hat\phi_i, \hat\psi_j)\) between estimated feature-direction pairs. They use coactivation correlation \(c_{i,j} = [\text{Corr}(\tilde z(x), \tilde z(y))]_{i,j}\) as a proxy for semantic correspondence.
Theoretical analysis: Four formal results characterize the behavior of SAEs under heterogeneity:
Corollary 1: With enough latent capacity (\(m \geq 2n\)), the reconstruction-optimal SAE assigns separate decoder columns to \(\phi_i\) and \(\psi_i\), inducing modality split.
Theorem 1: With limited capacity (\(m < 2n\)), the optimal decoder columns are top eigenvectors of grouped outer-product matrices; close directions share a column, but semantic correspondence does not guarantee directional closeness.
Propositions 1 & 2: Group-sparse and Iso-Energy alignment losses introduce critical thresholds \(\lambda^\star(\rho)\) and \(\lambda^{\star\star}(\rho)\); below the threshold reconstruction is perfect but latents are unaligned, above it latents collapse to a shared direction but reconstruction degrades by \(\frac{1-\rho}{2}\mathbb{E}[z^2]\).
Proposed method — two-stage pipeline:
Stage 1: Train modality-specific SAEs (separate weight matrices \(V\) and \(W\)) on image and text embeddings, each minimizing its own reconstruction loss with no cross-modal constraint.
Stage 2: Post-hoc alignment via \(\hat P = \arg\max_{P \in \mathcal{P}_m} \text{tr}(C P)\) using the Hungarian algorithm on the coactivation correlation matrix \(C\), then reindex text latents by \(\hat P\).

Key Contributions¶

Characterization and formal definition of cross-modal feature heterogeneity in VLM joint embedding spaces.
Proof that modality split is a necessary consequence of heterogeneity under reconstruction-optimal SAEs.
Theoretical demonstration that existing alignment losses (group-sparse, Iso-Energy) have a fundamental reconstruction–alignment trade-off with sharp phase transitions at \(\lambda^\star(\rho)\).
A two-stage method (modality-specific SAEs + Hungarian-post-hoc alignment) that preserves each modality's feature geometry while identifying cross-modal correspondences.

Results¶

Empirical evidence across four VLMs (CLIP ViT-B/32, MetaCLIP B/32, OpenCLIP B/32, SigLIP Base) on MS-COCO shows that even high-correlation feature pairs (\(c \geq 0.8\)) have cosine distances centered well above zero (e.g., \(d_{\text{cos}} \approx 0.42\) for a baseball concept pair with \(c = 0.67\)).
Synthetic experiments with controlled cross-modal feature distance \(d_{\text{cos}}\) show that modality-specific SAEs avoid feature collapse, while shared SAEs incur growing reconstruction error and feature recovery error as \(d_{\text{cos}}\) increases.
The proposed method is claimed to improve reconstruction fidelity, cross-modal retrieval, and concept steering on real VLM embeddings (Section 7, details not in provided text).

Limitations¶

Post-hoc alignment relies on coactivation correlation as a proxy for semantic correspondence, which may miss features that rarely co-occur in paired data.
Hungarian algorithm complexity scales as \(\mathcal{O}(m^3)\) with latent dimension \(m\), potentially costly for large SAEs.
The method treats alignment as a hard permutation of indices rather than a soft correspondence, which may not capture hierarchical or compositional feature relationships.
The real-VLM results (Section 7) are referenced but their specifics are not provided in the excerpt.
All theoretical analysis assumes perfect sparsity (\(s \to 1\)) and Top-1 activation; real Top-K SAEs with dead neurons may deviate from the predicted behavior.

Relevance to Vision-Language Models¶

This work directly addresses a fundamental architectural assumption underlying all multimodal SAE-based interpretability research: that the joint embedding space is directionally coherent across modalities. By showing it is not, and by characterizing the precise trade-offs this creates, the paper reframes modality split from a "bug to be fixed" into a structural consequence of heterogeneous feature geometry. For anyone building or interpreting VLM representations, it provides both a diagnostic tool (measure \(d_{\text{cos}}\) between corresponding features) and a principled alternative (modality-specific SAEs + post-hoc alignment) that avoids the reconstruction–alignment trade-off of shared SAE approaches.