VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

VL-SAE is a sparse autoencoder that jointly encodes vision and language representations into a shared hidden layer where each neuron corresponds to a single semantically unified concept. It resolves the "concept mismatch" problem that arises when separate SAEs are applied per modality, enabling both interpretable alignment analysis and downstream performance gains. Applied to CVLMs and LVLMs, it improves zero-shot classification and reduces object hallucinations.

Problem¶

Existing interpretability methods for VLMs either analyze vision or language representations in isolation, using separate concept sets. When two separate SAEs are trained on different modalities, neurons at the same index end up correlated with different concepts (concept mismatch), making cross-modal comparison impossible. No prior method maps both modalities' representations into a unified concept set in a scalable, annotation-free way.

Method¶

VL-SAE is a sparse autoencoder with three key design choices targeting cross-modal concept alignment:

Explicit alignment for LVLMs: An auxiliary autoencoder (encoders Eᵥ, E_l; decoders Dᵥ, D_l) is trained with InfoNCE contrastive loss + reconstruction loss to convert implicitly aligned LVLM representations into an intermediate space where semantic similarity correlates with cosine similarity. CVLMs skip this step since their representations are already cosine-aligned.
Distance-based encoder: Instead of inner product or raw cosine similarity (which violates the triangle inequality), neuron activation is computed as E^s(xᵉ)[i] = 2 − √(2 − 2·cos(xᵉ, wᵢ)) — the complement of the Euclidean distance between ℓ2-normalized inputs and weight vectors. Because this metric satisfies the triangle inequality, if two representations are close in cosine space, their activation differences are bounded: |g(xᵥᵉ, wᵢ) − g(x_lᵉ, wᵢ)| ≤ g(xᵥᵉ, x_lᵉ). Top-K sparsification then selects the k most-activated neurons.
Modality-specific decoders: Two separate decoders (Dᵥˢ, D_lˢ) reconstruct each modality from shared activations. This prevents the encoder from absorbing distributional differences into neuron activations, which would otherwise reintroduce concept mismatch.

Training uses only reconstruction loss (no concept annotations) on CC3M (3M image-text pairs). At inference, concept-level similarity augments the original cosine score for CVLMs (y = cos(xᵥ, x_l) + αc·cos(hᵥ, h_l)), and for LVLMs a refined language representation is constructed by blending vision concept activations into the language concept activation before decoding.

Key Contributions¶

VL-SAE architecture (distance-based encoder + dual modality-specific decoders) that maps both vision and language representations to a single unified concept set without annotation.
Auxiliary autoencoder that converts implicit LVLM alignment into explicit cosine-correlated alignment, enabling the same pipeline for both CVLMs and LVLMs.
Interpretability application: visualization of per-modality concept activations and their intersection during inference, surfacing which concepts align and which are modality-specific.
Enhancement of CVLMs via concept-augmented similarity scores; consistent zero-shot classification gains across 13 datasets and 4 OpenCLIP model sizes.
Hallucination elimination in LVLMs by blending vision concept activations into language representations before contrastive decoding; outperforms VCD on POPE.

Results¶

Concept quality (Intra-Similarity / Inter-Similarity on OpenCLIP-ViT-H/14):
Standard SAE: 0.1890 intra / 0.1688 inter
VL-SAE: 0.2134 intra / 0.1149 inter (higher intra = better semantic consistency within a concept; lower inter = better diversity across concepts)
Zero-shot image classification (OpenCLIP, mean accuracy across 13 datasets):
ViT-B/32: 68.7 → 69.5 (+0.8)
ViT-B/16: 69.8 → 70.4 (+0.6)
ViT-L/14: 72.2 → 72.9 (+0.7)
ViT-H/14: 76.9 → 77.8 (+0.9)
Hallucination elimination (POPE, LLaVA1.5, adversarial split):
Regular decoding: F1 77.16
VCD: F1 80.13
VL-SAE: F1 82.29 (+2.16 over VCD)
Hallucination elimination (POPE, Qwen-VL, adversarial split):
VL-SAE F1: 84.23 vs. VCD 82.39

Limitations¶

The auxiliary autoencoder for LVLMs adds non-trivial training overhead (50 epochs on CC3M before VL-SAE training); not needed for CVLMs but increases pipeline complexity for LVLMs.
Concept quality and downstream gains improve monotonically with dataset volume (CC3M 10%→100%: Intra-Similarity 0.2029→0.2299), suggesting scaling beyond CC3M may be needed for richer concept coverage.
The enhancement strategies are modality/task-specific (separate formulations for CVLMs vs. LVLMs), limiting off-the-shelf generalizability.
The αc and αl/β hyperparameters for enhancement are task-specific, requiring tuning per downstream application.
Evaluated only on OpenCLIP variants and LLaVA1.5/Qwen-VL; broader coverage of VLM families (e.g., BLIP-2, InternVL) is left to future work.

Relevance to Vision-Language Models¶

VL-SAE directly addresses a fundamental opacity in VLMs: why two representations are considered aligned is currently a black box. By providing a shared, human-readable concept space for both modalities, it opens a mechanistic interpretability pathway analogous to SAE-based analysis in LLMs, now extended to cross-modal settings. The finding that single-modal representations activate semantically irrelevant concepts (e.g., a motorcycle image activating car concepts) that are then filtered by cross-modal co-activation sheds light on the geometry of contrastive alignment. For VLM researchers, the concept-level enhancement pipeline is immediately applicable to both CLIP-style retrieval models and instruction-tuned LVLMs, and the hallucination analysis via concept divergence provides a diagnostic tool for understanding modality misalignment as a root cause of generation errors.