When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities¶

🕒 Published (v1): 2026-07-09 15:35 UTC · Source: Arxiv · link

Why this paper was selected

Sparse autoencoders for cross-modal mechanistic interpretability in VLMs — novel probe into internal representations

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Vanilla sparse autoencoders (SAEs) applied to vision-language models produce polysemantic visual features because they treat image patches as independent tokens, ignoring spatial and semantic coherence. This paper proposes S²AE, which partitions images into coherent regions via Transformer attention and spatial proximity, then regularizes SAE training with exclusive and group sparsity to enforce inter-region disentanglement and intra-region consistency. The visual structural prior improves monosemanticity in both the visual and language sides of shared multimodal SAE features.

Problem¶

Vanilla SAEs trained on VLMs optimize only reconstruction fidelity and element-wise sparsity, producing conceptual features with residual polysemanticity: a single SAE feature activates across semantically inconsistent visual regions (e.g., a "park bench" feature also fires on grass patches). Vision-side monosemanticity scores (~0.492 at layer 15) are significantly lower than language-side scores (~0.869), indicating that visual token discretization without spatial priors is a fundamental bottleneck. This fragmented visual coverage also contaminates shared multimodal features, degrading cross-modal semantic consistency.

Method¶

Patch grouping. For each image, a symmetric adjacency matrix is constructed combining (1) average multi-layer Transformer attention similarity and (2) Manhattan spatial distance between patch coordinates: $$d_{i} = d_{i}^{A} \odot (d_{i}^{S})^{\alpha}$$ Agglomerative clustering over $d$ partitions image patches into $G$ coherent visual regions.

Group-level activation matrix. For each region $g$, patch-wise SAE activations are $\ell_2$-pooled across patches to form a group-level profile vector $s^g \in \mathbb{R}^N$. These are stacked into $S \in \mathbb{R}^{G \times N}$.

Structured sparsity regularization. Two complementary penalties are added to the Top-K SAE loss: - Exclusive sparsity ($\ell_{1,2}$ on $S$): enforces inter-group competition so each SAE feature specializes in one region. $$\mathcal{L}_{\text{es}} = \frac{1}{N}\sum_{j=1}^{N}\left(\sum_{g=1}^{G}|s_j^g|\right)^2$$ - Group sparsity ($\ell_1$ per group): enforces intra-group coherence so patches within a region share a sparse SAE feature subset. $$\mathcal{L}_{\text{gs}} = \frac{1}{G}\sum_{g=1}^{G}\|s^g\|_1$$

Total loss: $\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{sae}} + \lambda_{\text{es}}\mathcal{L}_{\text{es}} + \lambda_{\text{gs}}\mathcal{L}_{\text{gs}}$

A binarization layer with a Straight-Through Estimator decouples activation selection from magnitude, so regularization targets feature co-occurrence rather than activation intensities.

Evaluation pipeline. A hierarchical semantic synthesis pipeline converts masked image patches to text via Qwen3-VL-8B, feeds them alongside masked text contexts to Qwen3-30B to generate per-modality concept explanations, and scores cross-modal consistency and monosemanticity automatically.

Key Contributions¶

S²AE: structured SAE with attention+spatial patch grouping and dual sparsity regularization applied to Qwen2.5-VL-7B-Instruct.
Exclusive sparsity for inter-group concept disentanglement; group sparsity for intra-group semantic coherence — both operating on binarized group-level activation profiles.
Demonstration that visual structural priors propagate to language-side monosemanticity in a shared multimodal SAE dictionary.
Automated hierarchical cross-modal interpretability pipeline (VLM → text descriptions → LLM → concept explanations + consistency scores).

Results¶

mIoU (semantic alignment): 6.06% average improvement; layer 15 specifically improves from 0.516 (vanilla) to 0.594.
$\ell_0$ norm (representational efficiency): 60.81 lower than vanilla SAE.
Reconstruction fidelity: Explained Variance remains above 99%.
Cross-modal semantic consistency: +3.08% average gain.
Monosemanticity scores (both modalities): +2.37% average gain.
Layer 5 consistent-concept proportion: +4.3 percentage points over vanilla SAE.
Baseline: vanilla Top-K SAE trained on the same Qwen2.5-VL-7B-Instruct model.

Limitations¶

Structural regularization is evaluated on a single VLM (Qwen2.5-VL-7B-Instruct); generalization to other VLM architectures (e.g., cross-attention-based models) is not demonstrated.
Patch grouping quality depends on attention quality at the targeted layer; using average multi-layer attention is a pragmatic fix but introduces a hyperparameter ($\alpha$, number of layers $L$, number of groups $G$).
The automated monosemanticity pipeline relies on Qwen3-VL and Qwen3-30B as judges; existing VLMs are noted to yield low-quality concept descriptions, motivating the hierarchical workaround but not fully resolving reliability concerns.
Regularization is imposed only on image patches; text-side structural priors are not explored.
Scalability to very high-resolution inputs with many patches may increase the cost of agglomerative clustering.

Relevance to Vision-Language Models¶

S²AE directly addresses a mechanistic interpretability gap in VLMs: standard SAE training ignores the spatial structure of visual tokens, producing polysemantic features that obscure how VLMs internally represent visual concepts. The finding that visual structural regularization improves language-side monosemanticity in a shared feature space has implications for understanding cross-modal alignment and feature entanglement in architectures like Qwen-VL and InternVL. For researchers tracking VLMs, this work provides a principled decomposition tool that could be applied to audit, diagnose, or steer multimodal representations — complementing activation patching and causal tracing approaches. It also establishes a reusable automated interpretability pipeline for scoring concept coherence across modalities.