Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations¶

🕒 Published (v1): 2026-06-23 15:39 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; first concept-annotation-grounded evaluation of SAE interpretability in VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Existing evaluation metrics for sparse autoencoders (SAEs) applied to vision and vision-language models fail basic sanity checks — they cannot reliably distinguish trained SAEs from untrained ones. This paper proposes a human-grounded evaluation framework using latent-concept matching (FBMP) and a new functional metric (TAPAScore) validated on two novel synthetic benchmarks (synCUB, synCOCO). Empirically, moderate dictionary sizes yield the best interpretability trade-off; excessive overcompleteness degrades perturbation alignment.

Problem¶

SAEs are increasingly used to extract interpretable features from CLIP, DINOv2, and other vision/vision-language encoders, but evaluation remains dominated by structural proxies (reconstruction error, sparsity) or qualitative inspection. Existing functional metrics (FMS, MS, CKNNA) fail sanity checks — they do not reliably distinguish trained SAEs from untrained (random-weight) baselines. Controlled intervention-style evaluation is also hard in vision because attribute changes in natural images rarely occur in isolation, leaving no benchmark that isolates single semantic changes in naturalistic imagery.

Method¶

The framework has three interlocking components:

Latent-Concept Matching. SAE activations $\mathbf{z} \in \mathbb{R}^L$ are binarized ($z^l_\text{bin} = \mathbf{1}[z^l > 0]$) and compared against binary human-annotated attribute vectors $\mathbf{y}_a \in \{0,1\}^N$. One-to-one matching selects $l_a^* = \arg\max_l F_1(\mathbf{y}_a, \mathbf{z}^l_\text{bin})$. To handle feature splitting, they introduce Fully-Binary Matching Pursuit (FBMP): a greedy coalition-building algorithm that operates entirely in the binary domain. At each step it selects the latent maximizing $F_\beta$ on the current residual, updates the residual with logical AND-NOT ($r \leftarrow r \wedge \neg z^{l*}_\text{bin}$), and aggregates via logical OR; it stops when $F_1$ no longer improves. This produces many-to-one mappings per attribute and many-to-many overall. An adjusted score $\Delta\text{MATCHScore} = \text{MATCHScore} - F_{1,\text{rng}}$ corrects for dictionary size bias by subtracting untrained-SAE baseline performance.

Synthetic Benchmarks. synCUB: paired bird images (from CUB-200-2011, 33 classes, 45 attributes) where Flux edits exactly one attribute (e.g., belly pattern solid→striped) while preserving identity, pose, and background; validated with an attribute predictor and manual curation. synCOCO: paired MS-COCO scenes where one object category is removed via Flux inpainting while all other objects are preserved.

TAPAScore. For each image pair $(x, \hat{x})$ differing in one attribute, signed responses are computed as: $$\delta_\text{add} = \max_{i \in I_\text{add}} \hat{z}^i_\text{bin} - \max_{i \in I_\text{add}} z^i_\text{bin}, \quad \delta_\text{rem} = \max_{i \in I_\text{rem}} \hat{z}^i_\text{bin} - \max_{i \in I_\text{rem}} z^i_\text{bin}$$ Averaged over $P$ pairs and combined: $\text{TAPAScore} = \Delta_\text{add} - \Delta_\text{rem}$. A positive score means matched latents fire more when the attribute is added and less when removed.

Experiments cover four SAE variants (JumpReLU, TopK, BatchTopK, Matryoshka) at dictionary sizes $\{128, 256, 512, 1024, 2048, 4096\}$ on CLIP (ViT-L-14) and DINOv2 (ViT-S-14) embeddings.

Key Contributions¶

FBMP: a novel fully-binary matching pursuit algorithm for many-to-one latent-concept alignment operating in the binary domain (logical OR aggregation, AND-NOT residual updates).
synCUB and synCOCO: first intervention-style benchmarks for SAE evaluation in naturalistic images, with single-attribute isolation via generative editing.
TAPAScore: a functional metric testing whether matched latents respond selectively and directionally correctly under targeted single-attribute perturbations.
Sanity check analysis: demonstration that FMS, MS, and CKNNA fail to distinguish trained SAEs from untrained/random baselines, while FBMP MATCHScore and TAPAScore pass.
Empirical finding: increased overcompleteness (larger dictionary) improves statistical matching but degrades perturbation alignment; moderate dictionary sizes offer the best interpretability trade-off.

Results¶

Sanity check: TAPAScore and FBMP $\Delta$MATCHScore are the only metrics that clearly drop under untrained TopK and random-activation conditions for both CUB/CLIP and COCO/CLIP; FMS, MS, and CKNNA show little sensitivity (CKNNA actually inflates for the untrained SAE).
FBMP vs. one-to-one: FBMP (k=3, $F_{0.5}$) consistently outperforms $F_1$-based one-to-one matching across all dictionary sizes and both backbones.
Overcompleteness trade-off: across CLIP and DINOv2, increasing dictionary size beyond moderate values improves $\Delta$MATCHScore but degrades TAPAScore, indicating reduced causal interpretability at high overcompleteness.
Recommended config: FBMP $F_{0.5}$ matching + TAPAScore; moderate dictionary sizes (exact optimum varies by backbone/dataset but in the hundreds rather than 2048–4096).
FBMP vs. magnitude-aware baseline: FBMP outperforms non-negative orthogonal matching pursuit in causal alignment at matched sparsity (Appendix S6.3).

Limitations¶

synCUB covers only 33 bird classes and 45 attributes (a subset of CUB's 312), limiting attribute diversity.
synCOCO uses object removal only (no attribute-level perturbations), so $\Delta_\text{add} = 0$ and TAPAScore reflects only removal alignment.
Binarizing latent activations discards magnitude information; polysemantic magnitude-encoded latents are penalized by design, which may be too strict for some SAE variants.
Synthetic benchmarks rely on Flux generative editing; residual artifacts or imperfect single-attribute isolation could introduce confounds despite validation.
Evaluation is limited to CLIP (ViT-L-14) and DINOv2 (ViT-S-14); generalization to other vision encoders or multimodal architectures (e.g., LLaVA, Flamingo internals) is untested.
FBMP stopping criterion can select small coalitions that may still miss distributed representations not captured by any compact subset.

Relevance to Vision-Language Models¶

CLIP is a central subject of evaluation here — the paper directly measures whether SAE features trained on CLIP embeddings align with human-interpretable visual attributes, which matters for anyone trying to understand or steer CLIP's internal representations. The TAPAScore framework and synCUB/synCOCO benchmarks fill a concrete methodological gap: trustworthy, human-grounded evaluation of feature interpretability in the vision encoders that underpin most VLMs. The finding that standard proxy metrics (FMS, MS, CKNNA) fail sanity checks should caution VLM interpretability researchers who rely on them. More broadly, as mechanistic interpretability of VLMs matures, this work provides reusable infrastructure (datasets, metrics, FBMP) for evaluating whether sparse decompositions of multimodal representations are genuinely concept-aligned.