Skip to content

From Weights to Concepts: Data-Free Interpretability of CLIP via Singular Vector Decomposition

🕒 Published (v1): 2026-03-25 17:59 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; data-free CLIP interpretability via SVD; no activation data required

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SITH (Semantic Inspection of Transformer Heads) is a data-free, training-free interpretability framework that explains CLIP's attention heads by decomposing their value-output (VO) weight matrices via SVD and mapping each singular vector to human-interpretable concepts. The core algorithm, COMP (Coherent Orthogonal Matching Pursuit), finds sparse, semantically coherent concept attributions by jointly optimizing reconstruction fidelity and intra-set semantic coherence. SITH enables weight-space edits—amplifying or suppressing specific singular vectors—that improve downstream performance without retraining.

Problem

Existing mechanistic interpretability methods for VLMs (e.g., Sparse Autoencoders, TextSpan) are activation-based, requiring large-scale image datasets, and are therefore susceptible to dataset bias. They also produce coarse, head-level explanations rather than identifying which internal sub-components within a head encode specific concepts. Weight-based interpretability for multimodal transformers has been underexplored; prior SVD analyses of transformer weights apply only to unimodal language models and use simple nearest-neighbor search rather than coherent sparse decompositions.

Method

SITH operates on the value-output matrix \(W_{VO}^{l,h} = W_V^h W_O^h \in \mathbb{R}^{D \times D}\) of each attention head \(h\) in layer \(l\) of CLIP's ViT. It applies SVD:

\[W_{VO} = U \Sigma V^T\]

The right singular vectors \(v_i\) correspond to output directions the head writes to the residual stream; their magnitudes \(\sigma_i\) quantify writing strength.

COMP maps each projected singular vector \(\hat{v}_j = v_j W_p\) (pushed into CLIP's shared \(d\)-dimensional space) to a sparse non-negative combination of textual concept embeddings \(\hat{\Gamma} = E_T(\Gamma) \in \mathbb{R}^{C \times d}\) by solving:

\[\min_c \|c\|_0 \quad \text{s.t.} \quad \|\hat{v} - \hat{\Gamma}^T c\|_2^2 \leq \epsilon, \quad c \geq 0\]

Unlike standard Non-Negative OMP (NNOMP), COMP's per-iteration concept selection criterion adds a coherence regularizer:

\[\text{score}(\hat{\gamma}_i) = \langle r_{k-1}, \hat{\gamma}_i \rangle + \frac{\lambda}{|S_{k-1}|} \sum_{j \in S_{k-1}} \langle \hat{\gamma}_i, \hat{\gamma}_j \rangle\]

where \(\lambda\) trades off reconstruction fidelity against intra-set semantic consistency. Model editing is performed by scaling \(\sigma_i\) for targeted singular vectors, guided by LLM-evaluated concept relevance to a downstream task.

Key Contributions

  • SITH framework: fully data-free and training-free interpretability of CLIP attention heads at the intra-head (singular vector) granularity, contrasting with head-level (TextSpan) or sample-level (SAEs) approaches.
  • COMP algorithm: coherence-regularized greedy sparse coding that outperforms both NNOMP and naive top-\(k\) in the interpretability–fidelity trade-off.
  • Interpretable weight-space editing: amplifying or suppressing singular values suppresses spurious correlations, reduces sensitivity to unsafe content, and improves zero-shot classification—all without data or retraining.
  • Fine-tuning analysis: SITH reveals that fine-tuning reweights a stable semantic basis rather than creating new features, providing mechanistic insight into model adaptation.

Results

  • Interpretability–fidelity: COMP (\(\lambda=0.3\), \(K=5\)) achieves the best balance between interpretability score (LLM-as-judge, 5-point Likert) and fidelity (cosine similarity) over all tested sparsity levels; NNOMP achieves high fidelity but low coherence; top-\(k\) achieves high coherence but poor reconstruction completeness.
  • Zero-shot classification after reconstruction (layer \(l=23\), ViT-L/14): replacing original singular vectors with SITH-reconstructed ones produces negligible accuracy drop across ImageNet, Flowers102, Oxford Pets, and Stanford Cars, validating reconstruction faithfulness.
  • Image grounding on CC12M: SITH achieves consistently higher mean image-concept agreement than TextSpan (with either its original dictionary or ConceptNet 5.5) across the final four ViT layers, with the strongest agreement in the last layer.
  • Retrieval safety editing (ViSU benchmark): SITH-based singular value suppression improves R@10 for safe queries while reducing unsafe retrievals (full numbers truncated in provided text).

Limitations

  • Certain singular vectors encode non-semantic or task-irrelevant "noisy" information that cannot be mapped to interpretable concepts, even at high sparsity \(K=50\).
  • The quality of concept attribution depends on the completeness of the concept pool (\(\Gamma\)); ConceptNet 5.5 may lack domain-specific terminology.
  • COMP requires tuning \(\lambda\) and \(K\); the optimal setting is validated empirically rather than theoretically derived.
  • Analysis is restricted to CLIP (ViT-L/14 with OpenCLIP weights); generalization to other VLMs or text-tower heads is not demonstrated.
  • LLM-as-a-judge evaluations (GPT-5-mini) introduce an external model dependency and potential subjectivity in interpretability scoring.

Relevance to Vision-Language Models

SITH addresses a central open problem in VLM research: mechanistically explaining what CLIP's visual encoder has learned without any inference-time data, directly in weight space. This is significant because most interpretability work on VLMs (TextSpan, SAEs) requires large activation datasets and yields coarse, head-level attributions, whereas SITH exposes sub-head structure at the singular-vector level—relevant to understanding how multimodal representations are composed. The weight-space editing capability opens a data-free path to targeted debiasing and concept suppression, a practical concern for safe deployment of CLIP-based retrieval and classification systems. The finding that fine-tuning primarily reweights a stable semantic basis rather than learning new directions has direct implications for understanding CLIP adaptation methods (e.g., prompt tuning, LoRA) and for the emerging literature on why pre-trained VLM representations transfer so effectively.