Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; Sparse CLIP co-optimizes interpretability and performance in CLIP

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Sparse CLIP integrates sparsity directly into CLIP pretraining via two minimal changes—ReLU activation after the projection layer and high-dimensional embedding expansion—yielding representations that are simultaneously interpretable and performant. Unlike post-hoc Sparse Autoencoders (SAEs), the resulting features are genuinely multimodal, match or exceed dense CLIP accuracy, and support interpretable vision-based steering in VLMs. This challenges the prevailing assumption that interpretability and accuracy are fundamentally at odds.

Problem

CLIP's dense latent representations are opaque and difficult to interpret. Post-hoc approaches using Sparse Autoencoders (SAEs) to add interpretability suffer from (1) degraded downstream task performance relative to dense counterparts, and (2) predominantly unimodal features—most SAE features activate exclusively for images or text, not both, defeating CLIP's cross-modal purpose.

Method

Sparse CLIP modifies standard CLIP pretraining with two changes: (1) a ReLU non-negativity constraint added after the final projection layer, and (2) high-dimensional embedding expansion (expansion factor of 72× for ViT-L/14, yielding a 55,296-dimensional representation). These changes are theoretically grounded: non-negative contrastive learning is equivalent to Non-negative Matrix Factorization (NMF), placing it in the same dictionary learning framework as SAEs. The sparsity level is controlled by tuning the logit scale cap; two models are trained at caps 50 and 40, producing 0.66% and 0.47% L0 sparsity respectively (ViT-L/14 Sparse and Sparse+). For the VLM application, ViT-L/14 Sparse+ is connected to Llama 3.1 8B Instruct via a 2-layer MLP adapter.

Key Contributions

• Demonstrates that non-negativity constraints + dimension expansion during CLIP pretraining induce sparsity without a reconstruction loss, producing interpretable multimodal features without the SAE decoder.
• Theoretically frames the approach within the dictionary learning / NMF equivalence established by prior work.
• Shows that Sparse CLIP features are predominantly multimodal (unlike SAE features), enabling direct text-based feature naming via a 98,619-word vocabulary.
• Provides first visualization of concept emergence and evolution throughout CLIP training using native interpretability of intermediate checkpoints.
• Demonstrates interpretable vision-based steering in a VLM built on Sparse CLIP representations.

Results

Zero-shot classification (Table 1, ViT-L/14 scale): - ViT-L/14 Sparse (0.66% L0) outperforms dense baseline by +0.5% average across 6 standard benchmarks and +0.7% across 6 fine-grained benchmarks. - ViT-L/14 Sparse+ (0.47% L0) performs approximately on par with the dense baseline. - Prisma SAE (post-hoc) incurs 1–2% accuracy loss on the same benchmarks.

Additional downstream tasks (Table 2, COCO): - Both Sparse models significantly outperform the baseline on bounding box classification. - Both Sparse models underperform on zero-shot retrieval (IR@1: 43.7 and 41.8 vs. baseline 45.5), attributed to the subject-focus bias induced by low L0 and reduced logit scale.

Interpretability (Table 3a): - Clarity score: ViT-L/14 Sparse+ achieves 0.549/0.559 vs. 0.519–0.521 for CLIP SAEs (ImageNet / MetaCLIP subsets). - Active feature percentage: 88.3%/85.5% vs. 45.1%–41.0% for SAEs. - L0: 468.5/344.3 vs. 916.0+ for SAEs.

VLM benchmarks (Table 3, ViT-L/14 Sparse+ backbone): - MMMU: 41.7 vs. baseline 39.6 - TextVQA: 48.5 vs. baseline 48.9 - POPE: 80.7 vs. baseline 80.8 - AI2D: 70.6 vs. baseline 67.2

Limitations

• Dimension expansion factor is bounded by GPU memory: 72× is the maximum achievable for ViT-L/14 on 80 GB GPUs; larger expansion might yield further gains.
• Sparse CLIP consistently underperforms on zero-shot retrieval (e.g., COCO), likely because low L0 sparsity and reduced logit scale bias representations toward dominant image subjects, hurting multi-subject caption matching.
• No complete theoretical explanation for why logit scale cap controls sparsity level.
• Interpretability analysis of feature evolution is qualitative and observational; causal mechanisms (knowledge-driven vs. noise-driven transformations) are not characterized.
• The VLM demonstration is minimal (small adapter, limited training data), and steering is only evaluated on ImageNet-1k classification prompts.

Relevance to Vision-Language Models

Sparse CLIP directly impacts VLMs because CLIP vision encoders are the default visual backbone in virtually all modern MLLMs (LLaVA, InternVL, etc.); improvements in interpretability and performance of the encoder propagate to all downstream systems. The demonstration of interpretable vision-based steering—suppressing or amplifying specific semantic concepts in VLM output by manipulating sparse activations before the adapter—offers a practical mechanism for fine-grained, concept-level control not possible with dense representations. For researchers tracking VLMs, the key insight is that the interpretability/accuracy trade-off may be a product of post-hoc methodology rather than an inherent constraint, suggesting a design direction where future vision backbones are trained with native sparsity. The finding that multimodal features emerge early in training also has implications for understanding cross-modal alignment in contrastive pretraining.