Skip to content

Label-Free Mitigation of Spurious Correlations in VLMs using Sparse Autoencoders

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; label-free SAE-based spurious correlation removal in VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DIAL is a fully zero-shot, label-free framework that uses Sparse Autoencoders (SAEs) to disentangle VLM embeddings, identify spurious feature directions, and remove them via orthogonal projection. No model retraining, auxiliary data, class labels, or LLM calls are required. DIAL+ extends this to automatically detect spurious attributes without any prior knowledge of what they are.

Problem

VLMs like CLIP learn spurious correlations (non-causal features coincidentally prevalent in training data) that degrade worst-group performance. Existing mitigation methods require at least one of: additional data, model fine-tuning, class or spurious-feature labels, domain expertise, or external LLM calls—all of which undermine the zero-shot scalability that makes VLMs valuable.

Method

DIAL operates on frozen VLM image embeddings in three stages:

  1. Candidate selection: Zero-shot pseudo-labels are used as class proxies; samples deviating from their pseudo-class centroid (confirmed by k-NN inconsistency) are flagged as likely spuriously affected.

  2. Spurious subspace identification: A pre-trained Matryoshka Sparse Autoencoder (MSAE) projects embeddings into a disentangled dictionary \(\{f_j\}_{j=1}^l\). For each known spurious attribute \(a\), an attribution score $\(S(f_j, a) = \left(\frac{1}{|P_a|}\sum_{i \in P_a} z_{i,j} - \frac{1}{|N_a|}\sum_{i \in N_a} z_{i,j}\right) \times \mathrm{CosSim}(f_j, e_a)\)$ ranks features by their differential activation and semantic alignment with \(a\). Top-\(k\) features covering fraction \(\alpha\) of total attribution mass form the spurious set \(K\).

  3. Orthogonal projection: Spurious vectors in \(K\) are weighted by alignment with the set mean, pruned by percentile, QR-decomposed into an orthonormal basis \(Q\), and subtracted from candidate embeddings: \(\hat{e}_{i,\text{clean}} = \hat{e}_i - \lambda QQ^\top \hat{e}_i\), where \(\lambda \in [0,1]\) controls mitigation strength.

DIAL+ replaces the attribute list with a data-driven detection step: it pools influential SAE concepts (features whose ablation changes the zero-shot prediction) across all candidate samples and selects the most frequently activated ones as spurious concepts. Hyperparameters \((\alpha, \lambda, k^*)\) are chosen via a grid search scored by alignment between embeddings and detected spurious features—no ground-truth labels needed.

Key Contributions

  • DIAL: fully zero-shot spurious-correlation mitigation—no training, labels, auxiliary data, or LLMs.
  • DIAL+: autonomous detection of spurious attributes from SAE activations alone, maintaining performance close to DIAL.
  • Novel attribution score adapted to zero-shot settings to rank SAE feature directions by spurious alignment.
  • Weighted QR orthogonal projection for denoised subspace removal.
  • Label-free hyperparameter selection algorithm using embedding–spurious-feature alignment as a proxy metric.
  • Validated across 5 benchmarks (CelebA, Waterbirds, FMOW, ISIC, COVID-19) and multiple CLIP backbones, plus debiased image retrieval on FairFace.

Results

  • CelebA (ViT-B/32): DIAL achieves WG 83.47% and Gap 2.17%, outperforming all zero-shot baselines and even label-/data-dependent methods (TIE: WG 82.63%, Gap 2.48%).
  • CelebA (ViT-L/14): DIAL WG 85.24%, Gap 1.63%—best among all methods; DIAL+ WG 85.15%, Gap 1.39%.
  • Waterbirds (ViT-L/14): DIAL WG 68.69%, Gap 13.91%; DIAL+ Gap 12.47%—significantly better than zero-shot (WG 31.93%, Gap 51.79%) and competitive with TIE (WG 78.82%, Gap 5.30%, which requires spurious feature labels per sample).
  • FMOW (ViT-L/14): DIAL WG 19.90% vs. zero-shot WG 18.06%; DIAL+ WG 19.24%.
  • ISIC: DIAL WG 68.42%, Gap 2.29%—surpasses all baselines including label-dependent TIE (WG 65.87%, Gap 4.03%); zero-shot baseline WG 42.21%, Gap 28.00%.
  • COVID-19: DIAL WG 48.28%, Gap 12.83% vs. zero-shot WG 44.83%, Gap 16.98%.

Limitations

  • Requires access to a pre-trained SAE for the specific VLM backbone; SAE quality directly bounds debiasing quality.
  • Performance on Waterbirds is weaker than TIE (which uses per-sample spurious labels), likely because highly varied spatial concepts like "water background" are hard to capture with a single semantic embedding.
  • DIAL requires at least high-level spurious attribute descriptions (e.g., "Male"/"Female"); DIAL+ can detect them automatically but relies on the SAE's disentanglement quality.
  • Hyperparameter grid search is needed per dataset, even though no labels are used—compute cost not analyzed.
  • Evaluation confined to CLIP-family models; generalization to non-contrastive VLMs (e.g., generative VLMs) is untested.

Relevance to Vision-Language Models

Spurious correlations in CLIP-style VLMs are a known failure mode that directly undermines their zero-shot value proposition, particularly in high-stakes domains like medical imaging. DIAL demonstrates that SAE-based representation disentanglement—a tool imported from mechanistic interpretability—can serve as a practical, post-hoc debiasing mechanism without touching model weights or requiring labeled data, which is a meaningful capability gap filled relative to prior work. The method is backbone-agnostic (tested on ViT-B/32, ViT-L/14, BiomedCLIP, and others) and applicable at inference time, making it immediately relevant to anyone deploying frozen CLIP embeddings. For researchers tracking VLMs, this establishes SAEs as a viable interpretability primitive not just for understanding but for actively correcting representation-level biases.