Matched Data, Better Models: Target Aligned Data Filtering with Sparse Autoencoders¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; SAE-based target-aligned data filtering for VLM pretraining

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Submodular Distribution Matching (SDM) is a data filtering framework for VLM pretraining that selects image-caption subsets whose concept distribution matches a target dataset, rather than scoring samples independently. It trains a sparse autoencoder (SAE) on CLIP embeddings to produce disentangled, monotone features, then maximizes a feature-based submodular objective equivalent to minimizing KL divergence to a target concept distribution. On DataComp-medium (128M pairs), SDM achieves state-of-the-art ImageNet-1K accuracy using only a single CLIP backbone and >5× fewer GPU hours than the leading approach.

Problem¶

Existing VLM data filtering methods (CLIP-Score, NegCLIPLoss, NormSim) evaluate each sample independently and retain those above a quality threshold. This ignores dataset-level distributional properties: individually high-quality samples can be redundant with each other, while individually low-quality samples may cover rare but important concepts. The result is imbalanced training sets that degrade generalization.

Method¶

SDM operates in three stages:

SAE training. A $k$-sparse autoencoder is trained on frozen CLIP ViT-L/14 image embeddings ($d_{in}=768$) to produce sparse, high-dimensional representations ($d_{sparse}=98{,}304$). The SAE is trained with a reconstruction loss plus an activity regularizer and a novel monotonicity loss $L_{\text{mono}}$: $$L_{\text{mono}}(E,M) = |\Delta(E|M)| \cdot \log\!\left(1 + \exp\!\left(1 - \frac{f(E)-f(M)}{\Delta(E|M)}\right)\right)$$ This contrastive "peripteral" loss biases features to increase monotonically with concept presence by pushing heterogeneous sets $E$ to score higher under $f(A)=\sum_i \log(1+m_i(A))$ than homogeneous sets $M$.
Target distribution estimation. The empirical concept distribution $p^{\text{tar}}$ is computed from a target dataset (ImageNet-1K training set by default) by aggregating SAE activations over all target samples.
Submodular subset selection. SDM maximizes the combined objective: $$\arg\max_{A \subseteq V,\, |A|=b}\ \lambda \sum_{i} p^{\text{tar}}_i \log(1+m_i^{\text{source}}(A)) + (1-\lambda)\sum_{i} u_i \log\!\left(1+\sum_{j\in A} \mathbf{1}[q(j)\in[b_{i-1},b_i)]\right)$$ The first term matches the source subset's concept distribution to the target; the second incorporates a quality score (NegCLIPLoss) quantized into $\ell$ bins to avoid quality dominating marginal gains. Maximization uses the stochastic greedy algorithm run 5 times with the intersection of results taken.

The KL minimization objective ($\min D_{\text{KL}}(p^{\text{tar}} \| p^{\text{source}}(A))$) is proved to be a Difference-of-Submodular (DS) problem; SDM maximizes a tractable submodular lower bound.

Key Contributions¶

Novel submodular data selection framework (SDM) that treats selection as distribution matching over disentangled concept features.
New monotonicity loss term for SAEs, ensuring feature magnitudes scale with concept presence—enabling valid aggregation of concept counts across a dataset.
Theoretical connection: minimizing $D_{\text{KL}}(p^{\text{tar}}\|p^{\text{source}}(A))$ is a DS-maximization problem; a tractable submodular lower bound is derived and justified.
Quantized quality-score integration into the submodular objective, avoiding quality-score dominance at large subset sizes.
State-of-the-art results on DataComp-medium using only a single CLIP backbone and no external curated models.

Results¶

ImageNet-1K: SDM (18M samples) achieves 35.2%, vs. prior SOTA NCL∩NormSim at 32.7–32.8% — a +2.5% gain with 18% fewer samples.
Average across 38 tasks: SDM 36.4% vs. NCL∩NormSim 35.7% (+0.7%).
VTAB: 38.6% vs. 37.5% for the next best.
Retrieval: 26.8% vs. 26.5%.
At 33% fewer samples than NCL∩NormSim (22M), SDM still outperforms by ~2% on ImageNet.
SDM matches the full DataComp-medium benchmark SOTA (which uses external models/more compute) within 1% while using >5× fewer GPU hours.
Gains hold across alternate CLIP backbones (ViT-B/32, ViT-L/14, DFN-P) and quality scores (CLIPScore, NegCLIPLoss).

Limitations¶

The target distribution choice (ImageNet-1K) biases selection; using a different target requires re-running stochastic greedy.
Subset selection runs 5× stochastic greedy (5 CPU hours total, parallelizable), but this adds pipeline complexity vs. simple threshold filtering.
SAE training (15 A100 hours) and feature extraction (60 A100 hours) are non-trivial upfront costs, though amortizable.
Monosemanticity/monotonicity of SAE features is defined relative to an unspecified concept granularity—the learned "concepts" are not guaranteed to be semantically meaningful at a fixed level.
Experiments restricted to DataComp-medium (128M pairs); scaling behavior to billion-scale datasets is unverified.
Quality score (NegCLIPLoss) measures only image-caption alignment; other quality dimensions are not captured.

Relevance to Vision-Language Models¶

SDM directly targets a core bottleneck in CLIP-style VLM pretraining: the quality and diversity of the pretraining corpus. By framing data curation as concept-distribution matching rather than per-sample quality thresholding, it provides a principled mechanism to control which visual concepts—and in what proportions—the VLM sees during training. The use of SAEs to disentangle CLIP embeddings is particularly relevant to the VLM interpretability line of work, as it bridges mechanistic interpretability (monosemantic feature identification) with practical data engineering. The submodular framework is general: it can be applied with any frozen VLM encoder and any target distribution, making it applicable to domain-specific VLM adaptation (e.g., biomedical vision-language models) where external scoring models are unavailable.