Skip to content

Filter Like You Test: Data-Driven Data Filtering for CLIP Pretraining

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FLYT (Filter Like You Test) is a meta-learning algorithm that trains a scoring model to select CLIP pretraining data by backpropagating gradient signals from downstream task performance, replacing heuristic proxies with a directly optimized objective. Combined with Mixing-FLYT (linear combination of existing scores as features) and Soft Cap Sampling (repetition-penalized probabilistic sampling), it achieves 40.1% ImageNet zero-shot accuracy on DataComp medium scale — a 2% absolute gain over all prior work and 5.5% over methods using only public resources.

Problem

All prior state-of-the-art data filtering methods for CLIP pretraining rely on heuristic proxies for example quality (CLIP cosine similarity, hyperbolic entailment scores, specificity metrics, etc.) rather than directly optimizing for the downstream performance these datasets are meant to enable. There is no principled way to aggregate multiple such heuristics, and existing aggregation strategies (score summation, intersection/union, pipelines) are themselves heuristic.

Method

FLYT frames data selection as learned data re-weighting via bilevel optimization:

  1. A scoring model \(q_\phi\) takes extracted features \(\Psi(z)\) from an upstream image-text pair and outputs a scalar score \(s\). Scores are converted to per-example weights via softmax over the batch.
  2. A reference CLIP model \(f_\theta\) is updated using a weighted CLIP loss \(L_\text{up}\) (standard contrastive loss where both the numerator and denominator terms are modulated by the example weights \(w_i\)).
  3. The updated reference model \(f_{\theta^+}\) is then evaluated on a downstream batch (ImageNet training set), computing a downstream loss \(\hat{L}_\text{down}\) (cross-entropy or CLIP loss).
  4. Gradients flow from \(\hat{L}_\text{down}\) back through \(f_{\theta^+}\), through the weights, to \(\phi\), training the scoring model to upweight examples that reduce downstream loss.

Mixing-FLYT (M-FLYT): Instantiates the feature extractor as a vector of 12 existing filtering scores (ViT-B/32 CLIP score, ViT-L/14, DFN-Base, DFN-FT, HYPE's \(-d_L\)/\(\epsilon_i\)/\(\epsilon_t\), s-CLIPLoss, NormSim\(_\infty\), two binary classifiers, E-FLYT). A linear scoring model learns optimal combination weights, outperforming all individual scores and heuristic aggregations.

Embedding-FLYT (E-FLYT): Uses a pretrained CLIP model to produce image+text embeddings as features. Contributes positively to M-FLYT but does not match M-FLYT alone.

Soft Cap Sampling (SCS): Rather than threshold-selecting the top-\(k\%\), samples from the FLYT probability distribution iteratively in batches of \(G=100K\), subtracting a repetition penalty \(\alpha\) from each sampled example's score after selection. This prevents over-representation (top example appears up to 56 times in the best dataset) while preserving the probabilistic structure that pure top-\(k\) discards.

Key Contributions

  • FLYT: First data filtering method for CLIP that directly optimizes a downstream task objective via meta-gradient descent through a bilevel loop (scoring model ↔ reference model ↔ downstream loss).
  • M-FLYT: A principled learned aggregation of existing filtering scores, replacing heuristic summation/intersection; linear model suffices and weights are interpretable.
  • Soft Cap Sampling (SCS): A sampling strategy that interpolates between sampling-with-replacement (too repetitive) and sampling-without-replacement (too few examples) via an additive repetition penalty, improving both diversity and score utilization.
  • State-of-the-art on DataComp medium and small filtering benchmarks using only public resources.
  • Full code, models, filtered datasets, and input scores released publicly.

Results

  • DataComp medium, ImageNet zero-shot: M-FLYT+SCS 40.1% vs. best public-resource prior (s-CLIPLoss) 33.3%; vs. best non-public prior (HYPE+DFN+s-CLIPLoss) 38.1%.
  • DataComp medium, 38-task average: M-FLYT+SCS 37.7% vs. best public-resource prior 35.2%; vs. best non-public prior 37.8% (within 0.1%).
  • DataComp medium, ImageNet distribution shifts: 0.311 vs. 0.310 (best non-public).
  • DataComp medium, VTAB: 0.396 vs. 0.392 (best non-public).
  • M-FLYT alone (top-20% threshold): 35.9% ImageNet, 37.1% average — already outperforms best public-resource baseline.
  • SCS applied to IN-weighted baseline: +2.6% ImageNet, −0.3% average; applied to M-FLYT: +4.2% ImageNet, +0.6% average — showing FLYT's probabilistic output is synergistic with SCS.
  • M-FLYT vs. heuristic aggregations: Outperforms standardized sum by 1.1% ImageNet, IN-weighted sum (best \(r\)) by 0.5% ImageNet.
  • DataComp small scale: M-FLYT+SCS 8.0% ImageNet, 18.5% average — outperforms all prior small-scale methods including HYPE (6.3%) and EcoDatum (5.3%).

Limitations

  • Scale untested: Only evaluated at DataComp medium and small; large-scale experiments were computationally infeasible; whether scoring model complexity needs to grow with data/model size is unresolved.
  • E-FLYT underperforms DFN standalone: End-to-end learning from embeddings does not yet surpass the best single heuristic filter, requiring M-FLYT's ensemble to compensate.
  • Downstream data diversity: Uses only ImageNet training set as downstream signal; the 38 DataComp evaluation tasks are more diverse, and optimizing solely for ImageNet may limit generalization (reflected in retrieval score 0.292, lower than some baselines).
  • Data weighting ≠ data selection: The FLYT objective approximates selection via continuous reweighting; direct optimization of discrete selection (e.g., via RL) is not attempted.
  • Repetition: Top examples appear up to 56 times in the final dataset; the implications of such extreme repetition for model robustness and memorization are unexamined.
  • Computational overhead: M-FLYT training requires ~1/3 the FLOPs of a full DataComp medium training run; cost scaling at larger data/model regimes is unknown.

Relevance to Vision-Language Models

CLIP pretraining data quality is the primary lever controlling the zero-shot capabilities of contrastive VLMs and, transitively, the vision encoders that underpin larger VLMs (LLaVA, InternVL, etc.). FLYT replaces the dominant paradigm of CLIP-score-based heuristic filtering with a principled meta-learning objective, demonstrating that directly optimizing downstream task gradients can substantially improve foundation model quality without modifying the training recipe. The M-FLYT aggregation framework is broadly applicable — any set of quality signals can be plugged in as features — and SCS introduces a novel probabilistic view of dataset curation that could apply beyond CLIP to multimodal pretraining generally. The concurrent connection to metagradient descent (MGD) suggests this family of methods will likely define the next generation of data curation for VLMs.