Skip to content

Dataset Distillation by Influence Matching

🕒 Published (v1): 2026-07-18 15:47 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; outcome-centric dataset distillation; broadly applicable to VLM training data

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Influence Matching (Inf-Match) reformulates dataset distillation as outcome alignment rather than process imitation: it learns a compact synthetic set whose influence on the final converged model parameters matches that of the full dataset. The key enabler is a fully differentiable, linear-time influence estimator that avoids inverse-Hessian computation and convexity assumptions. The method achieves state-of-the-art results on standard classification benchmarks and extends to vision-language distillation on Flickr30K.

Problem

Existing dataset distillation methods match intermediate training surrogates (per-step gradients or trajectories) rather than the final trained model, creating an optimization gap: synthetic data can excel at proxy objectives while still underperforming on downstream accuracy and generalization. Classical influence estimators that could enable outcome-level matching are computationally prohibitive (require inverse-Hessian products) and assume convex losses that do not hold for deep networks.

Method

Inf-Match defines the addition influence of a synthetic set \(S\) and the removal influence of the real dataset \(D\) on the converged parameters, then minimizes their sum: $\(S^* = \arg\min_S \|I_{-D} + I_{+S}\|\)$ This forces the parameter shift induced by \(S\) to cancel that of removing \(D\), achieving outcome alignment without retraining.

The core is a first-order Taylor estimator derived by unrolling SGD dynamics along a pre-recorded training trajectory \(\{(\theta^t_D, \eta_t)\}_{t=1}^T\): $\(I_{+Z} \approx \sum_t \frac{\eta_t \sum_{k \geq t} \eta_k}{|D|} \left(H^t_D G^t_Z + H^t_Z G^t_D\right)\)$ Hessian-gradient products are computed via finite differences (\(O(p)\) per step), yielding linear-time complexity. A provable error bound scales polynomially in \(T\) (versus exponentially for prior estimators). During optimization, both synthetic images and soft labels are treated as learnable variables; mini-batches \(B_S \subset S\) and \(B_D \subset D\) are sampled each iteration for efficiency. Checkpoint sampling follows an easy-to-hard curriculum (early checkpoints first, later ones as training progresses).

Key Contributions

  • Outcome-centric dataset distillation objective: minimize the residual parameter displacement \(\|I_{-D} + I_{+S}\|\) rather than a proxy surrogate.
  • Differentiable, linear-time influence estimator requiring no inverse-Hessian and no convexity assumption, with a polynomial (tight) error bound.
  • First application of influence-based distillation to vision-language datasets (Flickr30K image-text retrieval).
  • Consistent state-of-the-art across all IPC settings on CIFAR-10, CIFAR-100, and Tiny-ImageNet, and strong cross-architecture generalization.

Results

  • CIFAR-10: 49.9% (IPC=1), 72.5% (IPC=10), 78.1% (IPC=50); ~0.7% above NCFM at IPC=10/50.
  • CIFAR-100: 49.3% (IPC=10), 57.4% (IPC=50); +2.7% over NCFM at IPC=50.
  • Tiny-ImageNet: 31.5% (IPC=10), 33.8% (IPC=50); +4.7% and +4.2% over NCFM respectively.
  • Flickr30K (vision-language): with 200 synthetic pairs, I2T Recall@1 exceeds DATM by 1.3%; over 200–1000 pairs, average image/text retrieval surpasses NCFM by 2.5%.
  • Cross-architecture generalization (CIFAR-100, IPC=50): consistently outperforms DATM across ConvNet, ResNet-18, VGG, and AlexNet (45.4%–57.4%).

Limitations

  • Error bound scales polynomially in training steps \(T\) and Lipschitz constant \(\ell\); for very long training runs or highly non-smooth losses, the bound may be loose.
  • Requires storing a full training trajectory of parameter checkpoints, adding memory overhead proportional to \(T\).
  • Vision-language experiments use encoders pretrained only on unimodal data with a linear projection bridge; it is unclear how the method scales to end-to-end pretrained VLMs (e.g., CLIP, LLaVA).
  • Experiments use relatively small synthetic set sizes (up to 1000 image-text pairs for VL), leaving scalability to larger distillation regimes untested.
  • The paper text is truncated; full ablation details and failure modes may not be fully reported here.

Relevance to Vision-Language Models

Inf-Match is the first dataset distillation method demonstrated on a vision-language retrieval benchmark (Flickr30K), showing that outcome-aligned distillation transfers beyond unimodal classification. For researchers tracking VLMs, this opens the question of whether large-scale multimodal pretraining datasets (e.g., for CLIP-style or generative VLMs) can be aggressively compressed while preserving cross-modal alignment. The influence estimator's avoidance of inverse-Hessian computation makes it plausible to scale to the larger parameter counts typical of VLM backbones. The approach also complements privacy-preserving and federated VLM training scenarios where sharing raw image-text pairs is restricted.