Skip to content

CovMatch: Cross-Covariance Guided Multimodal Dataset Distillation with Trainable Text Encoder

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CovMatch is a multimodal dataset distillation framework that synthesizes compact image-text pairs for training vision-language models by aligning the cross-covariance matrices of real and synthetic features. Unlike prior methods that freeze the text encoder for computational tractability, CovMatch jointly optimizes both image and text encoders via a closed-form linearized contrastive objective, eliminating the need for expert trajectory storage. With only 500 synthetic pairs, it achieves up to 6.8% absolute improvement over state-of-the-art baselines on image-text retrieval.

Problem

Existing multimodal dataset distillation methods (MTT-VL, LoRS) freeze the text encoder (e.g., BERT, 450MB) during distillation to avoid the prohibitive cost of bi-level optimization and expert trajectory matching over large encoders—requiring 120GB+ storage and 5+ days of training on A100s. However, a frozen text encoder cannot adapt to visual context, causing text embeddings of semantically related captions to remain unclustered and retrieval performance to saturate or degrade as synthetic data size grows beyond N=1,000 pairs.

Method

CovMatch derives a closed-form solution for the inner optimization of bi-level dataset distillation by (1) fixing the encoders during each distillation step and optimizing only the linear projection layers Gv and Gl, and (2) adopting a linear multimodal contrastive loss. Under this setup, the inner optimization has the closed-form optimal solution Ĝ_v^⊤Ĝ_l = (1/ρ)C^S, reducing the outer objective to maximizing Tr(C^T · C^S)—i.e., aligning the cross-covariance matrices of real (C^T) and synthetic (C^S) image-text features.

The final CovMatch loss is:

L_CovMatch = L_cov + λ·(L_v^feat + L_l^feat)

  • Cross-covariance matching loss (L_cov): Frobenius norm between ρ·C^T and C^S, correcting for dataset size mismatch via scaling factor ρ.
  • Feature matching loss (L_m^feat): MMD between projected real and synthetic mean embeddings, per modality; regularizes against degenerate solutions arising from the low-rank structure of C^T.

Only the token/position embedding layer of the text encoder is frozen; transformer layers participate in distillation. An online model update (one gradient step on real data) precedes each distillation step, with periodic reinitialization to pretrained weights every T steps to prevent overfitting to a fixed encoder state. No expert trajectories are stored.

Key Contributions

  • Theoretical reduction of multimodal dataset distillation to cross-covariance alignment via linearized contrastive loss, yielding a closed-form inner solution that avoids unrolled optimization or trajectory matching.
  • CovMatch algorithm enabling joint optimization of both image and text encoders at 0GB expert storage, 15GB distillation memory, and 1.2 sec/iter vs. 132h+ for trajectory-based methods.
  • Feature matching regularizer to handle the empirically observed low-rank structure of real cross-covariance matrices.
  • Demonstrated cross-architecture generalization of distilled datasets to unseen encoders (NF-ResNet, NF-RegNet, ViT, DistilBERT).

Results

  • Flickr30K, 500 pairs: CovMatch Avg=38.4% vs. LoRS 31.6% (+6.8% absolute); MTT-VL 25.0%.
  • COCO, 500 pairs: CovMatch Avg=19.6% vs. LoRS 13.5% (+6.1% absolute); MTT-VL 12.6%.
  • Flickr30K, 100 pairs: CovMatch Avg=30.5% vs. LoRS 27.4%, MTT-VL 20.4%.
  • COCO, 100 pairs: CovMatch Avg=11.5% vs. LoRS 9.4%, MTT-VL 7.4%.
  • Cross-architecture (Flickr30K, 100 pairs, BERT+NFNet distill, BERT+NF-ResNet eval): CovMatch 15.5% vs. LoRS 8.8%, MTT-VL 8.8%.
  • LoRS performance saturates/degrades beyond N=1,000 pairs; CovMatch continues to improve with more synthetic data.
  • Full dataset upper bounds: Flickr30K Avg≈75.7%, COCO Avg≈52.7% across six metrics.

Limitations

  • Requires pretrained unimodal encoders (NFNet, BERT); the framework has not been validated for training contrastive models from scratch without pretrained backbones.
  • Evaluated only on image-text retrieval; generalization to generation or captioning tasks is unexplored.
  • Linear contrastive loss is an approximation of InfoNCE; the gap between the surrogate objective and the true contrastive objective is not formally bounded.
  • Hyperparameters ρ and λ require tuning and exhibit dataset-size-dependent optimal values, complicating practical deployment.

Relevance to Vision-Language Models

CovMatch directly addresses scalable training data compression for CLIP-style contrastive VLMs, a setting where data efficiency is critical given the reliance on hundreds of millions of image-text pairs. The finding that freezing the text encoder is a hard performance bottleneck for cross-modal alignment is directly actionable for practitioners designing distillation or fine-tuning pipelines for models like CLIP, ALIGN, and BLIP. The cross-covariance matching objective offers a principled, computationally tractable alternative to trajectory matching for bi-modal representation learning. As VLMs scale toward larger encoders, CovMatch's elimination of expert trajectory storage (120GB → 0GB) and its support for joint encoder optimization make it a practically relevant efficiency technique.