Multimodal Distribution Matching for Vision-Language Dataset Distillation¶

🕒 Published (v1): 2026-05-22 10:41 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; multimodal distribution matching for vision-language dataset distillation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

MDM (Multimodal Distribution Matching) is a geometry-aware dataset distillation framework that compresses large image-text paired datasets into compact synthetic surrogates by aligning joint image-text feature distributions on the unit hypersphere, rather than replaying expensive training trajectories. It introduces cluster-based synthetic data initialization, angle-guided weight-space model merging, and a geodesic kernel energy matching objective. MDM achieves competitive or superior retrieval performance versus trajectory-based state-of-the-art methods at substantially lower computational cost.

Problem¶

Existing multimodal dataset distillation methods (MTT-VL, LoRS) rely on trajectory replay—bi-level optimization over teacher/student training dynamics—which incurs heavy compute and memory overhead and introduces architectural bias that limits cross-architecture generalization. No prior method efficiently aligns the joint image-text distribution directly in a geometry-aware, architecture-agnostic manner.

Method¶

MDM operates at three levels:

Data initialization: All real image-text pairs are embedded via a joint encoder $\Psi$ by concatenating $\ell_2$-normalized image and text features. K-means clustering ($K = |D_\text{syn}|$) partitions this joint space; each synthetic pair is seeded with the real sample whose joint feature is cosine-closest to its cluster centroid, ensuring broad semantic coverage.

Model initialization: Rather than using a single pretrained or finetuned encoder, MDM merges $N$ independently finetuned experts with the pretrained anchor via layer-wise weight-space interpolation: $$\theta^{m,\ell}_* = \theta^{m,\ell}_0 + \alpha \, t^m_\ell \cdot \tfrac{1}{2}(\Delta^m_{1,\ell} + \Delta^m_{2,\ell}),$$ where the merging ratio $t^m_\ell$ is determined by the mutual angle between displacement vectors $\Delta^m_{i,\ell} = \theta^m_{(i),\ell} - \theta^m_{0,\ell}$. Large angular divergence between experts increases reliance on the pretrained anchor; small divergence allows greater shift toward the real-data distribution. Expert checkpoints are randomly refreshed each distillation iteration.

Multimodal distribution matching: For each image-text pair with embeddings $(z^v, z^t) \in \mathcal{S}^{d-1}$, MDM decomposes into cross-modal agreement and discrepancy directions: $$u = \text{normalize}(z^v + z^t), \quad g = \text{normalize}(z^v - z^t), \quad u \perp g.$$ Real and synthetic sets $\{u^r\}, \{u^s\}$ and $\{g^r\}, \{g^s\}$ are compared via geodesic kernel energy (GKE), which uses the geodesic Gaussian kernel $k_\text{geo}(a,b) = \exp(-\phi(a,b)^2 / 2\sigma^2)$ with $\phi(a,b) = \arccos\langle a,b\rangle$, and minimizes: $$\text{GKE}(A,B) = \left[\frac{1}{m^2}\sum_{i,i'} k_\text{geo}(a_i,a_{i'}) + \frac{1}{n^2}\sum_{j,j'} k_\text{geo}(b_j,b_{j'}) - \frac{2}{mn}\sum_{i,j} k_\text{geo}(a_i,b_j)\right]^{1/2}.$$ The total loss combines geodesic matching terms with a symmetric InfoNCE objective over synthetic pairs: $$\mathcal{L}_\text{MDM} = \mathcal{L}_\text{InfoNCE} + \lambda_\text{agr} \cdot \mathcal{L}_\text{agr} + \lambda_\text{dis} \cdot \mathcal{L}_\text{dis}.$$

Key Contributions¶

A geometry-aware multimodal distribution matching framework (MDM) that avoids trajectory replay, directly aligning joint image-text feature distributions on $\mathcal{S}^{d-1}$ at significantly lower distillation cost.
Joint-space cluster-based synthetic data initialization that seeds synthetic pairs from diverse semantic modes in the shared embedding space rather than per-modality or randomly.
Angle-guided weight-space interpolation of finetuned experts with a pretrained anchor, yielding an architecture-agnostic teacher that enhances cross-architecture generalization.
A geodesic kernel energy objective over decomposed cross-modal agreement ($u$) and discrepancy ($g$) directions, with a contrastive alignment term enforcing intra-synthetic image-text correspondence.

Results¶

All experiments use NFNet (vision) + BERT (text), evaluated on Flickr8k, Flickr30k, and MS COCO with 100/200/500 synthetic pairs (condensation rates as low as 0.8‰ for COCO at 100 pairs).

100 pairs, Flickr8k: MDM Mean = 21.9 vs. LoRS 19.4, MTT-VL 5.1, TESLAWBCE 9.7.
100 pairs, Flickr30k: MDM Mean = 26.4 vs. LoRS 27.4, MTT-VL 20.4 (competitive with LoRS).
100 pairs, COCO: MDM Mean = 10.3 vs. LoRS 9.4, MTT-VL 7.4 (outperforms LoRS by noticeable margin).
200 pairs, COCO: MDM Mean = 13.1 vs. LoRS 11.4, MTT-VL 9.2.
500 pairs, COCO: MDM Mean = 15.3 vs. LoRS 13.5, MTT-VL 15.1.
MDM consistently outperforms MTT-VL and TESLAWBCE across all datasets and budget regimes; competitive with or exceeding LoRS, especially on the largest (COCO) dataset.
Full dataset upper bound (COCO): Mean ~41.2.

Limitations¶

Performance on Flickr30k at small budgets is occasionally matched (not exceeded) by LoRS, indicating the advantage is not universal across all dataset scales.
Text embeddings (768-d vectors) rather than raw captions are distilled, sidestepping the challenge of optimizing discrete token sequences; this may limit applicability to generation tasks.
Evaluation is restricted to contrastive image-text retrieval with NFNet+BERT; generalization to other VLM architectures (e.g., transformer vision encoders, decoder-based models) is not empirically demonstrated in the main paper.
The merging ratio $\alpha = 0.5$ and $N = 2$ experts are fixed hyperparameters; sensitivity analysis appears only in supplementary material (not shown in extracted text).
Condensation rates remain very low (down to 0.8‰), so absolute retrieval numbers are well below full-dataset performance, limiting near-term practical deployment.

Relevance to Vision-Language Models¶

MDM directly addresses a scalability bottleneck for VLMs: as contrastive models like CLIP require massive image-text corpora, dataset distillation offers a path to preserve cross-modal alignment semantics in orders-of-magnitude smaller surrogates, enabling faster iteration and edge deployment. The geodesic kernel energy framework operates natively in the hyperspherical joint embedding space characteristic of contrastive VLMs, making the method well-suited to CLIP-style representation geometry. The cross-architecture generalization emphasis is particularly relevant as VLM encoders diversify (ViT variants, NFNet, etc.), since trajectory-based distillation tends to overfit to the training architecture. This work establishes a compute-efficient distribution matching baseline for multimodal distillation that could accelerate research into data-efficient VLM training and privacy-preserving data surrogates.