Asynchronous Matching with Dynamic Sampling for Multimodal Dataset Distillation¶
🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link
Why this paper was selected
Asynchronous matching for multimodal dataset distillation; efficient VLM training
Ask a follow-up
Open an assistant pre-loaded with this paper's context.
TL;DR¶
AMD (Asynchronous Matching with Dynamic Sampling) is a multimodal dataset distillation framework that compresses large image-text datasets into small synthetic sets for efficient VLM training. It decouples the trajectory matching of image and text modalities to reflect their inherently different optimization dynamics, and uses semantics-aware K-means clustering to initialize synthetic samples. On Flickr30k (200 pairs), AMD improves IR@1/IR@5/IR@10 by +4.5%/+9.6%/+10.9% over the prior best (LoRS).
Problem¶
Multimodal Dataset Distillation (MDD) compresses image-text corpora into small synthetic datasets \(\mathcal{S}\) (\(M \ll N\)) so VLMs trained on \(\mathcal{S}\) approximate performance on the full set \(\mathcal{T}\). Existing MTT-based MDD methods (MTT-VL, LoRS) use synchronous trajectory matching — selecting the same training step \(t\) for both image and text expert trajectories — which ignores the empirically observable asynchrony between visual and language encoder optimization dynamics. A second gap is that without discrete class labels, random initialization of synthetic pairs produces semantically redundant samples with poor coverage of the continuous image-text semantic space.
Method¶
Asynchronous Trajectory Matching (AMD): Instead of forcing \(t_v = t_l\), AMD independently samples starting epochs \(t_v \in [0, R_V]\) and \(t_l \in [0, R_L]\) for image and text expert trajectories. The matching loss minimizes normalized \(\ell_2\) distances between student and expert parameters after \(N\) and \(M\) steps respectively:
Dynamic Sampling Ranges via MMD: The sampling bounds \(R_V\) and \(R_L\) are set automatically using Maximum Mean Discrepancy between consecutive-epoch parameter snapshots. The ratio \(\text{MMD}_{V,t} / \text{MMD}_{L,t}\) is compared against its median \(T_{\text{median}}\) across all epochs; text sampling is truncated before the crossing point (where text has stabilized) and visual sampling extends beyond it, without introducing new hyperparameters.
Semantics-Aware Prototype Mining (SPM): Joint features \(f_i = [v_i; l_i]\) are constructed by concatenating image and text encoder outputs for every real pair. K-means is run with \(K = B\) (budget size) clusters; the real sample closest to each centroid initializes the corresponding synthetic pair, replacing random initialization.
Key Contributions¶
- Empirical demonstration that image and text expert trajectories evolve asynchronously (text converges faster; visual encoder maintains high update magnitude throughout), motivating the decoupled design.
- Asynchronous trajectory matching loss \(\mathcal{L}_{AMD}\) with independently sampled start epochs \(t_v, t_l\) for each modality.
- Parameter-free dynamic sampling range determination based on per-epoch MMD ratio relative to its median.
- Semantics-Aware Prototype Mining via joint-space K-means clustering to improve initialization diversity and distribution coverage.
Results¶
Flickr30k (31,783 images; 5 captions each): - 200 pairs, I2T: IR@1 13.1% (+4.5 pp), IR@5 34.9% (+9.6 pp), IR@10 47.5% (+10.9 pp) vs. LoRS - 200 pairs, T2I: TR@1 16.9% (+2.4 pp), TR@5 42.3% (+3.6 pp), TR@10 56.2% (+2.8 pp) vs. LoRS - Performance advantage over LoRS in IR@1 scales with budget: +2.1 pp (99 pairs) → +4.5 pp (199) → +5.8 pp (499), suggesting synchronous methods hit a bottleneck at larger budgets - Full-dataset NFNet+BERT upper bound: IR@1=27.3, IR@5=57.1, IR@10=69.7
COCO (123,287 images; 5 captions each): - 200 pairs, I2T: IR@1 3.8% (+1.4 pp), IR@5 13.4% (+4.1 pp), IR@10 21.4% (+5.9 pp) vs. LoRS - AMD consistently outperforms all coreset selection baselines (Random, Herding, K-Center, Forgetting) and MTT-VL across all budget levels on both datasets
Ablation (Flickr30k, 200 pairs, IR@1): - Baseline (LoRS): 8.6% - +AMD alone: 12.1% (+3.5 pp) - +SPM alone: 9.1% (+0.5 pp) - +AMD+SPM (full): 13.1%
Limitations¶
- Evaluated only on cross-modal retrieval tasks (Flickr30k, COCO); generalization to other VLM objectives (e.g., VQA, captioning) is not demonstrated.
- Backbone fixed to NFNet+BERT; behavior with modern large-scale VLM architectures (CLIP, BLIP-2, LLaVA) is untested.
- BERT weights are frozen during training; only the appended linear layer is optimized, limiting the generality of the text trajectory analysis.
- SPM requires running K-means over the full dataset's joint feature space, which may be non-trivial at very large scale despite being described as low overhead.
- The dynamic sampling range derivation assumes that MMD under a linear kernel adequately captures modality-specific convergence; this may not hold for all architectures.
Relevance to Vision-Language Models¶
AMD directly targets the efficiency of VLM training by reducing the data requirement while preserving cross-modal alignment quality, which is central to the dataset distillation line of work for vision-language systems. The core insight — that image and text optimization trajectories are inherently asynchronous — is an architectural observation about VLMs generally, with implications for any trajectory-matching or curriculum-based training scheme. The Semantics-Aware Prototype Mining module addresses the structural difference between unimodal classification (discrete labels) and multimodal contrastive learning (continuous semantic space), a gap that will recur in any attempt to apply distillation or coreset methods to image-text data. For researchers tracking VLMs, AMD provides a practical compression path to reduce training cost of CLIP-style models while offering ablation evidence that naive synchronization of multimodal training dynamics is suboptimal.