Skip to content

Distributional Vision-Language Alignment by Cauchy-Schwarz Divergence

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; Cauchy-Schwarz divergence for distributional VL alignment beyond pairwise InfoNCE

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CS-Aligner is a vision-language alignment framework that augments InfoNCE with Cauchy-Schwarz (CS) divergence to perform both pairwise and distributional alignment between image and text modalities. The method resolves the inherent alignment–uniformity conflict in InfoNCE by decoupling per-modality uniformity from cross-modal alignment. It supports unpaired data and token-level fine-grained alignment as natural extensions of the distributional objective.

Problem

CLIP-style contrastive learning with InfoNCE operates purely at the pairwise sample level: it maximizes mutual information between matched pairs but does not constrain the marginal distributions \(p(x)\) and \(p(y)\) to be close, producing a persistent modality gap in the shared embedding space. Formally, high mutual information is compatible with high distributional divergence (Fig. 2a), so the gap can persist even after pairwise alignment converges. Additionally, InfoNCE's uniformity term \(L_\text{uniform}\) conflicts with its alignment term \(L_\text{align}\) in the multimodal setting—uniformity across both modalities simultaneously opposes cross-modal pull—causing local minima and suboptimal alignment.

Method

CS-Aligner minimizes a joint objective over lightweight adapters (or LoRA) on top of frozen pretrained encoders:

\[\min -I(x; y) + \lambda D_\text{CS}(p(x), p(y))\]

CS divergence between image distribution \(p(x)\) and text distribution \(p(y)\) is:

\[D_\text{CS}(p; q) = -\log\frac{\left(\int p(\omega)q(\omega)d\omega\right)^2}{\int p(\omega)^2 d\omega \int q(\omega)^2 d\omega}\]

Estimated non-parametrically via KDE with a Gaussian kernel \(\kappa_\sigma(x,y) = \exp(-\|x-y\|_2^2/2\sigma^2)\):

\[\hat{D}_\text{CS} = \log\frac{1}{M^2}\sum_{i,j}\kappa(x_i,x_j) + \log\frac{1}{N^2}\sum_{i,j}\kappa(y_i,y_j) - 2\log\frac{1}{MN}\sum_{i,j}\kappa(x_i,y_j)\]

When \(\lambda=1\) and this is combined with InfoNCE's alignment–uniformity decomposition (Eq. 10), the resulting objective promotes cross-modal pair alignment and per-modality uniformity independently, eliminating the conflict. CS divergence also admits a RKHS interpretation as cosine similarity between mean embeddings \(\hat{D}_\text{CS} = -2\log\text{sim}(\mu_x, \mu_y)\), making it structurally complementary to InfoNCE's pairwise cosine similarity.

Two extensions leverage the distributional (non-pairwise) nature of the CS estimator: 1. Unpaired alignment: images with multiple captions or entirely unpaired image/text sets, since \(M \neq N\) is allowed. 2. Token-level alignment: per-sample CS divergence between \(V\) vision tokens and \(L\) text tokens (unequal counts, so InfoNCE is inapplicable), summed as \(L_\text{token} = \frac{1}{B}\sum_i \hat{D}_\text{CS}(p(x_i); p(y_i))\).

Key Contributions

  • Formal analysis showing InfoNCE's alignment and uniformity terms conflict in the multimodal (cross-modal) setting, with Taylor-expansion derivation (Eqs. 4–5).
  • CS-Aligner: a distributional alignment framework combining CS divergence with InfoNCE, shown to resolve the alignment–uniformity conflict while remaining compatible with contrastive learning.
  • Non-parametric KDE-based CS divergence estimator that is symmetric, differentiable, and assumption-free about distribution shape.
  • Extension to unpaired data (multi-caption and fully unpaired) without additional computation in the divergence term.
  • Token-level intra-sample alignment via CS divergence, enabling fine-grained vision–language correspondence beyond CLS-token matching.
  • Parameter-efficient implementation via adapters and LoRA on top of frozen CLIP/LLM encoders.

Results

Text-to-image generation (FID ↓, MSCOCO 30K eval): - CS-Aligner + Kandinsky decoder (0.08M COCO): FID 12.62 vs. Eclipse 16.53, IB 150.52 (small-scale); vs. Kandinsky 20.48, Karlo 20.64, DALL-E2 10.65, SD v2.1 14.51 (large-scale, 115–2000M data) - CS-Aligner + Karlo decoder: FID 11.27 vs. Eclipse 23.67 - CS-Aligner + SD-unCLIP decoder: FID 10.88 - With token alignment: FID improves from 12.62 → 12.14 - LoRA variant (6M params): FID 13.52 vs. adapter (34M params): 12.62

Multi-dataset training (FID, CS-Aligner vs. Eclipse): - CC3M: 22.88 vs. 26.73; CC12M: 22.72 vs. 26.98; LAION-HighRes-5M: 14.79 vs. 19.16

TSNE visualization: CS-Aligner eliminates the visible modality gap present in CLIP features and reduces pairwise \(\ell_2\) distance between aligned image-text features.

Limitations

  • Experiments focus on T2I generation and retrieval; broader evaluation on VQA, captioning, or zero-shot classification is left to appendices and may be limited in scope.
  • CS divergence estimation via KDE scales quadratically in batch size (\(O(M^2 + N^2 + MN)\) kernel evaluations); the paper references a complexity analysis in Appendix E but does not detail mitigation strategies in the main text.
  • The method is validated primarily as a post-hoc adapter on top of CLIP, not as a from-scratch pretraining objective, so scalability to large-scale pretraining remains undemonstrated.
  • Token-level alignment improvement is modest (FID 12.62 → 12.14) and only qualitatively illustrated.
  • Unpaired alignment experiments (Fig. 5b) are shown but not numerically ablated in the main paper.

Relevance to Vision-Language Models

CS-Aligner directly targets the modality gap—a well-documented failure mode of CLIP-based VLMs—by providing a theoretically grounded fix to InfoNCE's alignment–uniformity conflict, which is a root cause of suboptimal cross-modal representation in the backbone models underlying most VLMs. The KDE-based distributional objective is a lightweight, plug-in component applicable to any pretrained encoder pair, making it immediately relevant for fine-tuning or adapting existing VLMs (CLIP, LLaVA-style architectures, unCLIP pipelines). The token-level CS alignment extension is particularly relevant for researchers working on dense cross-modal alignment tasks (grounding, retrieval, referring expression comprehension) where CLS-token-only matching is insufficient. More broadly, the framework's tolerance for unpaired and noisily paired data addresses a key scalability bottleneck in VLM training data curation.