Skip to content

Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation

🕒 Published (v1): 2026-06-28 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RAHA (Rank-Aware Hyperbolic Alignment) is a vision-language dataset distillation method that lifts multimodal representations to hyperbolic space and decomposes cross-modal correlation into a dominant range subspace and a residual subspace, enforcing geodesic alignment only where cross-modal semantics concentrate. By combining a hyperbolic contrastive loss with entropy-regularized optimal transport matching in each subspace, RAHA produces compact synthetic image-text pairs that better preserve hierarchical and ranking structure than Euclidean baselines.

Problem

Existing vision-language dataset distillation (VLDD) methods—whether trajectory-matching, generative, or distribution-statistics-based—apply uniform alignment pressure across all feature dimensions in Euclidean space. This is overly restrictive because image-text correlation is effectively low-rank: a compact shared subspace captures dominant semantics while the remaining directions carry modality-private variation. Uniformly aligning the residual directions under tight data budgets suppresses complementary information and degrades transfer robustness. Additionally, Euclidean geometry lacks inductive bias for the nested, hierarchical structure of cross-modal semantics.

Method

RAHA jointly optimizes two objectives over learnable synthetic image-text pairs \(\{(\tilde{x}_j, \tilde{E}_j)\}_{j=1}^M\):

  1. Hyperbolic contrastive loss (\(\mathcal{L}_\text{hITC}\)). Euclidean projected features \(z^v, z^t \in \mathbb{R}^d\) are scaled and lifted to the Lorentz hyperboloid via the exponential map \(\mathrm{Exp}_o^c\). Pairwise logits use negative geodesic distance \(\ell_{ij} = -d_c(h_i^v, h_j^t)/\tau\), and symmetric cross-entropy is applied over the synthetic batch only.

  2. Rank-aware relevance distillation (\(\mathcal{L}_\text{range} + \mathcal{L}_\text{residual}\)). Features are mapped back to hyperbolic tangent space via the logarithmic map. The real-batch cross-covariance \(C_\text{real} \in \mathbb{R}^{d \times d}\) is factored by SVD; adaptive rank \(k\) is chosen as the smallest integer capturing fraction \(\rho\) of total squared singular-value energy: $\(k = \min\!\left\{k' : \frac{\sum_{i=1}^{k'}\sigma_i^2}{\sum_{i=1}^{d}\sigma_i^2} \geq \rho\right\}.\)$ Range coordinates \(a^v = x^v U_k\) and residuals \(x^v_\text{res} = x^v - a^v U_k^\top\) form separate cross-modal similarity matrices \(G^\text{range}, G^\text{res}\). Row-wise softmax distributions over each matrix are matched from real to synthetic via entropy-regularized optimal transport (Sinkhorn-Knopp), with the coupling \(T\) kept stop-gradient. The range loss adds a one-sided energy regularizer preventing synthetic range coupling energy from collapsing below real energy. The residual loss adds a compression regularizer driving the ratio of residual-to-range energy toward zero.

Total objective: $\(\mathcal{L}_\text{total} = \mathcal{L}_\text{hITC} + \lambda_\text{range}\,\mathcal{L}_\text{range} + \lambda_\text{residual}\,\mathcal{L}_\text{residual}.\)$ Only the synthetic parameters are updated; encoders are held fixed and evolved on real data in alternating inner-loop steps.

Key Contributions

  • Rank-aware hyperbolic VLDD formulation: decomposes cross-modal correlation in hyperbolic tangent space via SVD into a range (dominant coupling) and residual subspace, with selective alignment enforced only in the range.
  • Range–residual relevance distillation objective: OT-based (Sinkhorn) matching of row-wise relevance distributions per subspace, with complementary regularizers—energy expansion for range and compression for residual.
  • Hyperbolic contrastive loss on synthetic pairs: replaces Euclidean InfoNCE with geodesic-distance logits on the Lorentz hyperboloid to embed hierarchical semantic structure.
  • Empirical evaluation with ablations on Flickr8k, Flickr30k, and MS COCO retrieval benchmarks, including cross-architecture transfer and robustness analyses.

Results

The results section is truncated in the provided text; no specific Recall@K numbers or direct comparisons to baselines (MTT-VL, LoRS, RepBlend, EDGE, CovMatch) are available from the supplied excerpt. The abstract states RAHA demonstrates "competitive cross-modal retrieval and improved transfer indicators under fixed budgets."

Limitations

  • Adaptive rank estimation via SVD is computed per real batch, adding per-iteration compute overhead compared to fixed-rank or covariance-only baselines.
  • Hyperbolic lifting introduces additional hyperparameters (curvature \(c\), scale \(s\)) that must be tuned alongside four loss weights (\(\lambda_\text{range}\), \(\lambda_\text{residual}\), \(\lambda_\text{comp}\), \(\rho\)).
  • The method distills text at the token-embedding level (not raw tokens), so the distilled captions are not human-readable or directly inspectable.
  • Results section was not available in the provided text, so claimed gains cannot be independently verified here.
  • Evaluated only on standard retrieval benchmarks; generalization to generation or grounding downstream tasks is not demonstrated.

Relevance to Vision-Language Models

RAHA directly targets a core bottleneck in scaling VLMs under data and compute constraints: replacing large paired corpora with compact, auditable synthetic surrogates without sacrificing contrastive alignment quality. The insight that image-text cross-covariance is low-rank motivates selective rather than uniform alignment, a principle applicable to VLM pre-training, fine-tuning, and continual learning regimes beyond distillation. The adoption of hyperbolic geometry aligns with a growing body of VLM work (e.g., MERU) showing that hierarchical visual-linguistic semantics are better modeled in negatively-curved spaces. For researchers tracking VLMs, this work contributes both a practical compression tool and a geometric analysis of cross-modal correlation structure that informs how contrastive objectives should be designed.