Uncertainty-guided Compositional Alignment with Part-to-Whole Semantic Representativeness in Hyperbolic Vision-Language Models

🕒 Published (v1): 2026-03-23 14:41 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; hyperbolic embeddings capture part-to-whole hierarchy in VL alignment

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

UNCHA (UNcertainty-guided Compositional Hyperbolic Alignment) is a hyperbolic VLM training framework that models the varying degree to which part images semantically represent their whole scene. It assigns lower hyperbolic uncertainty to more representative parts and higher uncertainty to less representative ones, then incorporates these uncertainty estimates into both the contrastive and entailment losses. UNCHA achieves state-of-the-art on zero-shot classification, retrieval, and multi-label classification benchmarks.

Problem

Prior hyperbolic VLMs (MERU, HyCoCLIP, ATMG) model part-whole entailment but treat all part images as equally representative of the whole scene. This homogeneous treatment causes semantically peripheral parts to contaminate alignment signals, leading to embedding collapse around narrow regions, degraded multi-object alignment, and inefficient use of hyperbolic embedding space.

Method

UNCHA operates on the Lorentz (hyperboloid) model \(\mathbb{L}^n\) with curvature \(-\kappa\) and uses three interacting components:

Uncertainty model. Hyperbolic radius (geodesic distance from origin \(o\)) is monotonically related to specificity. UNCHA defines a smooth, differentiable per-embedding uncertainty: $\(u(\mathbf{x}) = \log\!\left(1 + \exp\!\left(-\|\mathbf{x}\|_2\right)\right)\)$ so points near the origin (abstract/representative) get low \(u\) and peripheral points (specific/less representative) get high \(u\). Empirically, \(u\) correlates with semantic representativeness at \(r = -0.739\) on an ImageNet subset.

Uncertainty-guided contrastive loss. The global-local contrastive loss uses adaptive per-part temperatures \(\tau^I_{\text{un},i}\) and \(\tau^T_{\text{un},i}\) that scale with each part's uncertainty. Higher-uncertainty parts exert a weaker pull in the contrastive objective, preventing low-quality regions from disrupting alignment: $\(\mathcal{L}^\text{un}_\text{con} = L^*_c(\mathbf{i}^\text{part}, \mathbf{t}; \tau^I_\text{un}) + L^*_c(\mathbf{t}^\text{part}, \mathbf{i}; \tau^T_\text{un}) + \text{global} + \text{local terms}\)$

Entailment loss with uncertainty calibration. The hinge-based entailment loss \(\mathcal{L}_\text{orig} = \max(0, \phi(\mathbf{p},\mathbf{q}) - \eta\omega(\mathbf{p}))\) is extended to a Leaky-ReLU-like formulation: $\(L^*_\text{ent}(\mathbf{p}, \mathbf{q}) = \max(0, \phi(\mathbf{p},\mathbf{q}) - \eta\omega(\mathbf{p})) + \alpha\,\phi(\mathbf{p},\mathbf{q})\)$ which preserves gradient flow even when \(q\) is inside \(p\)'s entailment cone. An uncertainty calibration loss then uses a stop-gradient entailment signal to update \(u(\mathbf{p})\), with an entropy regularizer \(\mathcal{H}(\tilde{u}(\mathbf{p})) = -\sum_i \tilde{u}(\mathbf{p}_i)\log\tilde{u}(\mathbf{p}_i)\) that prevents uncertainty from collapsing to a uniform or constant distribution. Training uses the GRIT dataset (20.5M grounded image-text pairs, 35.9M part annotations), batch size 768, 500K iterations.

Key Contributions

• UNCHA framework: first hyperbolic VLM that explicitly models per-part semantic representativeness for the whole scene
• Uncertainty model grounded in hyperbolic radius, quantifying how representative each part is
• Uncertainty-guided contrastive loss with adaptive per-part temperature modulation
• Piecewise-continuous (Leaky-ReLU-like) entailment loss that maintains gradient flow inside the cone
• Entropy-regularized uncertainty calibration loss preventing embedding collapse

Results

• Zero-shot classification (ViT-B/16, 16 benchmarks): UNCHA 48.8 / 90.4 vs. HyCoCLIP 45.8 / 88.8, CLIP 40.6 / 78.9, MERU 40.1 / 78.6, ATMG 34.3 / 68.8 (first two reported metrics from Tab. 1)
• Zero-shot retrieval: Evaluated on COCO val and Flickr30K test; UNCHA shows steady improvement over all baselines
• Hierarchical classification: Consistently strong on hierarchy-aware metrics from HyCoCLIP benchmark
• Uncertainty correlation: Part-to-whole semantic similarity vs. uncertainty gives \(r = -0.739\) on ImageNet subset, validating the uncertainty model
• Results hold across ViT-S and ViT-B encoders

Limitations

• Requires a large dataset with explicit grounded part annotations (GRIT); not straightforwardly applicable to annotation-free corpora
• Introduces multiple hyperparameters (\(\lambda_1\), \(\lambda_2\), \(\lambda_\text{ent}\), \(\alpha\), three temperature parameters) requiring careful tuning
• Part images are randomly cropped, a coarse proxy for semantically meaningful parts
• The text is truncated before full ablation and multi-label classification results are reported

Relevance to Vision-Language Models

UNCHA directly addresses a known failure mode of contrastive VLMs (CLIP-style) in multi-object and compositional scenes by introducing principled uncertainty weighting in hyperbolic space, complementing the line of work on hyperbolic VLMs (MERU → HyCoCLIP → UNCHA). The entropy-regularized uncertainty calibration is a general mechanism for preventing embedding collapse in non-Euclidean spaces, transferable to other hierarchical VLM training schemes. For practitioners building retrieval or classification systems requiring fine-grained compositional understanding, UNCHA's part-to-whole framework offers a geometry-aware alternative to Euclidean approaches.