Modality Alignment across Trees on Heterogeneous Hyperbolic Manifolds¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Hyperbolic manifold alignment for hierarchical multimodal feature fusion in VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Existing VLMs align hierarchical textual features against a single visual feature, creating an asymmetric modality gap. This paper proposes Alignment across Trees (AT), which constructs symmetric hierarchical feature trees for both image and text, then aligns them on separate hyperbolic manifolds with an analytically optimal intermediate manifold. The method consistently outperforms prompt-learning baselines on taxonomic open-set classification.

Problem¶

In hierarchical semantic tasks (e.g., biological taxonomy), textual encoders naturally produce multi-level features (coarse-to-fine labels), but visual encoders output a single class token. This asymmetry forces alignment between a flat visual representation and a tree-structured textual hierarchy, causing suboptimal modality integration. Existing hyperbolic VLM methods also assume identical curvature for both modalities, ignoring their geometric differences.

Method¶

Semantic-aware visual feature extraction. Class tokens from $m$ intermediate ViT layers $\{h^{p_j}\}_{j=1}^m$ and the final layer $h^n$ are extracted. Each intermediate token $h^{p_j}$ is mapped to the final layer's representation space by forwarding through only linear projections, residual connections, and MLPs (self-attention disabled), preserving its coarse semantics. A cross-attention module then produces hierarchy-specific visual features: $$[v_1;\ldots;v_H] = \mathrm{Softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V_{\mathrm{attn}}$$ where textual features $\{t_i\}_{i=1}^H$ serve as queries and the mapped class tokens serve as keys/values. This yields symmetric feature trees $T_e = \{t_i\}_{i=1}^H$ and $V_e = \{v_i\}_{i=1}^H$.

Heterogeneous manifold alignment. Text and image trees are embedded into separate Lorentz-model hyperbolic manifolds with learnable curvatures $c_1$ and $c_2$ via the exponential map. To bridge them, an intermediate manifold $L^{c_3}$ is found by minimizing the sum of KL-divergence-based distances (Theorem 1): $$c_3^* = \arg\min_{c_3} \left[D_L(L^{c_1}, L^{c_3}) + D_L(L^{c_2}, L^{c_3})\right]$$ where $D_L(L^{c_1}, L^{c_3}) = \frac{-\sqrt{c_1} + 2\sqrt{c_3}\cosh[(\sqrt{c_3}-\sqrt{c_1})r]}{2\sqrt{c_1 c_3}}$. Proposition 2 proves this minimizer $c_3^* \in (\min\{c_1,c_2\}, \max\{c_1,c_2\})$ is unique; golden section search finds it efficiently.

Geometric alignment losses. On $L^{c_3}$, hyperbolic entailment cones force $v_i^{c_3}$ inside $\omega(t_i^{c_3})$ at each hierarchy level (cross-modal). On $L^{c_1}$ and $L^{c_2}$, in-modal constraints push finer features into coarser cones (within each modality). Curvature gradients $\partial c_3^*/\partial c_1$ and $\partial c_3^*/\partial c_2$ are computed via the implicit function theorem since the golden section search is non-differentiable.

Key Contributions¶

Alignment across Trees: first method to build and align symmetric hierarchical feature trees for both image and text modalities.
Semantic-aware visual feature extraction: cross-attention over intermediate ViT layer tokens guided by textual queries to produce coarse-to-fine visual features.
Heterogeneous manifold alignment: embeds each modality at its own learnable curvature; derives a provably unique optimal intermediate manifold with a closed-form KL distance; uses entailment cones for both cross-modal and in-modal hierarchical constraints.
Theoretical guarantees: existence and uniqueness proofs for the intermediate manifold (Propositions 1 and 2).

Results¶

Evaluated on Taxonomic Open-Set (TOS) classification across Cifar100, SUN, ImageNet, Rare Species using Leaf Accuracy (LA), Hierarchical Consistent Accuracy (HCA), Mean Treecut Accuracy (MTA); baselines are MaPLe, PromptSRC, and their +ProTeCt variants.
1-shot: HCA improves by up to 7.72% over strongest baseline (+ProTeCt) across datasets.
16-shot: up to 19.02% LA, 28.83% HCA, 8.48% MTA improvement over +ProTeCt.
Base-to-novel generalization (Cifar100): +1.38% LA, +5.66% HCA, +4.90% MTA vs. +ProTeCt on novel classes.
Consistent gains across all four datasets and all four evaluation settings (few-shot, base-to-base, base-to-novel, base-to-whole).
Ablations confirm both hyperbolic embedding and heterogeneous curvatures are necessary (Ours-Euc and Ours-HypV1 variants underperform full method).

Limitations¶

Evaluation is restricted to taxonomic open-set classification; applicability to standard VLM benchmarks (VQA, image captioning, retrieval) is not demonstrated.
Requires explicit hierarchical label structure at training time; not applicable to flat-label settings without taxonomic annotation.
Computational cost of constructing per-sample visual feature trees and solving the intermediate manifold at each training step is not analyzed.
The paper text is truncated before full Table 2 results; complete base-to-novel generalization numbers across all four datasets are not visible.

Relevance to Vision-Language Models¶

This work directly targets a fundamental structural asymmetry in CLIP-style VLM alignment: flat visual features vs. hierarchical textual concepts. By introducing hyperbolic geometry with modality-specific learnable curvatures, it advances the line of work (HyCon, Entailment CLIP) that argues Euclidean spaces are ill-suited for hierarchical multimodal semantics. The intermediate-manifold approach provides a theoretically grounded alternative to forcing shared embedding spaces, relevant to any VLM that must operate over structured concept hierarchies (e.g., medical ontologies, biodiversity classification). The prompt-learning integration means the technique is drop-in compatible with existing CLIP-based models such as MaPLe and PromptSRC.