Is the Geometry Doing the Work? An Operating-Point Audit of Hierarchy in Hyperbolic Vision-Language Models¶

🕒 Published (v1): 2026-07-06 16:18 UTC · Source: Arxiv · Venue: TMLR · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper audits three families of hyperbolic vision-language models—MERU, HyCoCLIP, and PHyCLIP—and finds that none of them actually instantiate the radial/cone-based hyperbolic hierarchy their geometry is supposed to provide. All audited checkpoints operate in a near-Euclidean regime ($\sqrt{c}\rho \approx 0.2$–$0.3$, $H(u) \approx 1$), entailment cone machinery is inoperative or saturated, and hierarchy-looking benchmark gains are attributable to angular organization and compositional supervision rather than nonlocal curvature.

Problem¶

Hyperbolic VLMs (MERU, HyCoCLIP, PHyCLIP) claim that replacing Euclidean CLIP-style representations with negatively curved spaces encodes directed semantic abstraction via radial depth and entailment cones. However, existing evaluations conflate angular semantic similarity with radial hierarchical ordering, and strong downstream retrieval or taxonomy-correlation scores can arise without any active hyperbolic mechanism. No prior work provided necessary-condition diagnostics to test whether the geometry is operationally engaged.

Method¶

The authors develop a geometry audit battery applied to seven released checkpoints plus matched interventions trained from scratch on a fixed GRIT snapshot (13.1 M image–text pairs, 25.0 M box annotations, crawled 2026-04-13). Two evidence tracks are kept strictly separated: released checkpoints as historical artifacts, and within-snapshot causal comparisons.

Geometry diagnostics center on the dimensionless operating point $$u = \sqrt{c}\,\rho,$$ and the local distortion factor $H(u) = \sinh(u)/u$, which quantifies deviation from Euclidean behavior ($H(u) \approx 1$ is locally Euclidean; $H(u)$ substantially $> 1$ signals active hyperbolic nonlinearity). $\sqrt{c}\rho > 1$ is used as a conservative marker for the nonlocal regime.

Hierarchy diagnostics cover: (1) Radial ordering—whether child embeddings have larger radius than parents, tested against shuffle-null controls ($\Delta_\text{pair} = \text{Order}_\text{real} - \mathbb{E}_\text{shuffle}[\text{Order}_\text{shuffle}]$); (2) Cone activity—aperture distributions, active violation rates, and directed-pair containment (a parameter-free identity shows cones saturate whenever $\sqrt{c}\rho \leq 2K$); (3) Traversal robustness—monotonicity of nearest-caption retrieval along radial interpolation paths.

Evaluation diagnostics decompose taxonomy-distance correlations into angular and radial components via regression comparison ($\Delta R^2_\text{norm}$ from adding $|\rho_i - \rho_j|$ to a cosine-only model), and use shuffle-controlled directed norm tests to rule out confounds from marginal norm distributions.

A per-loss gradient decomposition is performed on two families to isolate the gradient contribution of the entailment term to curvature collapse.

Key Contributions¶

Empirically establish that all audited checkpoints (MERU, HyCoCLIP, PHyCLIP) operate near-Euclidean: median $H(u) \in [1.005, 1.009]$, 99th-percentile $\sqrt{c}\rho \leq 0.37$, and 0% of samples reach $\sqrt{c}\rho > 1$.
Demonstrate that standard downstream metrics (retrieval, zero-shot classification, compositionality) are decoupled from the entailment/cone mechanism: curvature collapse and cone saturation leave downstream numbers comparable under matched comparison.
Provide a gradient-level mechanistic explanation for curvature collapse: the entailment objective admits a low-curvature, wide-cone shortcut (reducing $c$ widens apertures and suppresses violations without learning order), and per-loss decomposition confirms the entailment gradient drives $c$ down more strongly than any depth signal counteracts.
Show that taxonomy-distance correlation gains are carried entirely by angular (cosine) distance, with negligible incremental explanatory power from radial norm differences ($\Delta R^2_\text{norm} \approx 0$).
Distill findings into a five-number geometry report (curvature, radius, $\sqrt{c}\rho$, $H(u)$, nonlocal-regime fraction) and enumerate design requirements for future hyperbolic VLMs.

Results¶

Operating point: All released checkpoints at $\sqrt{c}\rho \approx 0.2$–$0.3$, $H(u) \leq 1.023$ across all model families and modalities; 0% of samples in the nonlocal regime ($\sqrt{c}\rho > 1$).
Curvature unclamping: Relaxing the curvature floor from 0.1 to 0.001 moves scalar $c$ and norms but keeps $\sqrt{c}\rho$ in the same near-Euclidean band; no substantial degradation in downstream metrics across 3 seeds (ViT-B).
Cone machinery: Entailment apertures are saturated or misaligned across all families; directed-pair cone containment shows no signal above shuffle-null controls.
Radial ordering: Parent–child norm ordering shows no signal above shuffle-null at quantified sensitivity; one small exception on the models' native box→caption relation is non-operative.
Taxonomy decomposition: $\Delta R^2_\text{norm} \approx 0$; taxonomy correlation is explained by cosine distance alone.
Traversal: Graded traversal fails across all families under controlled readouts.
Entailment ablation: Removing the entailment term does not restore curvature; contrastive/alignment training alone also fails to hold $\sqrt{c}\rho$ up, confirming the shortcut is the dominant accelerator rather than the sole cause of collapse.

Limitations¶

Audit covers only three published families (MERU, HyCoCLIP, PHyCLIP); ARGENT (concurrent stabilization work) was not available and is not directly audited.
Current-GRIT snapshot (13.1 M pairs) is smaller than the originally documented corpus (20.5 M pairs) due to URL expiry; causal comparisons are valid within-snapshot but may not fully replicate original training scale.
Diagnostics are necessary-condition tests for the radial/cone mechanism—they cannot rule out alternative notions of hyperbolic hierarchy not based on radial depth or aperture containment.
The paper does not propose a corrected model, so whether design recommendations (curvature-identifying supervision, norm regularization, alternative entailment operators) suffice remains open.

Relevance to Vision-Language Models¶

This paper directly challenges a growing subfield of VLM research that posits hyperbolic geometry as a principled solution for encoding visual abstraction hierarchies (e.g., "golden retriever" → "dog" → "animal"). By showing that reported benchmark gains in MERU, HyCoCLIP, and PHyCLIP are attributable to angular organization and compositional supervision rather than nonlocal curvature or entailment cones, the paper reframes what these models have actually learned. The five-number geometry report and diagnostic battery provide concrete tools that future VLM work proposing hierarchical representations—hyperbolic or otherwise—should adopt before claiming geometric mechanisms are active. More broadly, it exemplifies a class of mechanistic auditing that is increasingly necessary as VLMs add architectural inductive biases whose advertised properties may not survive the training dynamics.