OpenHype: Hyperbolic Embeddings for Hierarchical Open-Vocabulary Radiance Fields

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OpenHype embeds 3D scene hierarchies into a continuous hyperbolic latent space within a Neural Radiance Field, enabling open-vocabulary segmentation at arbitrary levels of semantic granularity via geodesic traversal. Unlike prior methods requiring multiple rendering passes or fixed discrete hierarchy levels, OpenHype uses a single rendering pass and continuously interpolates between part-level and object-level semantics. It outperforms state-of-the-art on both hierarchical and standard open-vocabulary 3D segmentation benchmarks.

Problem

Existing open-vocabulary NeRF methods (LERF, LangSplat, N2F2) either treat scenes as flat feature fields ignoring hierarchy, require multiple rendering passes at different scales (LERF, LangSplat), or fix the number of hierarchical levels to the embedding dimensionality (N2F2/Matryoshka). These constraints make them brittle for compositional queries (e.g., "door of a washing machine") and real-world environments where hierarchies are continuous and open-ended.

Method

Hyperbolic auto-encoder. A contrastive auto-encoder maps CLIP features (extracted from Semantic SAM mask crops per image) into the Lorentz hyperboloid \(\mathcal{L}^n\). The loss combines geodesic distance contrastive learning (\(L_d\)), exterior angle contrastive learning (\(L_a\)), and MSE reconstruction (\(L_r\)). Leaf nodes (object parts) land near the boundary; root nodes land near the origin; co-hierarchy items share a geodesic path.

NeRF training. A standard NeRF MLP is extended to additionally output a hyperbolic feature vector. Supervision uses a geodesic distance loss between predicted and target hyperbolic features, plus a norm-regularization term. Target features are extrapolated to the boundary to resolve per-image multi-view inconsistency.

Hierarchy traversal at inference. For a text query, \(T\) evenly-spaced points are sampled along the geodesic from the rendered boundary feature toward the hyperboloid origin. Each point is decoded to CLIP space and cosine-similarity-scored against the text prompt. A softmax-weighted sum aggregates the \(T\) scores into a final segmentation relevancy per pixel.

Key Contributions

• Continuous hyperbolic embedding of scene hierarchies in NeRF, replacing discrete scale levels with geodesic interpolation.
• Single-pass inference: hierarchy is traversed analytically in latent space rather than via repeated rendering at different scales.
• Contrastive auto-encoder training scheme combining geodesic distance and exterior angle losses to enforce hyperbolic hierarchical structure.
• Adaptation of the Search3D dataset to NeRF-based open-vocabulary hierarchical segmentation evaluation.
• Softmax-weighted geodesic path aggregation outperforming max, mean, and fixed weighted-sum baselines.

Results

Search3D (hierarchical benchmark, ScanNet++ scenes): - Object mIoU: 51.4 vs. LangSplat 41.6, OpenNeRF 28.3, LERF 25.4 - Object localization accuracy: 89.7 vs. LangSplat 68.9 - Part mIoU: 19.9 vs. LangSplat 11.0, OpenNeRF 9.7, LERF 6.2 - Part localization accuracy: 50.8 vs. LangSplat 19.3 - Average mIoU: 35.1 vs. LangSplat 25.8

LERF dataset (general open-vocabulary segmentation): - Average mIoU: 54.6 vs. N2F2 54.4, LangSplat 51.4, OpenNeRF 39.0, LERF 37.4

Ablation highlights: - Hyperbolic supervision + feature extrapolation (30.9 mIoU) vs. raw CLIP supervision (10.9 mIoU) - Feature dimension 32 optimal; dimension 3 (LangSplat-style) insufficient (19.6 mIoU) - Angle-only AE loss causes training collapse; combined distance + angle is necessary

Limitations

• Evaluated only on NeRF backbone; Gaussian Splatting adaptation is left as future work.
• Hierarchical supervision relies on Semantic SAM mask containment heuristic, which may fail in cluttered or occluded scenes.
• Evaluation hierarchy is only two levels deep (object / part); true open-vocabulary multi-depth hierarchies are not quantitatively assessed.
• N2F2 code unavailable at submission time; LERF dataset comparison uses N2F2's self-reported numbers.
• Computational cost of hyperbolic auto-encoder training and per-scene NeRF fitting is not reported relative to baselines.

Relevance to Agentic AI / LLM Agents

The abstract explicitly frames hierarchical 3D scene understanding as a prerequisite for autonomous agents, and OpenHype directly enables the kind of multi-granularity spatial reasoning (part → object → room) that embodied LLM agents need to ground natural language instructions in the physical world. The single-pass, continuous hierarchy traversal removes a practical bottleneck for real-time agent perception loops that cannot afford multiple rendering passes per query. The open-vocabulary interface — querying arbitrary noun phrases including compositional descriptions — aligns with how language-conditioned agents receive instructions from users or planners. For researchers tracking agentic AI, this work represents a foundational improvement in 3D perception modules that could be plugged into scene-graph builders or spatial memory systems used by robot or navigation agents.