Skip to content

Beyond Isolated Objects: Relationship-aware Open Vocabulary Scene Understanding via 3D Scene Graph Analysis

🕒 Published (v1): 2026-07-06 17:29 UTC · Source: Arxiv · link

Why this paper was selected

3D scene graph + VLM open-vocab scene understanding beyond isolated object lifting

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RelGraphOV is a framework for open-vocabulary 3D semantic segmentation that builds relationship-aware scene graphs from multi-view RGB-D observations using VLM reasoning, then refines CLIP/LSeg features via a novel dual-stream graph attention network. It addresses the blind spot of existing methods that treat 3D objects in isolation, ignoring inter-object context. The method achieves state-of-the-art performance on ScanNetV2, ScanNet200, and demonstrates zero-shot generalization to ScanNet++ and Replica.

Problem

Existing open-vocabulary 3D segmentation methods (both feature-lifting and instance-proposal paradigms) rely on context-independent semantic representations: each object's features are computed in isolation, making fine-grained disambiguation hard (e.g., "curtain" vs. "shower curtain" without knowledge of nearby bathtub/toilet). Naively incorporating relational context via graph neural networks causes feature interference, where geometric noise from dense LSeg features contaminates the CLIP semantic embedding space during message passing. Additionally, large-scale relationship annotations for 3D scenes are scarce.

Method

The pipeline has four stages:

  1. Scene graph construction. 3D class-agnostic object proposals (from MaskClustering) become graph nodes. A composite viewpoint scoring function \(S(I_k, W) = W \cdot \mathbf{S}\) (weighing completeness, projected area, centrality, and sharpness) selects keyframes per node. Candidate edges are initialized by spatial proximity (3-NN or bounding-box overlap). A VLM (Qwen-VL-Max) is then applied with Set-of-Mark visual prompts and chain-of-thought reasoning to validate each candidate edge and generate asymmetric SVO-triplet descriptions; invalid ("none") edges are pruned. Edge features \(\mathbf{F}_e\) are built by CLIP-encoding the subject, verb, and object of each SVO triplet and concatenating them.

  2. Node annotation. Two complementary keyframes per node are processed by the Describe Anything Model to produce view-specific captions, which a lightweight LLM adjudicates for category consistency. The final annotation \(f^i_{anno}\) is the CLIP text encoding of the resolved description.

  3. Adaptive Gated Dual-Stream Contextual GAT. Two decoupled streams are maintained per layer:

  4. Main stream (\(h_m\)): initialized with LSeg features + geometry, aggregated with global context via cross-attention to a virtual super-node, then updated via edge-feature-guided message passing (edge feature \(\mathbf{F}_e\) injected into the attention Key).
  5. Auxiliary stream (\(h_a\)): initialized with CLIP features, independently propagated to preserve high-level open-vocabulary priors.

After each layer, a learned gate \(\text{Gate} = \sigma(\text{MLP}([\hat{h}^{(l)}_{m,i} \| \hat{h}^{(l)}_{a,i}]))\) performs unilateral injection: $\(h^{(l+1)}_{m,i} = \hat{h}^{(l)}_{m,i} + \alpha \cdot \text{Gate} \odot \hat{h}^{(l)}_{a,i},\)$ while the auxiliary stream is kept unmodified (\(h^{(l+1)}_{a,i} = \hat{h}^{(l)}_{a,i}\)).

  1. Hierarchical contrastive training. A loss \(\mathcal{L}_{total} = \lambda_{\text{hie}}\mathcal{L}_{\text{hie}} + \lambda_{\text{reg}}\mathcal{L}_{\text{reg}} + \lambda_{\text{aux}}\mathcal{L}_{\text{aux}}\) enforces: (i) self-consistency with VLM annotations, (ii) intra-class consistency via an inferred positive mask \(M_{pos}\) thresholded by cosine similarity \(\tau\), (iii) inter-class separation; dual-alignment terms prevent catastrophic forgetting of the original LSeg/CLIP priors.

Key Contributions

  • RelGraphOV: first framework to use automatically-constructed 3D scene graphs as contextual priors for open-vocabulary 3D segmentation (not for graph prediction itself).
  • Annotation-free scene graph construction via VLM chain-of-thought edge reasoning and multi-view VLM adjudication, eliminating need for manual relationship labels.
  • Adaptive Gated Dual-Stream Contextual GAT that prevents cross-modal feature interference while enabling selective semantic borrowing from CLIP representations.
  • Hierarchical contrastive loss that enforces both instance-level consistency and category-level discrimination against VLM-generated pseudo-labels.
  • State-of-the-art results on ScanNetV2, ScanNet200, and zero-shot transfer to ScanNet++ (100 classes) and Replica (51 classes).

Results

The paper text provided is truncated before the quantitative results tables. Based on what is present: - The method is evaluated on ScanNetV2 (20 classes, 312 val scenes), ScanNet200 (200 classes), ScanNet++ (zero-shot, 100 classes), and Replica (zero-shot, 51 classes). - Authors claim RelGraphOV "consistently outperforms existing methods and achieves state-of-the-art performance" across all four benchmarks. - Specific mIoU numbers and per-baseline comparisons are not available in the provided excerpt.

Limitations

  • Scene graph construction relies on external VLMs (Qwen-VL-Max) at inference-time graph building, introducing latency and API dependency.
  • Proposal quality is inherited from upstream modules (e.g., MaskClustering); errors there propagate directly.
  • Background structures (floors, walls, ceilings) are pruned from the graph, which may drop relationships involving structural context.
  • Evaluated only on indoor scenes; generalization to outdoor or dynamic scenes is untested.
  • Quantitative cross-dataset gains vs. strong baselines are not visible in the provided excerpt, making it difficult to assess the magnitude of improvement.

Relevance to Vision-Language Models

RelGraphOV directly exploits CLIP as a dual role player: as a semantic feature extractor (instance-level embeddings) and as a text encoder for relationship edge features derived from VLM-generated SVO triplets, making the scene graph itself a CLIP-aligned structure. The framework shows a concrete recipe for how VLMs (both CLIP-style and generative, e.g., Qwen-VL) can be composed at inference time to build structured world representations—scene graphs—that then guide downstream feature learning without requiring human annotation. For VLM researchers, the Adaptive Gated Dual-Stream GAT is a reusable architectural pattern for fusing dense vision features with sparse VLM semantic embeddings while avoiding representation collapse. The hierarchical contrastive loss using VLM-generated pseudo-labels is also relevant to the growing literature on distilling VLM knowledge into structured 3D representations.