Skip to content

Think While You Map: Asynchronous Vision-Language Agents for Incremental 3D Scene Graphs

🕒 Published (v1): 2026-06-30 10:49 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; asynchronous incremental 3D scene graphs for embodied agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ThinkGraphs decouples lightweight incremental 3D mapping from heavyweight VLM semantic reasoning by running two background agents asynchronously, keeping the scene graph continuously queryable during exploration. A probabilistic voxel-based tracker maintains stable object identities while a Critic Agent performs semantic loop closure and a Description Agent attaches fine-grained visual attributes. The system outperforms batch-based open-vocabulary scene graph methods on segmentation and surpasses prior state-of-the-art on three visual grounding benchmarks by 15.3–18.8 A@0.25.

Problem

Existing open-vocabulary 3D scene graph methods either (a) defer all VLM/LLM enrichment to a post-reconstruction batch stage—leaving the graph unqueryable during exploration—or (b) update incrementally but restrict semantics to closed-set vocabularies or shallow CLIP/DINO feature matching. The core blocker for online enrichment is incremental tracking instability: greedy pairwise object association accumulates drift and fragmented tracks, making mid-exploration VLM calls wasteful and unreliable.

Method

ThinkGraphs is a three-component pipeline operating on sequential posed RGB-D frames \(\{(I_t^\text{rgb}, I_t^\text{depth}, \theta_t)\}_{t=1}^N\):

Frontend. Qwen3-VL-2B-Instruct generates noun prompts per frame; Grounded-SAM v2 (GroundingDINO + SAM2) produces 2D instance masks with grounded labels and confidence scores.

Backend (3D association). Adapts OpenVox's probabilistic voxel-consistency formulation. Each candidate track \(T_i\) scores a new observation \(O\) by: $\(S(T_i, O) = \lambda_\text{geo} \cdot S_\text{geo} + \lambda_\text{feat} \cdot S_\text{feat}\)$ with \(\lambda_\text{geo}=0.8\), \(\lambda_\text{feat}=0.2\). The geometric score $\(S_\text{geo}(T_i, O) = \frac{1}{|V(O)|}\sum_{v\in V(O)}\frac{c_{v,i}}{c_v}\)$ is a voxel-occupancy vote; the feature score is CLIP text-label cosine similarity. Tracks only become confirmed nodes after \(N_\text{conf}\) successful associations. Consensus labels use confidence-weighted histogram voting. A single representative CLIP visual embedding per track is chosen by text-guided scoring: \(\mathbf{f}_i^\star = \arg\max_n \, \mathbf{f}_n^\text{vis\top} \mathbf{f}_{\ell_i^\star}^\text{text}\). Spatial edges (left, under, etc.) are derived deterministically from axis-aligned bounding boxes.

Asynchronous VLM agents. Two background workers refine the graph without blocking the mapper: - Critic Agent (semantic loop closure): proposes duplicate-track merge candidates via a two-gate filter (3D IoU gate + SBERT semantic similarity gate), then dispatches the top-\(K_\text{critic}\) pairs to a VLM verifier using Set-of-Mark (SoM) image overlays. The VLM returns a structured decision \(D = \{a \in \{\text{Merge, Keep}\}, r \in \text{String}\}\). - Description Agent (attribute enrichment): a multi-target frame scheduler selects up to \(K_\text{desc}\) frames over a tumbling window by greedily maximizing a coverage utility $\(U(f) = \sum_{T\in\mathcal{T}_f} w_T \cdot \ln(1 + \gamma \hat{A}_{T,f})\)$ then submits multi-image SoM-annotated batches to a VLM that returns refined labels and per-attribute responses (material, color, texture, state), accumulated via confidence-weighted histograms.

Key Contributions

  • ThinkGraphs: first asynchronous architecture for open-vocabulary 3D scene graph construction, enabling continuous queryability during exploration.
  • Semantic loop closure via VLM: Critic Agent detects and merges fragmented object tracks mid-exploration, analogous to geometric loop closure in SLAM.
  • Multi-target frame scheduler: amortizes VLM cost by selecting frames that simultaneously cover multiple undescribed nodes, reducing redundant API calls.
  • Probabilistic voxel-consistency tracking adapted for open-vocabulary settings with CLIP text embeddings, stricter filtering, and confidence-weighted label voting, yielding more stable tracks than greedy pairwise matching.

Results

  • Semantic segmentation (Replica, 8 scenes): matches or outperforms batch-based open-vocabulary baselines (ConceptGraphs, BBQ, PoVo) on mAcc, mIoU, f-mIoU.
  • Semantic segmentation (ScanNet, 8 scenes): similarly matches or outperforms all batch-based comparisons.
  • Visual grounding (ScanRefer): surpasses prior state-of-the-art by 15.3 A@0.25.
  • Visual grounding (Sr3D+): surpasses prior state-of-the-art by 18.8 A@0.25.
  • Visual grounding (Nr3D): surpasses prior state-of-the-art (margin in same 15.3–18.8 A@0.25 range reported).
  • All grounding improvements are measured against batch-based open-vocabulary methods (ConceptGraphs, BBQ, PoVo, Open3DSG).

Limitations

  • The paper text is truncated before full ablation and runtime analysis tables are presented; concrete per-component latency breakdowns are not available in the provided excerpt.
  • The Critic Agent uses a periodic flush-and-dispatch scheme; very high-frequency scene changes or large track counts could saturate the candidate buffer or delay merge decisions.
  • Spatial edges are derived from axis-aligned bounding boxes with heuristic room-center anchoring, which may be unreliable in cluttered or non-Manhattan environments.
  • Evaluation is limited to eight ScanNet scenes (matching BBQ's protocol), so generalization to more diverse or outdoor scenes is unverified.
  • The Description Agent's multi-target scheduler uses a greedy frame-selection approximation; optimality of coverage is not guaranteed.
  • Semantic loop closure is merge-only—it cannot split incorrectly fused tracks.

Relevance to Vision-Language Models

ThinkGraphs is a concrete study of how to integrate VLMs asynchronously into a latency-sensitive perception pipeline, directly addressing the throughput vs. richness tradeoff that limits deploying large VLMs in streaming settings. The multi-target frame scheduler and SoM-grounded structured VLM outputs are transferable techniques for any embodied agent that must query VLMs selectively over a live spatial representation. For researchers tracking VLMs, the work demonstrates that fine-grained attribute extraction via a single multi-image VLM call substantially lifts 3D visual grounding accuracy (+15–18 A@0.25) over CLIP-embedding baselines, validating that VLM reasoning adds signal beyond retrieval-style similarity. The semantic loop closure mechanism—using a VLM as a verifier for ambiguous identity decisions—is also a novel and broadly applicable pattern for VLM-in-the-loop spatial reasoning.