Skip to content

ReScene: Structured Indoor Scene Reconstruction from Multi-View Captures

🕒 Published (v1): 2026-06-26 13:08 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ReScene reconstructs object-level, simulation-ready 3D indoor scenes from casually captured multi-view video by threading multi-view geometry as a unifying prior through view selection, instance registration, and relation-aware assembly. HierView replaces the largest-mask heuristic with semantic consistency and 3D coverage completeness for choosing reconstruction views, while Relation-Aware Assembly fuses multi-frame VLM predictions with geometric and room-shell priors into a confidence-weighted scene graph. It achieves 17% lower Chamfer Distance and 26% lower LPIPS than the strongest prior baseline, runs up to 10× faster than prior multi-view methods, and the reconstructed scenes enable an embodied VQA dataset that boosts open-source VLMs to compete with closed-source models on spatial reasoning.

Problem

Constructing simulation-ready 3D scenes (object-level structure, explicit inter-object relations, physical plausibility) from multi-view indoor captures is a bottleneck for Embodied AI. Existing approaches suffer from single-view bias in object reconstruction and produce layouts that are geometrically reasonable but physically inconsistent. The core bottleneck is cross-view relation fusion and physically plausible scene assembly, not single-object reconstruction quality.

Method

ReScene operates in three stages, all driven by multi-view geometry as a shared prior:

  1. HierView (Hierarchical View Selection): A three-stage filter for picking the best single view from which to reconstruct each object instance. Stage 1 discards candidates below visibility thresholds (mask pixel count, bounding-box ratio). Stage 2 scores semantic alignment via CLIP cosine similarity between the masked image crop and the category text label, discarding candidates below a threshold \(\tau_s\). Stage 3 ranks survivors by 3D coverage completeness: $\(\text{Comp}(v; i) = \frac{|\{p \in P_i : \pi_v(p) \in m_{i,v}\}|}{|P_i|}\)$ which measures what fraction of the object's 3D extent projects inside the mask, unlike mask area which conflates image occupancy with object coverage. The three criteria are applied hierarchically, not as a weighted sum.

  2. Instance Registration: Aligns each reconstructed mesh \(M_i\) to its semantic point cloud \(P_i\) via FPFH feature matching + RANSAC initialization followed by Sim(3) ICP refinement, with trust-region constraints on per-step rotation (\(d_{SO(3)}(\delta R_i^{(l)}, I) \leq \theta_{\text{step}}\)), accumulated rotation (\(d_{SO(3)}(\Delta R_i^{\text{ref}}, I) \leq \theta_{\text{tot}}\)), and scale bounds (\(s_i \in [s_{\min}, s_{\max}]\)).

  3. Relation-Aware Assembly: For each key frame, Qwen3-VL-32B-Instruct predicts (id, category, relation, parent) tuples for visible instances, validated for JSON correctness, ID coverage, acyclicity, and relation validity. Multi-frame fusion scores each candidate edge \(k = (c_k, r_k, p_k)\): $\(S_k = \alpha V_k + \beta G_k + \gamma \Phi_k, \quad \alpha + \beta + \gamma = 1\)$ where \(V_k\) is the voting ratio across frames, \(G_k\) is a geometric prior (floor contact, wall alignment, or parent surface accessibility), and \(\Phi_k\) is a category-conditioned relation prior. The resulting confidence-weighted scene graph is compiled into staged attachment operations (floor support, wall mounting, object support) solved via Lie-algebra scaling on \(\text{Sim}(3)\) propagated in topological order, followed by a non-penetration pass.

  4. Embodied VQA Generation: A Blender-based agent navigates the assembled scene collecting egocentric views; 5,688 QA pairs are automatically generated across object counting, distance judgment, relation reasoning, and viewpoint estimation with analytic ground truth.

Key Contributions

  • ReScene: an object-level 3D scene reconstruction framework unifying reconstruction, relation fusion, and physically grounded assembly with multi-view geometry as a consistent prior.
  • HierView: hierarchical view selector using semantic consistency (CLIP) and 3D coverage completeness, fixing the largest-mask heuristic failure mode.
  • Relation-Aware Assembly: confidence-weighted scene graph fusion from multi-frame VLM predictions with geometric/category/room-shell priors, compiled into staged attachment operations.
  • Scene-driven embodied VQA pipeline: automatically generates 5,688 QA pairs with analytic ground truth; fine-tuning on this data brings open-source VLMs to match closed-source models on spatial reasoning.

Results

  • Geometry: Chamfer Distance 8.40 (×10\(^{-2}\)), F@5 68.99, Normal Consistency 79.90 — best among all methods on ScanNet. 17% CD reduction over SimRecon (10.13).
  • Rendering: PSNR 19.63, SSIM 0.770, LPIPS 0.440 — 26% LPIPS reduction over strongest prior baseline.
  • Perceptual (GPT-5 scoring, 1–5): Visual 4.77, Completeness 4.80, Aesthetics 4.73 — substantially ahead of SimRecon (4.03/3.97/3.90).
  • Physical plausibility: Out-of-bound rate 0.90% vs. 9.88% (SimRecon) — over 10× reduction. Collision pairs 16.10 vs. 53.27 (SimRecon).
  • Runtime: 36.3 min per scene — up to 10× faster than DRAWER (375.5 min) and 7× faster than RICO (262.2 min).
  • Embodied VQA: Fine-tuned Qwen3-VL-2B-SFT substantially improves over base Qwen3-VL-2B (Avg 0.5479), approaching GPT-4o and Gemini-2.5-pro on spatial reasoning tasks.

Limitations

  • Still depends on per-instance single-view 3D reconstruction; HierView mitigates but does not eliminate view bias.
  • VLM relation predictions (Qwen3-VL-32B) can produce errors in highly cluttered or ambiguous scenes; the fusion stage can correct only some failure modes.
  • Evaluated on only 30 ScanNet scenes; generalizability to other datasets (e.g., Matterport3D, ARKitScenes) is not demonstrated.
  • Requires calibrated cameras and coarse planar surface estimation (floor + dominant walls) as input.
  • Collision rate (6.86%) is modestly lower than SimRecon (8.25%) rather than drastically so, and absolute collision count (16.10) reflects partly the moderate object count (22.17 vs. SimRecon's 36.43).

Relevance to Vision-Language Models

ReScene uses VLMs (Qwen3-VL-32B) as a core component for per-frame relation prediction, demonstrating that VLMs can serve as reliable 3D relational reasoners when their outputs are fused across multiple views with geometric priors. The embodied VQA results further show that structured 3D scene representations — with explicit object categories, poses, and support relations — provide effective supervision for improving VLM spatial reasoning, narrowing the gap between open-source and closed-source models.