PlanarGS: High-Fidelity Indoor 3D Gaussian Splatting Guided by Vision-Language Planar Priors¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

PlanarGS is a 3D Gaussian Splatting framework for indoor scene reconstruction that addresses the geometric ambiguity of large, low-texture planar surfaces by injecting language-prompted planar priors and multi-view depth priors into the optimization. A pipeline called LP3 uses GroundedSAM (a VLM segmentation model) with cross-view fusion and geometric inspection to generate reliable per-plane masks, which then drive co-planarity and geometric constraints on the Gaussians. Results on Replica, ScanNet++, and MuSHRoom substantially exceed prior 3DGS-based surface reconstruction methods.

Problem¶

Standard 3DGS relies solely on photometric loss, which is ambiguous on large texture-less planar regions common in indoor environments (walls, floors, ceilings). Existing mitigations — monocular depth/normal priors or multi-view consistency — produce locally smooth but globally curved surfaces on planar regions rather than true planes. Prior plane detectors (e.g., Manhattan-SDF's MLP, heuristic detectors) are trained on limited datasets and fail to generalize to novel planar objects or scenes.

Method¶

LP3 Pipeline (Language-Prompted Planar Priors): Text prompts (e.g., "wall", "floor", "door") are fed into GroundingDINO + SAM (GroundedSAM) per frame to detect planar regions. Two refinement steps follow: 1. Cross-view fusion: back-projects plane masks from neighboring frames using DUSt3R depth priors to recover planes missed in individual frames. 2. Geometric inspection: converts prior depth to surface normals, applies K-means clustering on the normal map to split non-parallel co-detected planes, and uses plane-distance outlier detection to separate parallel planes at geometric edges.

Planar Prior Supervision (three components): - Plane-guided initialization: densely back-projects pixels in detected plane regions using DUSt3R depth to seed Gaussians where SfM yields sparse points. - Gaussian flattening: penalizes the minimum scale factor (L_s) so Gaussians collapse into planar discs; renders depth via plane-distance α-blending. - Co-planarity constraint (L_p): fits a plane Am to back-projected rendered depth points in each planar region via least squares, then enforces the rendered depth to match the fitted planar depth.

Geometric Prior Supervision: DUSt3R multi-view depth Dr (scale-shift aligned to SfM sparse depth) provides: (1) prior depth constraint L_rd on low-texture pixels masked by Canny edges; (2) prior normal constraint L_rn on plane regions only; (3) depth-normal consistency L_dn between rendered GS-normals and surface normals derived from rendered depth on low-texture pixels.

Total loss: L_RGB + L_s + λ1·L_dn + λ2·L_p + λ3·L_rd + λ4·L_rn (λ = 0.05, 0.5, 0.05, 0.2).

Key Contributions¶

LP3 pipeline combining VLM segmentation (GroundedSAM) with cross-view fusion and geometry-based inspection to produce reliable, per-plane masks for novel indoor scenes via swappable text prompts.
Co-planarity constraint that globally enforces flat geometry on detected planar regions by fitting 3D planes to rendered depth and penalizing deviation.
Integration of DUSt3R multi-view geometric priors (view-consistent depth + normals) as supervision signals, replacing less consistent monocular alternatives.
Plane-guided dense Gaussian initialization to compensate for SfM sparsity on texture-less surfaces.

Results¶

MuSHRoom (5 scenes): Chamfer Distance 4.49 cm vs. DN-Splatter 5.77, PGSR 10.51, 3DGS 11.92; F1 77.14% vs. DN-Splatter 61.86%, PGSR 59.11%; NC 83.35% vs. DN-Splatter 77.13%.
ScanNet++ (4 scenes): CD 3.66 cm vs. DN-Splatter 4.16, 2DGS 6.70; F1 82.78% vs. DN-Splatter 75.86%, 2DGS 53.46%; NC 90.52% vs. DN-Splatter 82.06%.
Replica (8 scenes): CD 4.13 cm vs. DN-Splatter 5.60, PGSR 8.56; F1 81.90% vs. DN-Splatter 68.12%, PGSR 62.98%; NC 89.88% vs. DN-Splatter 80.80%.
NVS on MuSHRoom: PSNR 26.42 dB (vs. 3DGS 25.79, best baseline); SSIM 0.8874.
Training time within one hour on a single RTX 3090, comparable to other 3DGS methods.
Ablation: removing co-planarity constraint degrades F1 from 87.32→73.96%; removing geometric prior degrades F1 to 64.11% and causes global plane tilting; replacing LP3 with raw GroundedSAM drops F1 to 80.22%.

Limitations¶

Planar priors only benefit detected planar regions; curved walls or organic surfaces receive no improvement from the co-planarity constraint.
Ineffective for outdoor or natural scenes lacking man-made planar structures.
DUSt3R depth is generated in groups with shared scale/shift, which may introduce group-boundary misalignments.
Performance is bounded by the VLM segmentation model's detection quality; prompt design requires scene-specific tuning (e.g., adding "blackboard" for classrooms).

Relevance to Vision-Language Models¶

PlanarGS demonstrates a concrete use case of open-vocabulary VLMs (GroundedSAM / GroundingDINO + SAM) as geometric scene understanding priors rather than as end-to-end perception systems — the VLM provides flexible, prompt-driven semantic segmentation that a downstream geometric pipeline refines and enforces. This pattern — freeze a VLM, extract structured semantic regions, compose with domain-specific geometric reasoning — is increasingly important for grounding VLM outputs in 3D spatial tasks such as reconstruction, SLAM, and embodied navigation. The robustness ablation (swapping GroundedSAM for YoloWorld+SAM with negligible accuracy drop) also illustrates that the LP3 pipeline is VLM-agnostic, suggesting a modular design philosophy relevant to any system that couples open-vocabulary models with geometric computation.