Skip to content

SpatialLM: Training Large Language Models for Structured Indoor Modeling

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SpatialLM fine-tunes an open-source LLM (Qwen2.5-0.5B) on 3D point clouds to produce structured indoor scene descriptions—walls, doors, windows, and oriented 3D bounding boxes—encoded as Python dataclass scripts. The authors collect a large-scale synthetic dataset of 12,328 indoor scenes and conduct ablation studies on point cloud encoder design and training schedules. The resulting model achieves state-of-the-art on Structured3D layout estimation (F1 IoU2D@0.5 = 93.5) and competitive 3D object detection on ScanNet.

Problem

Existing 3D scene understanding methods rely on task-specific network designs or domain-specific token vocabularies (e.g., SceneScript), preventing direct reuse of modern foundation LLMs. Real-world RGBD datasets are too small (≤5k scenes) and lack joint layout + object annotations, making it infeasible to train LLMs end-to-end for structured indoor reconstruction.

Method

SpatialLM adopts the standard "Encoder-MLP-LLM" architecture: a Sonata (PTv3-variant, self-supervised, encoder-only) point cloud encoder maps N 6-dim (XYZRGB) points to K feature embeddings, a two-layer MLP projector aligns them to the LLM's embedding space, and Qwen2.5-0.5B generates structured outputs autoregressively.

Structured scene elements (Wall, Door, Window, Bbox) are represented as Python dataclass instances with integer-quantized geometric parameters. Ablations establish that (1) learned 3D CNN or Sonata encoders outperform naive mapping-based (DINOv2-projected) encoders by large margins, (2) resolution ≈2× (avg ~510 tokens/scene) is optimal, and (3) single-stage full-model fine-tuning outperforms multi-stage protocols—unlike image-based MLLMs where staged alignment helps.

Training follows a two-phase strategy: pre-train on the SpatialLM synthetic dataset, then fine-tune on the target benchmark.

Key Contributions

  • Python-script output representation: structured 3D scenes expressed as general-purpose Python dataclass code, enabling human readability, easy class extension, and exploitation of LLM coding priors.
  • Large-scale synthetic dataset: 12,328 professionally designed indoor scenes (54,778 rooms), 412,932 annotated object instances across 59 categories, photo-realistically rendered with RGBD and full 3D ground truth.
  • Empirical study on 3D-to-LLM alignment: systematic comparison of six point cloud encoders and six training schedule configurations under a unified MLLM framework.
  • State-of-the-art layout estimation: outperforms RoomFormer and SceneScript on Structured3D after two-phase training.
  • Zero-shot video generalization: qualitative demonstration of inference on noisy point clouds reconstructed by MASt3R-SLAM from monocular video without fine-tuning.

Results

  • Layout estimation (Structured3D benchmark):
  • SpatialLM (pre-train Ours → ft. Structured3D): F1 IoU2D@0.25 = 94.3, @0.5 = 93.5
  • vs. SceneScript: 90.4 / 89.2; vs. RoomFormer: 83.4 / 81.4
  • 3D object detection (ScanNet):
  • SpatialLM (pre-train Ours → ft. ScanNet): F1 IoU3D@0.25 = 65.6, @0.5 = 52.6
  • vs. V-DETR (specialist): 65.1 / 56.8; vs. SceneScript: 49.1 / 36.8
  • Training on either target dataset alone (without pre-training on SpatialLM dataset) degrades severely: ScanNet-only yields 2.9 / 0.7.
  • Best encoder is Sonata at 1× resolution: IoU2D@0.25 = 84.6, IoU3D@0.25 = 65.1; scaling to 2× improves detection (IoU3D@0.5: 49.4 → 61.1).

Limitations

  • Not a universal cross-source model: RGBD, monocular video, and LiDAR point clouds differ enough that per-dataset fine-tuning is still required for peak performance.
  • Impact on the base LLM's natural language and reasoning capabilities is unquantified (catastrophic forgetting not assessed).
  • Fixed 59-category vocabulary precludes open-vocabulary detection and 3D VQA.
  • Small objects (volume < 0.1 m³, e.g., "picture", "sink") and categories absent from the synthetic dataset ("showercurtain") remain challenging.

Relevance to Vision-Language Models

SpatialLM demonstrates that the standard MLLM "Encoder-MLP-LLM" paradigm extends cleanly to the 3D point cloud modality without task-specific decoders, a direct architectural parallel to 2D VLMs such as LLaVA. The finding that single-stage full fine-tuning (including the encoder) outperforms staged alignment diverges from 2D VLM practice and suggests that current 3D encoders are less versatile than CLIP/SigLIP, informing modality-agnostic VLM design. For VLM researchers, SpatialLM shows one path toward grounding language in metric 3D space—relevant to embodied AI, spatial reasoning benchmarks, and the broader question of how LLMs can represent structured geometric outputs rather than free-form text. The Python-script output format is a notable design choice for any multimodal LLM where structured prediction must interface with downstream symbolic systems.