Skip to content

ZeroSplat: Generalized Referring Segmentation in 3D Gaussian Splatting

🕒 Published (v1): 2026-07-21 07:27 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; multi-target referring segmentation extending 3DGS to language-guided scene understanding

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ZeroSplat is a training-free, zero-feature framework for Generalized Referring 3D Gaussian Splatting Segmentation (GR3DGS), a new task requiring segmentation of an arbitrary number of targets (0, 1, or \(N\)) from free-form language queries. It lifts 2D VLM priors into 3D Gaussian space via multi-view geometric constraints, requiring no per-scene optimization and storing zero semantic features per Gaussian. On two new benchmarks (GR-LERF, GR-ScanNet), it outperforms all baselines by substantial margins while using roughly half the peak VRAM.

Problem

Existing Referring 3D Gaussian Splatting (R3DGS) methods are hard-constrained to single-target queries, failing when instructions correspond to zero or multiple objects—a common real-world condition. Two structural bottlenecks cause this: (1) they reason on 2D rendered pixels rather than discrete 3D Gaussian primitives, losing native 3D geometric structure; and (2) they require per-scene optimization to embed large semantic feature vectors into millions of Gaussians, incurring ~2h training time, ~3 GB extra storage, and ~20–28 GB peak VRAM per scene.

Method

ZeroSplat operates in three stages without modifying or storing any semantic parameters in the 3DGS representation:

Stage 1 — Geometry-Guided VLM Semantic Parsing. A compact set of \(K=30\) keyframes is greedily selected by maximizing marginal coverage of high-curvature voxels, where per-point surface variation is \(\sigma_\mathbf{p} = \lambda_3/(\lambda_1+\lambda_2+\lambda_3+\epsilon)\) and voxel weight is the max \(\sigma\) within the voxel. A VLM (Qwen3-VL-30B) then performs two-stage hierarchical reasoning: first extracting a concise semantic label set \(\mathcal{C}\) from the query, then predicting normalized 2D bounding boxes for each label per keyframe.

Stage 2 — 2D Mask Generation and 3D Lifting. SAM3 generates multi-view candidate masks per category. An adaptive view-selection strategy filters frames into high-confidence (\(\tau_{high}=0.6\)), base (\(\tau_{base}=0.3\)), and fallback tiers, selecting up to \(N_{target}=30\) views. Sub-optimal masks are merged if confidence exceeds \(\tau_{merge}=0.8\). Masks are lifted to 3D by accumulating per-Gaussian foreground/background scores via the volume rendering weight \(w(r, g_j) = T(r, g_j)\,\alpha(r, g_j)\) across all valid views. A cross-view background cropping step rejects Gaussians whose conflict ratio \(\rho_g = N_{conflict}(g)/(N_{val}(g)+\epsilon)\) exceeds \(\tau_{conf}=0.8\).

Stage 3 — Multi-View Verification and Spatial Refinement. VLM-predicted bounding boxes filter outlier Gaussians: a Gaussian is removed if it is visible in \(\geq\tau_{views}=8\) keyframes yet falls outside the box in \(>80\%\) of those views. Internal cavities from occlusions are recovered by KD-Tree KNN label diffusion: an unlabeled Gaussian is promoted to foreground when \(\geq\tau_{knn}=0.8\) of its \(k=40\) nearest neighbors are foreground.

Key Contributions

  • Introduces the GR3DGS task: generalized referring segmentation in 3DGS handling 0, 1, or \(N\) targets from free-form text.
  • Constructs two benchmarks: GR-LERF (pixel-level, 4 LERF scenes) and GR-ScanNet (point-level, 10 ScanNet indoor scenes), both with manual annotation and cross-validation.
  • Proposes ZeroSplat, a training-free, zero-feature pipeline that stores no semantic features per Gaussian and requires no per-scene optimization.
  • Achieves state-of-the-art on both benchmarks with ~10 GB peak VRAM (vs. 20–28 GB for most baselines) and zero additional training time.

Results

  • GR-LERF mean mIoU: ZeroSplat 50.8 vs. next-best LUDVIG 23.8, GS-Grouping 28.3, Occam's LGS 21.2, LangSplat 18.6.
  • GR-ScanNet mIoU: ZeroSplat 41.2 vs. InstanceGaussian 24.5, OpenGaussian 19.8, Dr.Splat 21.5, LUDVIG 15.1.
  • Per-scene LERF scores: Ramen 46.5, Teatime 56.3, Figurines 52.1, Waldo Kitchen 48.4.
  • Efficiency: 0 extra feature storage per Gaussian (vs. ~3 GB for most prior work), ~10 GB peak VRAM (vs. 20–28 GB), no training time (vs. ~1–2 h per scene).

Limitations

  • Depends on the quality of the underlying VLM (Qwen3-VL-30B); erroneous bounding boxes or label extraction propagate to 3D segmentation errors.
  • Introduces a large number of manually tuned thresholds (\(\tau_{high}\), \(\tau_{base}\), \(\tau_{conf}\), \(\tau_{box}\), \(\tau_{knn}\), etc.) whose sensitivity is not systematically analyzed in the provided text.
  • The adaptive fallback threshold \(\tau_{safe}\) differs between open-vocabulary and referring-expression datasets (0 vs. 0.15), indicating the pipeline is not fully hyperparameter-free across settings.
  • The paper text is truncated before completing Table 3 and the single-target benchmark comparison, so full ablation and standard R3DGS results are not available in this excerpt.
  • No-target (empty-mask) recall performance is not explicitly reported as a separate metric.

Relevance to Vision-Language Models

ZeroSplat demonstrates how frozen VLMs can be used as zero-shot semantic oracles for 3D scene understanding without any fine-tuning or feature distillation, making it directly relevant to researchers studying VLM generalization beyond 2D perception. The two-stage hierarchical VLM querying strategy—semantic label extraction followed by spatial grounding—illustrates a compositional prompting pattern applicable broadly to VLM-driven 3D pipelines. The work also validates that VLM 2D priors (bounding boxes and segmentation masks) can be sufficiently strong to replace expensive per-scene learned embeddings when fused with multi-view geometric constraints. For researchers tracking VLMs, this is an example of the "lift-and-verify" paradigm, where 2D VLM outputs are reprojected into 3D via rendering geometry rather than trained feature fields.