Training-Free Open-Vocabulary 3D Point-Cloud Segmentation on the Generalized Few-Shot Benchmark¶

🕒 Published (v1): 2026-07-16 16:03 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper proposes a fully training-free, open-vocabulary method for Generalized Few-Shot 3D Point-Cloud Segmentation (GFS-PCS) that pairs a frozen 3D vision-language model (RegionPLC) with a frozen promptable 2D segmenter (SAM3) lifted to 3D via multi-view projection. The key insight is that cross-view consistency—requiring a point to be labeled as novel only when a sufficient fraction of views agree—suppresses 2D false positives without sacrificing base-class accuracy. The method requires no 3D labels, no training, and not even the few-shot support samples, yet recovers one-third of the gap to the fully trained state of the art on ScanNet200 and nearly doubles novel mIoU on ScanNet++.

Problem¶

State-of-the-art GFS-PCS methods (e.g., GFS-VL) reconcile a dense but noisy 3D vision-language prior with sparse few-shot support, but require (1) base 3D ground-truth labels to pretrain a 3D segmentor, (2) per-episode fine-tuning on the support, and (3) the support annotations themselves. This makes deployment to new label sets and environments expensive. The paper asks how far the dense-sparse reconciliation can proceed with zero supervision: no training, no 3D labels, and no few-shot support.

Method¶

The pipeline has three frozen stages over a scene point cloud $P = \{p_i\}_{i=1}^N$ and its posed RGB frames $\{(I_t, T_t, K)\}$:

Dense branch: Frozen RegionPLC (3D-VLM with SparseUNet32 backbone) produces per-point zero-shot logits, yielding a dense label $d_i = \arg\max_c f_d(P)_{i,c}$ and confidence $\kappa_i = \max_c \operatorname{softmax}(f_d(P)_i)_c$.
Sparse branch (SAM3-PCS lifted): For each novel concept $n$ and frame $I_t$, frozen SAM3 concept segmentation yields a 2D mask $M_t^n$. Each point is projected into every frame via $[u,v,z]^\top \sim K T_t^{-1}[p_i; 1]$, and cross-view agreement is computed as: $$s_i(n) = \frac{\#\{t : i \text{ visible in } t,\; M_t^n(u,v)=1\}}{\#\{t : i \text{ visible in } t\}}$$ with $s_i = \max_n s_i(n)$ and $\hat{n}_i = \arg\max_n s_i(n)$.
Reconciliation: The dense label is overwritten by the sparse novel label only when agreement clears a single threshold $\tau$: $$y_i = \begin{cases} \hat{n}_i & s_i \ge \tau \\ d_i & \text{otherwise} \end{cases}$$ $\tau$ is fixed conservatively per benchmark ($\tau=0.5$ on ScanNet200; $\tau=0.15$ on ScanNet++) and never tuned to maximize the reported metric. No learned parameters exist beyond this scalar.

The paper also ablates injecting few-shot support as a fusion gate and as a prototypical dense classifier, finding both provide no improvement (and the classifier is significantly detrimental), formally licensing the support-free, open-vocabulary setting.

Key Contributions¶

A fully training-free, open-vocabulary GFS-PCS pipeline using no 3D labels, no fine-tuning, and no few-shot support—strictly less supervision than all prior GFS-PCS work.
Cross-view consistency as the reconciliation rule: a principled, parameter-free mechanism that suppresses single-view 2D false positives and lifts novel accuracy while preserving base accuracy, where naive sparse-over-dense overwrite trades one for the other.
Empirical demonstration that view-count-visibility gating, dense-confidence gating ($\kappa_i$), and few-shot prototype classifiers each add nothing over consistency alone—and the prototype classifier is significantly negative—justifying the support-free framing.
Establishment of the training-free, support-free floor for the GFS-PCS benchmark on ScanNet200 and ScanNet++, quantifying the remaining gap to supervised methods.

Results¶

ScanNet200 GFS-PCS (312 scenes, 12 base / 45 novel):
Dense-only baseline (RegionPLC zero-shot): mIoU-N=23.78, mIoU-B=46.93, HM=31.56
Naive sparse-over-dense: mIoU-N=24.51, mIoU-B=44.39, HM=31.58
Ours (cross-view consistency): mIoU-N=26.42 (+2.6 over dense-only), mIoU-B=46.44, HM=33.68
Upper reference — trained GFS-VL (5-shot): mIoU-N=31.67, HM=43.12
Novel gain is statistically significant: 95% CI [+2.13, +3.23] over 10⁴ bootstrap resamples
Recovers 33% of the novel-class gap to trained GFS-VL
ScanNet++ (50 scenes, 12 base / 18 novel):
Dense-only: mIoU-N=16.2, HM=21.5
Ours: mIoU-N=31.9 (+15.7), HM=31.1 at a cost of −1.7 mIoU-B
Novel mIoU nearly doubles; harmonic mean rises by +9.6
All experiments run on a single NVIDIA T4; nothing is trained.

Limitations¶

The single threshold $\tau$ is calibrated per benchmark because $s_i$ is a fraction of agreeing views whose numeric scale depends on capture density (dense video vs. sparse wide-baseline DSLR); raw transfer of $\tau$ across datasets is not demonstrated.
ScanNet++ has no separate development split for $\tau$ selection; the threshold is committed from a single in-set probe scene, creating a mild but acknowledged leakage risk.
Novel class recognition depends entirely on the quality of bare class-name prompts to SAM3; no synonym expansion or prompt engineering is used, creating sensitivity to official class string formulations.
The dense prior (RegionPLC) is weak on novel classes in ScanNet++; while this enlarges the relative gain, the absolute novel mIoU (31.9) remains far below what a stronger dense prior might enable—the paper leaves such substitution to future work.
Only 48 of 50 ScanNet++ validation scenes have posed DSLR imagery; the remaining 2 fall back to the dense prediction.
The method is evaluated only on indoor benchmarks (ScanNet200, ScanNet++); generalization to outdoor scenes (e.g., nuScenes) is not shown.

Relevance to Vision-Language Models¶

This work directly probes the zero-shot transfer capability of 3D vision-language models (RegionPLC) and 2D promptable VLMs (SAM3) in a structured open-vocabulary segmentation protocol, revealing that their complementary failure modes—dense-VLM noise on long-tail classes vs. 2D-segmenter view-specific false positives—can be reconciled purely geometrically without any learned fusion. The finding that few-shot visual prototypes add no signal over cross-view consistency is a strong negative result for VLM-based feature alignment in this regime, suggesting that geometric multi-view aggregation already extracts the available discriminative signal from the frozen VLM representations. For researchers tracking VLMs, this paper establishes a concrete lower bound on what frozen 3D- and 2D-VLM combinations can achieve on a challenging fine-grained benchmark, and demonstrates a scalable deployment pattern—no annotations, no retraining—relevant to open-world 3D perception.