Skip to content

FHGS: Feature-Homogenized Gaussian Splatting

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FHGS resolves the fundamental conflict between the anisotropic rendering of 3D Gaussian Splatting (3DGS) primitives and the isotropic consistency requirements of semantic feature fields by freezing pre-trained 2D features (SAM, CLIP) as non-differentiable attributes on each Gaussian primitive. A physics-inspired dual-drive optimization (external potential field + internal clustering) enforces cross-view semantic coherence without re-optimizing the frozen features. The result is 15× faster training than Feature3DGS with comparable or better feature fusion quality at ≥60 FPS rendering.

Problem

Existing 3DGS-based feature distillation methods (e.g., Feature3DGS, LangSplat) attach learnable, differentiable feature embeddings to Gaussian primitives and optimize them jointly with appearance. This introduces an inherent contradiction: the anisotropic, view-dependent rasterization pipeline that governs RGB color conflicts with the view-independent (isotropic) consistency required for semantic features. The result is cross-view feature inconsistency, noise in segmentation boundaries, and corruption of the self-attention structure in transformer-derived embeddings. Additionally, these methods incur prohibitively long training times (>24h for Feature3DGS on some scenes).

Method

FHGS builds on a 2DGS backbone and augments each Gaussian primitive with a frozen, non-differentiable semantic feature vector \(f_i\) extracted from a pre-trained model (SAM image encoder by default). The three core components are:

  1. General Feature Fusion Architecture: A spatial hash table maps 3D SfM points to their 2D projections across all visible views. At initialization, each Gaussian inherits a frozen feature sampled from the ground-truth 2D feature map \(F^\text{gt}\); no feature rendering is needed during the forward pass.

  2. Non-Differentiable Feature Driving (NDFD): The feature loss \(L_\text{feat}\) is computed directly from frozen \(f_i\) and the differentiable blend weights \(w_i\). Although \(f_i\) has no gradient, \(\partial L / \partial w_i\) propagates to the geometric parameters \(\{p_i, R_i, S_i, \text{op}_i\}\), driving primitives to spatially reorganize toward feature-consistent regions without modifying the feature vectors themselves.

  3. Physics-Inspired Dual-Drive Mechanism:

  4. External Potential Field (\(L_\text{gt}\)): Computes cumulative similarity between each primitive's frozen feature and the ground-truth ray feature using a sigmoid activation \(\sigma_i = 1/(1+e^{k(\phi-\lambda)})\) on cosine similarity, analogous to charge polarity in an electric field.
  5. Internal Feature Clustering (\(L_\text{cf}\)): Compares each primitive's feature against the cumulative weighted feature of farther primitives along the ray, reducing \(O(N^2)\) pairwise comparisons to \(O(N)\) via prefix accumulation. This suppresses both inter-cluster noise and intra-cluster outliers.
  6. Total loss: \(L = L_\text{rgb} + \lambda_1 L_\text{gt} + \lambda_2 L_\text{cf}\).

Key Contributions

  • Identifies and resolves the anisotropy–isotropy conflict in 3DGS feature distillation by making feature vectors non-differentiable frozen attributes rather than learned embeddings.
  • Physics-inspired dual-drive optimization combining external semantic field alignment and internal clustering regularization, both operating through the existing differentiable blend weights \(w_i\).
  • New cross-view consistency metric FE (Feature Entropy) based on \(L_\text{gt}\) for evaluating global feature coherence.
  • Demonstrated plug-in compatibility with large-scale pre-trained vision models (SAM, CLIP) without architectural modifications to those models.

Results

  • Training time: FHGS trains in 4.8–7.8 min vs. Feature3DGS's 73–181 min (≈15× speedup); on TnT-Caterpillar, Feature3DGS runs out of GPU memory entirely.
  • Feature consistency (FE↓): FHGS achieves FE of 0.15–0.25 vs. Feature3DGS's 0.52–0.88 and baseline 2DGS's 1.31–1.73 across all benchmarks (DTU-24/37, MN360-Kitchen/Garden, TnT-Caterpillar).
  • Feature similarity (FL1↓): Comparable to Feature3DGS (e.g., 0.22 vs. 0.24 on DTU-24); both substantially outperform raw 2DGS (0.61).
  • Geometry (Chamfer Distance on DTU): FHGS CD=0.75 vs. 2DGS CD=0.83 and Feature3DGS CD=1.85; PSNR 34.21 vs. 33.42 (2DGS) and 35.25 (Feature3DGS).
  • Model size: 183 MB vs. Feature3DGS 745 MB; primitive count 196k vs. 642k.
  • Rendering: ≥60 FPS maintained.
  • Ablation: Removing \(L_\text{cf}\) drops PSNR from 30.9 to 27.2 and increases FL1 from 0.16 to 0.21, confirming complementary roles of both loss terms.

Limitations

  • Similarity-activation hyperparameters (\(\lambda\), \(k\)) are manually tuned; no adaptive scheme is proposed.
  • Spatial hash table and cumulative-weight buffers incur high GPU memory in large-scale scenes.
  • PSNR degrades in outdoor scenes relative to Feature3DGS (e.g., 26.5 vs. 28.2 on MN360-Garden), attributed to filtering of low-semantic regions (soil, shadows) that hurts RGB reconstruction metrics.
  • Evaluation is limited to SAM image embeddings; generalization to other foundation model features (e.g., CLIP language-aligned features) is claimed but not benchmarked quantitatively.

Relevance to Vision-Language Models

FHGS provides a principled mechanism for distilling VLM/vision foundation model features (CLIP, SAM) into 3D scene representations without destroying the learned representation structure—a key concern when bridging 2D VLM outputs to 3D. The non-differentiable freezing strategy directly preserves transformer self-attention structure and class relationships encoded in CLIP embeddings, enabling downstream open-vocabulary querying over 3D scenes. This connects to the broader effort of building language-queryable 3D scene graphs (cf. LangSplat, LERF) but with a more scalable, training-efficient approach. For VLM researchers, the method demonstrates that anisotropic 3D rendering and isotropic semantic representations need not be co-optimized—a design principle applicable to any 3D-grounded VLM pipeline.