Skip to content

Domain Generalizable Adaptation of 3D Vision-Language Models via Regularized Fine-Tuning

🕒 Published (v1): 2026-06-16 20:31 UTC · Source: Arxiv · Venue: TMLR · link

Why this paper was selected

TMLR; regularized fine-tuning for domain-generalizable 3D vision-language model adaptation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ReFine3D is a regularized fine-tuning framework for 3D vision-language models (specifically ULIP/ULIP-2) that addresses domain generalization under limited labeled data. It combines layer-selective fine-tuning with augmentation consistency regularization, LLM-generated text diversity, point-rendered vision supervision, and test-time augmentation. On standard 3D domain generalization benchmarks it consistently outperforms the prior SOTA (PointPRC) across base-to-new, cross-dataset, corruption, and few-shot settings.

Problem

Adapting tri-modal 3D VLMs (point cloud + image + text) to downstream domains with scarce labels causes overfitting and catastrophic forgetting. Existing PEFT methods (e.g., PointPRC) naively port image-domain prompt tuning to 3D, overlook the structural differences of unordered point clouds, and discard the pre-trained CLIP image encoder during fine-tuning—losing rich visual priors. Simultaneously, the smaller capacity of 3D encoders (PointBERT: 22.8 M vs. CLIP ViT-B/16: 86 M) makes them behave differently from image encoders under limited-data PEFT.

Method

ReFine3D operates on a frozen CLIP text encoder and frozen CLIP ViT-B/16 image encoder, fine-tuning only the last layer of a 12-layer PointBERT encoder (layers 1–11 frozen).

Augmentation-based consistency regularization. \(K=16\) geometric augmentations (random \(z\)-axis rotation \([0°,360°]\), scale \([0.8,1.2]\), jitter \(\sigma=0.01\)) are applied to each input point cloud \(x\), yielding \(\{x_k\}_{k=1}^K\). The total loss is:

\[L_\text{aug} = \frac{1}{K}\sum_{k=1}^K L^k_\text{CE}, \quad L^k_\text{CE} = -\log\frac{\exp(\text{sim}(z^k_P, z^y_T)/\tau)}{\sum_{c=1}^C \exp(\text{sim}(z^k_P, z^c_T)/\tau)}\]

Text diversity regularization. For each class, 7 synonyms are retrieved from WordNet, then Qwen-2.5-7B-Instruct generates 7 descriptive sentences per synonym (prompt template: "A 3D point cloud of {synonym}"). Point cloud embeddings are aligned against the averaged diverse text embeddings rather than a single hand-crafted template.

Point-rendered vision supervision. 12 views per object are rendered via Blender/BlenderProc at 512Ă—512. Rendered image features from the frozen CLIP image encoder are pulled into a tri-modal contrastive alignment with \(z_P\) and \(z_T\), recovering the visual priors that prior methods discard.

Test-time augmentation. \(K_\text{test}=5\) augmented point cloud views and 7 text prompts per class are generated; the top-\(H=3\) embeddings ranked by maximum softmax probability are selected and aggregated by majority voting.

Key Contributions

  • ReFine3D framework: layer-selective fine-tuning of 3D VLMs (only the last transformer layer is trainable) designed for domain-generalizable adaptation.
  • Augmentation-based consistency regularization: averages CE loss across \(K=16\) augmented views to reduce memorization of specific instances.
  • Text synonymization: WordNet + LLM-generated diverse descriptive sentences as class text embeddings, promoting semantic robustness.
  • Point-rendered vision supervision: Blender-rendered images fed to the frozen CLIP image encoder re-introduce visual priors discarded by prior PEFT methods.
  • Confidence-based test-time augmentation: top-\(H\) selection by max softmax probability with majority voting improves inference robustness.

Results

All improvements are reported over PointPRC (Sun et al., 2024a) as the prior SOTA, with ULIP-2 backbone unless noted.

  • Base-to-new generalization (avg over 5 datasets): Base 84.71% vs. 83.18% (+1.53%), New 77.46% vs. 76.10% (+1.36%), HM 80.40% vs. 79.48% (+0.92%).
  • Cross-dataset OOD transfer (avg over 5 target domains): 53.62% vs. 51.19% (+2.43%); source ShapeNetV2 → targets ModelNet40, ScanObjectNN variants, Omni3D.
  • Corruption robustness (ModelNet-C, severity=2): +1.80% average over corruption types.
  • Few-shot learning: +3.11% with 1-shot supervision; gains persist through 2, 4, 8, 16 shots.
  • Per-dataset highlights (ULIP-2, base-to-new HM): ShapeNetCoreV2 90.26% vs. 88.34%; S-OBJ_BG 86.06% vs. 84.28%; S-PB_T50_RS 75.90% vs. 73.97%.

Limitations

  • Incremental novelty: Authors explicitly acknowledge that none of the individual components are novel; contribution is their combination.
  • Architecture dependency: Requires a tri-modal backbone (point cloud + image + text encoders simultaneously available); not applicable to unimodal or asymmetric 3D models.
  • Rendering overhead: Blender-based point-to-image rendering adds preprocessing cost and requires offline rendering pipelines.
  • Modest absolute gains: Improvements are 1–3 percentage points over PointPRC across most benchmarks.
  • Narrow backbone evaluation: Experiments use PointBERT as the sole 3D encoder; generalization to other architectures (e.g., Point-MAE, Point-M2AE) is not tested.
  • Limited corruption severity coverage: Corruption experiments use severity=2 only; behavior at higher severity levels is unreported.

Relevance to Vision-Language Models

ReFine3D directly extends the CLIP/ULIP paradigm—central to the VLM literature—into the 3D point cloud domain, showing that the tri-modal alignment backbone can be adapted with minimal parameter updates while preserving generalizable priors. The text diversity strategy (WordNet synonyms + LLM paraphrasing) is a domain-agnostic regularizer applicable to 2D VLM fine-tuning as well. The finding that retaining the frozen image encoder improves 3D adaptation reinforces a broader principle in VLM fine-tuning: pre-trained cross-modal supervision from non-target modalities should not be discarded. For researchers tracking VLMs, this paper demonstrates how CLIP-style models can be specialized for non-image sensing modalities under distribution shift, a direction increasingly relevant as VLMs are deployed in robotics and autonomous driving pipelines.