Skip to content

Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MLMP is the first test-time adaptation (TTA) framework for open-vocabulary semantic segmentation (OVSS) with VLMs like CLIP. It jointly minimizes entropy across multiple intermediate ViT layers and multiple text-prompt templates at both pixel-wise and global CLS-token levels. A 87-scenario benchmark suite is introduced alongside the method.

Problem

Existing TTA methods address image classification, not dense prediction. OVSS models built on CLIP are brittle under domain shift (corruptions, weather, rendered scenes) but no TTA mechanism existed for this setting — the intersection of TTA and OVSS was entirely unexplored.

Method

MLMP has two phases, adaptation and evaluation, applied to any OVSS backbone (demonstrated on NACLIP with ViT-L/14):

Uncertainty-Aware Multi-Level (UAML) Fusion: Features from all intermediate ViT layers \(q^\ell_i\) are aggregated via confidence-weighted averaging \(Q = \sum_\ell \alpha_\ell Q^\ell\), where weights \(\alpha_\ell \propto \exp(-\beta \cdot h^\ell)\) are derived from per-layer prediction entropy \(h^\ell\). During adaptation \(\beta=0\) (uniform); during inference \(\beta=1\) (sharpened to prefer confident layers).

Multi-Prompt Local-Global Adaptation: Seven text-prompt templates are used simultaneously. For each template \(T_t\), both a pixel-wise entropy loss \(L_\text{UAML}(T_t)\) over patch tokens and a global image-level entropy loss \(L_\text{ILE}(T_t)\) over the CLS token are computed. The final loss is \(L_\text{final}(\theta) = \frac{1}{T}\sum_t (L_\text{UAML}(T_t) + L_\text{ILE}(T_t))\). Averaging gradients across \(T\) templates reduces gradient variance by \(1/T\) (proven). Only LayerNorm parameters of the vision encoder are updated; the text encoder is frozen. Adaptation runs for 10 Adam steps (lr=1e-3, batch=2) with full reset per batch.

Key Contributions

  • First plug-and-play TTA framework for OVSS; compatible with any OVSS backbone without labels or source data
  • Uncertainty-aware multi-level ViT feature fusion with entropy-based confidence weighting
  • Loss-level multi-prompt adaptation treating each template as an independent gradient critic, with a proven 1/T variance reduction
  • Joint CLS-token (global) + patch-token (local) entropy minimization
  • Comprehensive OVSS-TTA benchmark: 9 datasets, 15 synthetic corruptions, real (ACDC) and rendered (GTA-V) shifts, 87 total test scenarios

Results

All numbers use NACLIP (ViT-L/14) as the OVSS backbone; baselines are TENT, TPT, WATT, CLIPArTT.

  • Pascal VOC 20 (clean): MLMP 83.76 vs. No-Adapt 77.00 (+6.76 mIoU); all baselines ≤77.00
  • Pascal VOC 20-C (15 corruptions avg): MLMP 77.58 vs. No-Adapt 69.33 (+8.25); TENT 69.33, WATT 63.39, CLIPArTT 48.30
  • Pascal VOC 21-C avg: MLMP 46.25 vs. No-Adapt 40.75 (+5.50)
  • Cityscapes (clean/+natural shift): MLMP 33.35 vs. No-Adapt 29.49 (+3.86)
  • ACDC (real adverse conditions, avg): MLMP 30.28 vs. No-Adapt 23.35 (+6.93); TENT 26.29
  • GTA-V (rendered): MLMP 28.84 vs. No-Adapt 25.09 (+3.75)
  • Ablation (V20): Last 75% of layers best (81.67 mIoU) vs. last-layer-only (77.00); uncertainty-aware weighting adds +4.29 on V20 clean vs. uniform averaging; loss-level prompt integration (72.58 V20-C avg) beats text-level averaging (71.92) and parameter averaging (68.44); full MLMP (83.76 / 77.58) exceeds any single component

Limitations

  • Layer confidence weights are computed using a shared projection head, which may not capture each layer's unique representational characteristics
  • Optimal layer range (last 75%) was validated on Pascal VOC; generalizability to radically different ViT architectures or non-ViT encoders is not established
  • Requires 10 gradient steps per batch at test time; computational overhead relative to zero-shot inference is not quantified in the main paper
  • Batch size of 2 means adaptation is sensitive to very small or single-sample batches in practice, and performance with a true single image is less thoroughly characterized
  • CLIPArTT could not run on some GPU memory-constrained settings, limiting full comparison coverage

Relevance to Vision-Language Models

This work directly addresses a gap in CLIP-based VLM deployment: while CLIP's zero-shot generalization is well-studied for classification, its fragility under domain shift in dense prediction is underexplored. MLMP exploits two properties intrinsic to VLMs — intermediate ViT feature richness and prompt-sensitivity — to construct a self-supervised adaptation signal without any labels, making it applicable to any CLIP-style backbone used for segmentation. The multi-prompt entropy framework is a general-purpose technique that could extend to other VLM-driven tasks requiring pixel-level understanding (e.g., referring segmentation, visual grounding). The newly released 87-scenario benchmark provides the community with a standardized protocol for evaluating robustness of VLM-based dense prediction methods.