Skip to content

Training-Free Test-Time Adaptation via Shape and Style Guidance for Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SSG introduces shape-sensitive (SHS) and style-insensitive (STI) generalizable factors into training-free test-time adaptation for CLIP, replacing the entropy-only paradigm. It measures these factors via a lightweight Perturbed Prediction Difference (PPD) computed from patch-shuffle (shape) and color-affine (style) corruptions, then uses PPD to reweight augmented feature aggregation and govern dynamic cache updates. SSG consistently outperforms prior training-free and prompt-tuning TTA methods with comparable inference speed to TDA (~15 min on ImageNet-A).

Problem

Existing training-free test-time adaptation (TTA) methods for VLMs (e.g., TDA, BoostAdapter) rely exclusively on entropy to select high-confidence augmented views and update dynamic visual caches. Entropy alone ignores generalizable factors—shape-sensitive and style-insensitive representations—that are theoretically shown to be commonly positively correlated with labels across domain shifts, while high-entropy filtering can admit spurious, domain-specific features that degrade OOD generalization.

Method

SSG operates at test time without any parameter optimization:

  1. PPD Measurement: For each test image, SSG generates shape-corrupted views via patch-shuffle (corrupts global structure, preserves local textures) and style-corrupted views via color affine transformation + hue adjustment. It runs entropy-based view selection on all three sets (raw, shape-perturbed, style-perturbed), then computes:
  2. PPDsh = Pe − P̂e (raw minus shape-perturbed predictions at the pseudo-label class)
  3. PPDst = Pe − P̃e (raw minus style-perturbed predictions)

  4. PPD = PPDsh − PPDst (large PPD → high shape-sensitivity AND style-insensitivity)

  5. Feature Reweighting: Instead of averaging entropy-selected features Fe, SSG uses softmax-normalized PPD scores to reweight features: ftest = Σ PPD(Pe)_i · f*_test,i. This emphasizes samples with strong generalizable factors.

  6. Cache Criteria: The dynamic visual cache uses a combined score H(ftest·W_C^T) − PPD as the replacement criterion, so stored samples must be both high-confidence (low entropy) and high-generalizable (high PPD). Both the cache keys and query features carry PPD-induced reweighting.

Shape and style branches are batched together with the augmented views in a single forward pass to avoid extra forward overhead.

Key Contributions

  • First integration of generalizable factors (shape-sensitive, style-insensitive) into training-free VLM TTA, with theoretical justification via disentangled latent factor analysis.
  • PPD metric that quantifies both shape-sensitivity (PPDsh) and style-insensitivity (PPDst) using extremely cheap image-space perturbations.
  • Dual use of PPD: (i) feature reweighting during aggregation, (ii) cache update criterion jointly with entropy.
  • Plug-and-play design: SSG+ extends BoostAdapter, demonstrating composability.
  • Efficiency parity with TDA (~15 min) while surpassing it by ~1.3% OOD average on ViT-B/16, and outperforming DPE at <40% of its memory footprint (3862 MiB vs. 9986 MiB).

Results

OOD benchmark (4 ImageNet variants), ViT-B/16: - SSG: 65.20% avg vs. TDA 63.55%, BoostAdapter 64.43%, TPT 59.91%, DPE 63.76% - SSG+: 66.24% avg (best overall) - Individual: ImageNet-A 62.02 (SSG), 65.16 (SSG+) vs. BoostAdapter 59.63

OOD benchmark, ResNet-50: - SSG: 47.78% avg vs. TDA 46.63%, BoostAdapter 48.20%, TPT 43.89% - SSG+: 49.16% avg (best)

Cross-domain benchmark (10 datasets), ViT-B/16: - SSG avg: 70.24%, SSG+ avg: 71.20% vs. BoostAdapter 68.68%, TDA 67.53%, DPE 69.40%

Efficiency (ViT-B/16, ImageNet-A): - SSG: ~15 min, +8.0% over zero-shot CLIP; TDA: ~15 min, +6.69%; DPE: ~28 min, +7.23%; TPT: 156 min, +3.61%

OpenCLIP ViT-L/14: - SSG OOD avg: 72.26% vs. TDA 70.69%, DPE 71.86%

Ablation: - PPDsh alone > PPDst alone; combined PPD > either alone - Optimal patch count for shape perturbation: 4–6 patches - PPD reweighting + PPD cache comparison > either alone

Limitations

  • Shape and style perturbations are simple (patch-shuffle, color affine); more expressive alternatives (Fourier-based, channel-based) may improve the PPD signal quality but are left for future work.
  • Style-insensitive factors are inappropriate for tasks where style is semantically informative (e.g., texture-based medical imaging); the method would require adaptation for such domains.
  • Memory usage increases relative to TDA (3862 MiB vs. 3098 MiB) due to triple-input batching, though this is mild.
  • Cache size is fixed at 3 per dataset; sensitivity to this hyperparameter across diverse fine-grained domains is not fully analyzed.

Relevance to Vision-Language Models

SSG directly addresses the central challenge of deploying CLIP-based VLMs under distribution shift without any retraining or prompt optimization, which is a practical bottleneck for production use. By introducing the shape/style PPD signal alongside entropy, it advances the understanding of what makes a test sample reliable beyond confidence—a question relevant to all retrieval-augmented and cache-based VLM adaptation paradigms. The plug-and-play nature means SSG can be grafted onto future training-free TTA baselines (demonstrated on BoostAdapter), making it a modular building block in the growing VLM TTA literature. The theoretical grounding via disentangled factor analysis also provides a principled lens for evaluating other generalizable-factor hypotheses in VLM adaptation.