Skip to content

Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

STS (Spectrum-Aware Test-Time Steering) is a test-time adaptation method for VLMs that steers frozen text prototype embeddings within a low-dimensional SVD-defined subspace, learning only a small shared coefficient vector per test sample. It achieves state-of-the-art OOD accuracy on CLIP while running 8× faster and using 12× less memory than TPT.

Problem

VLMs like CLIP degrade under test-time domain shifts. Existing episodic TTA methods (TPT and variants) require backpropagation through large frozen encoders, incurring high latency and memory cost. Memory-bank streaming methods assume well-distributed data streams, which is unrealistic. LoRA-based TTA (TTL) modifies internal architecture. Unconstrained prototype shifting (TPS) underperforms TPT. The gap is a TTA method that is black-box, backprop-free, parameter-minimal, and competitive in accuracy.

Method

STS pre-computes a spectral basis from the initial class text embeddings by performing SVD: Z_T^init = U_T S_T V_T^T. The top k_t right singular vectors (selected via the Gavish–Donoho optimal hard-thresholding criterion applied to singular values) form an adaptation basis B_T ∈ R^{D×k_t}. At test time, for each sample, a shared coefficient vector γ ∈ R^{k_t} (initialized to zero) is learned; the steering shift is ΔzT = B_T γ, applied uniformly to all class prototypes: (z_T^adapted)_c = normalize((z_T^init)_c + ΔzT). The objective minimizes Shannon entropy of the marginal probability over the top-10%-confidence augmented views (63 random crop/flip augmentations), plus L2 regularization on ΔzT, using a single AdamW step (lr=5e-3). Encoders remain entirely frozen; no encoder backprop occurs.

Key Contributions

  • First TTA method to constrain adaptation to the SVD-defined latent subspace of text prototypes, exploiting the empirically observed low intrinsic dimensionality of VLM embeddings.
  • Operates as a true black-box: frozen encoders, no architectural modification, applicable to proprietary models.
  • Learns only k_t ≪ D parameters per sample (the coefficient vector γ), achieving 8× lower latency and 12× smaller memory footprint than TPT on a single RTX8000.
  • Principled automatic rank selection via Gavish–Donoho optimal singular value thresholding.
  • Compatible with prompt ensembling (STSEnsemble), which further boosts accuracy without dataset-specific templates.

Results

  • ImageNet OOD average (CLIP ViT-B/16): STS 62.64% vs. TPT 60.71%, Zero-Shot 57.2%; +4.23% over TPT, +7.76% over zero-shot.
  • ImageNet-A: STS 61.23% vs. TPT 54.39%, TPS 57.46% — largest single-dataset gain.
  • STSEnsemble OOD average: 64.96%, best among all methods in that group.
  • Fine-grained (10 datasets, CLIP ViT-B/16): STSEnsemble 65.06% vs. TPT 64.78%, C-TPT 63.58%, TPS 63.49%; STS 63.86% outperforms all non-ensemble TTA baselines.
  • MaPLe+STS OOD average: 65.34% vs. MaPLe+TPT 62.31%; +3.03% improvement.
  • CIFAR10-C (severity 5): STS matches TPT within 0.05%, outperforms TPS; STSEnsemble reaches 67.24%.
  • Efficiency (ImageNet): STS 0.09s/sample, 1.4 GB vs. TPT 0.75s/sample, 17.6 GB; 8× faster, 12.6× less memory.

Limitations

  • Subspace steering is linear; highly non-linear domain shifts may not be addressable by linear shifts in the SVD subspace alone.
  • Visual processing cost scales linearly with the number of augmented views (each requires an independent encoder forward pass); latent-space visual augmentation is left as future work.
  • EuroSAT is a known failure mode for STS (as for other TTA methods), attributed to the unusual nature of satellite imagery requiring task-specific augmentations.

Relevance to Vision-Language Models

STS directly addresses a core challenge for deployed VLMs: maintaining zero-shot robustness under distribution shift without retraining or modifying frozen encoders, which is critical for proprietary and large-scale VLMs. It advances the line of work from TPT → TPS by introducing a principled geometric constraint (SVD subspace) that improves both accuracy and efficiency, offering a practical alternative to prompt tuning at inference time. The method's black-box nature and minimal parameter count make it broadly applicable across CLIP-family models and initialization strategies (e.g., MaPLe), connecting to the growing interest in efficient, training-free adaptation of foundation VLMs. The insight that VLM text embeddings exhibit strong low-rank structure (>90% energy in a small spectral subspace) is a useful geometric property for the community designing future latent-space adaptation methods.