Robustifying Vision-Language Models via Test-Time Prompt Adaptation

🕒 Published (v1): 2026-07-10 14:19 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML 2026; test-time prompt adaptation for adversarial robustness in VLMs; strong venue, practical defense method

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RITA is a test-time prompt adaptation framework that defends CLIP against adversarial perturbations by modeling both visual and textual inputs as discrete distributions and aligning them via entropy-regularized optimal transport, rather than relying on per-sample confidence heuristics. A dynamic cache progressively accumulates low-entropy augmented views to refine alignment across the test stream. RITA significantly improves adversarial robustness without degrading clean accuracy and is plug-and-play compatible with adversarially fine-tuned CLIP variants.

Problem

Existing test-time prompt tuning methods (TPT, R-TPT, C-TPT) filter augmented views using sample-level entropy but treat each view as an isolated point, ignoring the latent distributional structure of the augmented view set. This prevents them from distinguishing confident adversarial mispredictions from genuinely semantically consistent views, limiting robustness under attack. Adversarial training is an alternative but requires expensive re-training and labeled data, sacrificing zero-shot flexibility.

Method

Key observation: White-box adversarial attacks optimized for the holistic image embedding are geometrically brittle—random crops and flips partially break the spatial correspondence of the perturbation pattern, so some augmented views retain semantic integrity (lower entropy, better class separability).

Distribution-level alignment via OT. For each test image \(\hat{x}_t\), RITA generates \(N=64\) augmented views and extracts features \(\{\hat{x}_t^n\}_{n=1}^N\). For each class \(k\), \(M=4\) learnable text prompts produce textual features \(\{z_k^m\}_{m=1}^M\). Both modalities are treated as uniform discrete distributions: $\(P_t = \frac{1}{N}\sum_{n=1}^N \delta_{\hat{x}_t^n}, \quad Q_k = \frac{1}{M}\sum_{m=1}^M \delta_{z_k^m}\)$ Alignment is measured by entropy-regularized OT distance with cosine cost \(C_{t,k}(n,m) = 1 - \cos(\hat{x}_t^n, z_k^m)\): $\(d_\text{OT}(P_t, Q_k; C_{t,k}) = \min_{T_{t,k} \in \Pi} \langle T_{t,k}, C_{t,k}\rangle - \lambda h(T_{t,k})\)$ The initial label prediction is \(\hat{y} = \arg\min_{k} d_\text{OT}(P_t, Q_k; C_{t,k})\).

Dynamic cache. Low-entropy views (per-view entropy \(H(p_t^n) \leq \gamma = 0.8\)) are pseudo-labeled and accumulated in a class-wise cache \(\{\hat{X}_k\}_{k=1}^K\). Cached features are aligned to textual space via Orthogonal Procrustes: $\(W_k^* = \arg\min_{W^\top W=I} \|\hat{X}_k W - \mathbf{1}_{N_k}\bar{z}_k^\top\|_F^2\)$ Final inference minimizes the joint transport cost: $\(\hat{y} = \arg\min_{k} \left[ d_\text{OT}(P_t, Q_k; C_{t,k}) + \alpha\, d^c_\text{OT}(P_t, \tilde{Q}_k; \tilde{C}_{t,k}) \right]\)$ Prompt parameters \(\omega\) are updated once per test sample (AdamW, lr=0.005) by minimizing this joint objective; CLIP backbone is frozen throughout.

Theoretical justification. For \(\ell_2\)-normalized features, the 2-Wasserstein distance decomposes as: $\(W_2^2(P_t, Q_k) \approx \underbrace{\|\mu_x - \mu_z\|^2}_{L_\text{mean}\text{ (TPT-equivalent)}} + \underbrace{B^2(\Sigma_x, \Sigma_z)}_{R_\text{var}\text{ (structural penalty)}}\)$ where \(B^2\) is the Bures metric. Standard centroid-based methods leave \(R_\text{var}\) unconstrained, allowing attacks to distort distribution geometry without increasing \(L_\text{mean}\).

Key Contributions

• RITA: a test-time prompt adaptation framework that shifts VLM adversarial defense from point-level to distribution-level alignment.
• Formulation of cross-modal alignment as entropy-regularized optimal transport over multi-view visual features and multi-prompt textual prototypes.
• Dynamic cache mechanism using Orthogonal Procrustes alignment to progressively refine inference from the test stream.
• Theoretical decomposition showing that OT strictly subsumes centroid alignment by additionally penalizing covariance mismatch.
• Plug-and-play compatibility with adversarially fine-tuned CLIP backbones (TeCoA, PMG, FARE).

Results

• vs. vanilla CLIP (ViT-B/32, PGD ε=1.0, 8 datasets avg.): RITA improves robust accuracy from 4.7% to 49.7% (+45.0 pp).
• vs. vanilla CLIP (ViT-B/16): 1.8% → 52.7% (+50.9 pp).
• vs. R-TPT (state-of-the-art TTA, ViT-B/32): +1.8 pp average robust accuracy; also higher clean accuracy (58.6% vs. 57.1%).
• vs. R-TPT (ViT-B/16): +2.2 pp average robust accuracy (52.7% vs. 50.5%).
• With FARE backbone on Caltech101 (PGD): 81.9% robust accuracy, +19.0 pp over FARE alone, and best among all adapter combinations.
• Under CW attack (ViT-B/16, avg.): RITA 54.0% vs. R-TPT 51.2%.
• Under DI attack (ViT-B/16, avg.): RITA competitive or superior across most datasets (partial table shown).
• Clean accuracy: RITA maintains highest or near-highest average clean accuracy across both backbones, showing no clean-robust trade-off.

Limitations

• Sinkhorn OT computation adds per-sample test-time overhead proportional to \(N \times M \times K\) (64 views × 4 prompts × \(K\) classes); runtime cost on large-vocabulary datasets is not analyzed.
• The dynamic cache starts empty; early test samples receive no cache benefit, so ordering effects on streaming accuracy are not characterized.
• Evaluation uses a fixed ε=4.0/7-step PGD for Table 1's fine-grained datasets; generalization across wider ε ranges is only partially explored.
• The DI attack results table appears truncated in the provided text, leaving several dataset columns unreported.
• Relies on the assumption that geometric augmentations (crop, flip) sufficiently break adversarial patterns; adaptive attacks specifically targeting augmentation-invariant perturbations are not evaluated.

Relevance to Vision-Language Models

RITA directly addresses a core vulnerability of CLIP-style VLMs: the fragility of zero-shot inference under adversarial perturbation at the cross-modal alignment stage. By repurposing optimal transport—already established in VLM representation learning (PLOT, ALIGN, AWT)—as an adversarial defense tool, the work bridges two active research threads in the field. The plug-and-play design, requiring no backbone updates and no labeled data, makes RITA a practical inference-time add-on for any frozen CLIP variant, including already robustified ones. The Orthogonal Procrustes bridge between cached visual features and the textual manifold is a transferable technique for any VLM architecture that separates visual and textual encoders.