Ramen: Robust Test-Time Adaptation of Vision-Language Models with Active Sample Selection¶

🕒 Published (v1): 2026-04-23 14:33 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

Active sample selection for robust CLIP test-time adaptation; CVPR 2026

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Ramen is a test-time adaptation (TTA) framework for CLIP that tackles mixed-domain test streams by constructing per-sample, domain-consistent support sets from a cached memory of past test images. Instead of recomputing gradients over the retrieved support set, it aggregates pre-cached sample-level gradients, achieving $O(1)$ additional compute per sample. Ramen outperforms all baselines on multiple corruption and domain-shift benchmarks under mixed-domain settings.

Problem¶

Existing TTA methods assume test samples are drawn from a single, consistent domain and adapt a shared model to the full test stream. When test data contains samples from multiple distinct domains simultaneously, the model is forced to adapt to an averaged representation across domains, causing performance degradation. Prior fixes (SAR's sharpness-aware minimization, UnMix-TNS's multi-statistic BatchNorm) remain limited to single-model paradigms or specific layer types.

Method¶

Ramen assigns each test sample $x_i$ a distinct adapted model weight $\mathbf{w}_i$ by constructing a sample-specific support set $S_i$ from a class-split FIFO memory $\mathcal{M} = \{\mathcal{M}_1, \ldots, \mathcal{M}_C\}$, where each queue $\mathcal{M}_c$ stores up to $K$ recently seen samples predicted as class $c$.

Active sample selection retrieves the support set via two criteria: 1. Domain consistency: for each class queue $\mathcal{M}_c$, retrieve the top-$k$ samples by cosine similarity with the query embedding $\mathbf{z}_i$: $$S_{ic} = \underset{j \in \mathcal{M}_c}{\text{Top-}k}\left(\mathbf{z}_i^\top \mathbf{z}_j\right), \quad S_i = \bigcup_{c=1}^C S_{ic}$$ 2. Prediction balance: equal $k$ samples are drawn from each class queue, ensuring the support set is class-balanced.

Embedding-gradient cache avoids recomputing gradients on $S_i$. Since the entropy loss is additive over samples, the batch gradient decomposes as: $$\nabla_{\mathbf{w}} H(S_i) = \sum_{j \in S_i} \alpha_{ij} \mathbf{g}_j, \quad \mathbf{g}_j = \nabla_{\mathbf{w}} H(x_j)$$ The memory stores $(z_j, g_j)$ pairs; aggregation uses combined entropy- and similarity-weighting: $$\alpha_{ij} = \exp(-H(x_j)) \cdot \exp(-\beta \|\mathbf{z}_i - \mathbf{z}_j\|_2)$$ Only normalization layer affine parameters (<0.05% of ViT-B/16 params) are updated per sample, then reset after inference.

A theoretical analysis (Theorem 4.1) shows entropy minimization amplifies features where $(t_{1h}-t_{0h})^2 \cdot M_{hh}$ is above average ($M_{hh}$ is the probability-weighted second moment of feature $h$ over the support set). Domain consistency reduces variance of domain-specific features in $M_{hh}$; prediction balance maintains high variance for class-discriminative features.

Key Contributions¶

Active sample selection framework with domain consistency and prediction balance criteria for mixed-domain TTA.
Embedding-gradient cache that eliminates $C \cdot k$ redundant backward passes per test sample by reusing cached gradients.
Theoretical characterization of entropy minimization over normalization layers, explaining why the two criteria improve adaptation under mixed-domain shift.
State-of-the-art results on CIFAR-10/100-C, ImageNet-C, and DomainNet under mixed-domain evaluation protocols.

Results¶

All numbers are mean accuracy (%) under mixture of 15 corruptions; Ramen is the proposed method.

ViT-B/32 on CIFAR-10-C: Ramen 72.7% vs. RoTTA 71.4% (prev. best), SAR 69.1%, CLIP 59.0%
ViT-B/16 on CIFAR-100-C: Ramen 46.1% vs. SAR 42.7% (prev. best), RoTTA 41.1%, CLIP 35.8%
ViT-L/14 on ImageNet-C: Ramen 49.2% vs. RoTTA 45.4% (prev. best), SAR 46.6%, CLIP 39.6%
Ramen ranks first or second on nearly all 15 individual corruption types across all three benchmarks.
DomainNet results are also reported but the table is truncated in the provided text.

Limitations¶

Theoretical analysis is restricted to a single normalization layer in a binary classification setting with a small learning-rate assumption; generalization to deeper, multi-class settings is informal.
The method resets model parameters after every sample, precluding persistent, progressive adaptation across the test stream.
Memory stores both image embeddings and full gradient vectors per sample, incurring $O(N \cdot d)$ storage where $d$ is the parameter count of the normalization layers—this could be non-trivial for large $C$ or $K$.
Evaluation is limited to image classification; robustness of the framework to other VLM tasks (e.g., retrieval, VQA) is not studied.
The method relies on pseudo-labels for class-queue assignment; persistent misclassification by the zero-shot classifier could degrade the quality of domain-consistent retrieval.

Relevance to Vision-Language Models¶

Ramen directly addresses the practical deployment gap of CLIP-style VLMs: zero-shot generalization degrades under the distribution shifts common in real-world data streams, and this paper provides a principled, efficient remedy that requires no labeled data, no source data replay, and no architectural changes. The theoretical analysis of how entropy minimization interacts with CLIP's normalization layers gives interpretable insight into why standard TTA is fragile under mixed domains—a finding relevant to any VLM using similar adaptation paradigms. The framework extends naturally to prompt-tuned or fine-tuned CLIP variants, making it a broadly applicable building block for robust VLM deployment. The class-split memory design also complements existing VLM-specific TTA methods (TDA, DMN-ZS) and could be combined with them.