Skip to content

Bilateral Information-aware Test-time Adaptation for Vision-Language Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; bilateral information-aware TTA handles covariate shifts without test distribution

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

BITTA (Bilateral Information-aware Test-Time Adaptation) addresses a systematic failure in VLM test-time adaptation: fixed low-entropy selection causes overfitting to atypical features and is suboptimal across diverse data distributions. It simultaneously applies entropy minimization on a dynamically-sized low-entropy subset and entropy maximization (unlearning) on a fixed-ratio high-entropy subset, improving CLIP robustness to corrupted images.

Problem

Standard TTA for VLMs selects a fixed proportion (e.g., top 10%) of low-entropy (high-confidence) samples for entropy minimization. This creates two failure modes: (1) misclassified samples within the confident set have their wrong predictions reinforced as the model memorizes atypical noise features, producing overconfident errors; (2) a single fixed ratio cannot be optimal across datasets or corruption types—the optimal ratio varies substantially with data distribution.

Method

BITTA jointly optimizes two objectives:

\[\mathcal{L} = \mathcal{L}_{\text{learning}} + \lambda \mathcal{L}_{\text{unlearning}}\]

Learning (low-entropy branch): A dynamically estimated proportion \(\tau_{lp}\) of confident samples is used to minimize entropy, enhance image–text cosine alignment, and maximize inter-class prototype separability (following BAT's objective).

Unlearning (high-entropy branch): A fixed proportion of the least-confident samples is used to maximize predictive entropy: $\(\mathcal{L}_{\text{unlearning}} = \sum_{i=1}^{K} p(y_i | X_{\text{high}}) \log p(y_i | X_{\text{high}})\)$ This forces the model to increase uncertainty on atypical-feature-heavy samples, which empirically reduces overconfident errors on low-entropy misclassifications via observed correlation between the two groups.

Dynamic ratio estimation: The threshold \(\tau_{ln}\) for low-entropy selection follows a linear calibration across datasets: $\(\frac{\tau_{ln}}{\max_i H(x_i)} = -0.00038K + 0.83\)$ where \(K\) is the number of categories. This converts to a predicted ratio \(\tau_{lp}\) using the empirical CDF of entropy values in the first batch. Conformal prediction (Theorem 3.2) provides a marginal coverage guarantee: \(P(Y_{n+1} \in [\hat{Y}_{n+1} \pm Q(1-\alpha)]) \geq 1 - \alpha\).

The framework is modular and wraps any existing TTA learning objective (TPT, TDA, BAT).

Key Contributions

  • Diagnoses two previously overlooked failure modes of fixed low-entropy TTA selection: atypical-feature memorization and ratio incompatibility across distributions.
  • Demonstrates theoretically (Theorem 3.1) and empirically that high-entropy samples are indistinguishable due to excessive distributional deviation, making them unsuitable for learning but useful for regularization via unlearning.
  • Proposes a calibration-based dynamic ratio estimator with conformal coverage guarantees (Theorem 3.2).
  • Achieves compatibility with multiple VLM TTA frameworks (TPT, TDA, BCA, DMN-ZS, DPE, BAT) without requiring architectural changes.

Results

Experiments on CIFAR-10-C, CIFAR-100-C, and ImageNet-C with CLIP (ViT-B/32 and ViT-B/16 backbones). Results cited below are for BAT vs. BAT+BITTA (the primary comparison), averaged over 15 corruption types:

  • CIFAR-10-C, ViT-B/32: BAT 61.15% → BAT+BITTA 69.02% (+7.87 pp average)
  • CIFAR-100-C, ViT-B/32: BAT 67.49% → BAT+BITTA 69.02% (+1.53 pp average, per reported ∆)
  • ViT-B/16, per-corruption improvements: consistently +0.31 to +3.48 pp across corruption types, with gains of +3.35 and +3.48 on Gaussian and shot noise
  • BITTA outperforms prior methods including TDA, BCA, DMN-ZS, and DPE across both architectures on corrupted benchmarks (table partially truncated in provided text)

Limitations

  • High-entropy selection ratio is fixed (not dynamically estimated), relying on the empirical finding that a small fixed fraction suffices for regularization—not theoretically justified at the same level as the low-entropy module.
  • Dynamic ratio calibration requires knowing \(K\) (category count) and depends on the entropy distribution of the first batch, which may be noisy in low-batch-size settings.
  • Experiments focus on common corruption benchmarks (ImageNet-C, CIFAR-C); generalization to other shift types (domain gaps, semantic shifts) is evaluated only in supplementary cross-domain experiments.
  • Theoretical guarantees (Theorem 3.1 distinguishability, Theorem 3.2 coverage) require assumptions (generative model structure, calibration set availability) that may not hold in practice.

Relevance to Vision-Language Models

BITTA targets the core deployment challenge for CLIP-style VLMs: maintaining zero-shot accuracy under real-world distribution shifts without access to labeled target data. By reframing TTA as a bilateral optimization problem—learning from confident predictions and actively unlearning from uncertain ones—it reveals that the standard low-entropy selection heuristic systematically degrades VLM adaptation through atypical-feature memorization. The modular design makes it a drop-in improvement over existing VLM TTA methods, and the dynamic ratio estimator addresses a fundamental dataset-sensitivity problem that prior works ignored. This work advances understanding of how entropy-based data curation interacts with foundation model adaptation, a critical issue as CLIP derivatives are deployed in safety-critical domains.