Skip to content

Adaptive Debiasing Tsallis Entropy for Test-Time Adaptation

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; Tsallis entropy debiasing corrects CLIP's built-in bias during TTA

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CLIP and similar VLMs suffer from prediction bias toward head/tail classes due to imbalanced pretraining data, which corrupts Shannon Entropy-based view selection during test-time adaptation (TTA). This paper proposes Adaptive Debiasing Tsallis Entropy (ADTE), a generalized entropy formulation with class-specific, bias-derived parameters \(q^l \in (0,1)\) that corrects this bias without distribution-specific hyperparameter tuning. ADTE outperforms all SOTA TTA methods on ImageNet and its five variants plus 10 cross-domain benchmarks.

Problem

Shannon Entropy (SE), \(H_{SE} = -\sum_l p \log p\), is the standard uncertainty measure in TTA view-selection pipelines (e.g., TPT, Zero). Because CLIP is pretrained on imbalanced web-crawled data, its predicted probability \(p\) for a class deviates from the true unbiased probability \(\hat{p}\), causing SE values themselves to be biased. SE applies a uniform \(-p \log p\) formula across all classes and thus cannot account for class-specific degrees of bias (head vs. tail), leading to suboptimal selection of high-confidence augmented views.

Method

Tsallis Entropy (TE) generalizes SE by introducing a non-extensive parameter \(q\): $\(H_{TE}(P) = \frac{1}{1-q}\left(\sum_{l=1}^{L} P(y=l|x_j^{test})^q - 1\right).\)$ As \(q \to 1\), \(H_{TE} \to H_{SE}\). When \(0 < q < 1\), the correction term \(F(p, q) = \frac{p^q}{1-q} - (-p \log p)\) is positive, lifting the entropy estimate for tail classes and reducing it for head classes, thereby alleviating bias. The paper proves that there always exists \(q^* < 1\) such that TE selects views with higher Top-K Cumulative Reliability (\(\text{TcrK}\)) than SE.

ADTE eliminates manual \(q\) tuning by assigning a class-specific \(q^l\). The per-class label bias \(\tilde{p}_l = P(l)\) is estimated online via Jacobi iteration over a memory bank of streaming test instances using pseudo-labels: $\(\tilde{p}_l^{(t+1)} = \sum_{l' \in Y^{test}} \tilde{p}_{l'}^{(t)} \cdot \mathbb{E}_{x \sim P(x|l')}[P(l|x)], \quad \text{normalized to } \|\tilde{p}^{(t)}\|_1 = 1.\)$ The estimated bias \(-\log \tilde{p}_l\) (larger for tail classes) is min-max normalized to \((\alpha, \beta) = (0.01, 0.9)\): $\(q^l = \alpha + (\beta - \alpha)\frac{-\log \tilde{p}_l - \min(\tilde{p})}{\max(\tilde{p}) - \min(\tilde{p})}.\)$ ADTE entropy per view is then \(H_{ADTE}(P) = \sum_l \frac{P_l^{q^l}}{1-q^l}\). Views with the lowest \(H_{ADTE}\) values are selected as high-confidence, and their predictions are averaged for the final output. The bias estimate also feeds directly into logit adjustment.

Key Contributions

  • Identifies SE's class-uniform formula as the root cause of biased view selection in VLM TTA, and provides formal analysis showing TE performance is an upper bound over SE.
  • Proves and empirically validates that for \(0 < q < 1\), TE corrects biased entropy estimates, with correction magnitude proportional to class-level bias.
  • Proposes ADTE with streaming, Jacobi-iterated, per-class \(q^l\) derived from online bias estimation — no distribution-specific hyperparameter tuning required.
  • Demonstrates TE and ADTE as drop-in replacements for SE in any TTA pipeline without architectural changes.

Results

OOD benchmark (ViT-B/16, template prompts vs. Zero baseline): - ImageNet: 71.8% vs. 70.9% (+0.9%) - IN-A: 65.5% vs. 64.0% (+1.5%) - IN-K: 53.5% vs. BCA's 50.9% (+2.6%) - OOD average: 66.5% vs. Zero's 65.0%

OOD benchmark (ViT-B/16, CuPL prompts vs. Frolic baseline): - ImageNet: 72.7% vs. 70.9% (+1.8%) - IN-A: 63.5% vs. 60.4% (+3.1%) - OOD average: 66.2% vs. 64.8%

Cross-domain (ViT-B/16): ADTETemplates 69.0%, ADTECuPL 71.8% vs. Frolic 71.1% — highest average across all 10 datasets.

Cross-domain (ViT-L/14): ADTECuPL 77.7% vs. Frolic 77.1%; ADTETemplates 74.8% vs. TDA 74.2%.

  • Gains hold across both ViT-B/16 and ViT-L/14 backbones and both hand-crafted and GPT-generated prompts.

Limitations

  • The bias estimation memory bank uses pseudo-labels (hard argmax), which can compound errors for severely misclassified tail classes early in the stream.
  • The normalization interval \((\alpha, \beta)\) for \(q^l\) is fixed at \((0.01, 0.9)\); while claimed to require no tuning, this interval itself is a hyperparameter chosen empirically.
  • Analysis focuses on the Zero TTA framework; integration complexity with other TTA families (e.g., fully TTA requiring batch normalization updates) is not characterized.
  • Experiments are restricted to image classification with CLIP; generalization to generative VLMs or other modalities is not evaluated.

Relevance to Vision-Language Models

This work directly addresses a structural weakness of CLIP-style VLMs: prediction bias inherited from imbalanced web-scale pretraining that undermines downstream uncertainty estimation at test time. By replacing SE with a theoretically grounded, bias-correcting entropy formulation (ADTE) that operates on streaming unlabeled data, it advances the practical deployment of VLMs under real-world distribution shifts without retraining or prompt engineering. The result is directly complementary to prompt-tuning TTA lines (TPT, DiffTPT) and logit-adjustment bias-correction work (Frolic, BCA), showing the entropy measurement itself is a first-class design choice. For researchers tracking VLM generalization, this highlights that the entropy estimator — not just the adaptation strategy — is a critical and often-overlooked source of bias.