When Can Test-Time Adaptation Help Zero-Shot CT Vision-Language Models?

🕒 Published (v1): 2026-07-17 01:54 UTC · Source: Arxiv · link

Why this paper was selected

Leonid Sigal (UBC); systematic TTA scope study on zero-shot 3D CT VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper asks when, not whether, test-time adaptation (TTA) helps zero-shot 3D CT vision-language models under distribution shift. It identifies two deployment preconditions (volumetric input integrity and transferable base representations) and introduces CARVE, the first cardinality-aware TTA method for the prompt-pair multi-label CT regime. CARVE consistently outperforms standard entropy-minimization baselines when the base model is already discriminative.

Problem

Zero-shot 3D CT VLMs (CT-CLIP, fVLM) score each abnormality as an independent Bernoulli variable via prompt-pair cosine similarity rather than a shared softmax. This structure breaks standard TTA objectives: cross-label entropy minimization collapses co-occurring positive labels, and per-label binary entropy vanishes near \(p=0.5\). Moreover, no prior work addresses unsupervised multi-label TTA for volumetric CT, and adaptation is assumed universally beneficial despite depending on input pipeline choices and base-model representational quality.

Method

CARVE (Cardinality-Aware Retained-View Entropy) proceeds in three stages per test scan, with episodic normalization-parameter updates:

1. Weak multi-view generation & selection. \(V=8\) stochastically augmented 3D copies are scored; the \(K = \max(1, \text{round}(V\rho))\) lowest-entropy views (by mean binary entropy across labels, \(E^{(v)} = \frac{1}{L}\sum_j h(p_j^{(v)})\)) are retained. Only retained views receive gradient computation, keeping memory tractable.

2. Sample-specific cardinality estimation. The soft Bernoulli count \(\hat{k} = \text{clipround}\!\left(\sum_{j=1}^L \bar{p}_j,\, k_\text{min},\, k_\text{max}\right)\) is derived directly from the retained-view averaged probabilities \(\bar{p}\), without caption retrieval.

3. Cardinality-aware top-\(\hat{k}\) objective. For each retained view \(v\), the top-\(\hat{k}\) labels \(P^{(v)}\) are sharpened and the rest \(N^{(v)}\) suppressed: $\(\mathcal{L}_\text{CARVE}^{(v)} = -\frac{1}{|P^{(v)}|}\sum_{j \in P^{(v)}} \log p_j^{(v)} - \frac{\lambda_\text{neg}}{|N^{(v)}|}\sum_{j \in N^{(v)}} \log(1 - p_j^{(v)})\)$ with \(\lambda_\text{neg}=0.8\). A prediction-dispersion gate \(\sigma_\text{pred}(x) = \text{std}(\{p_1,\ldots,p_L\})\) skips adaptation when label probabilities lack contrast. Only affine visual normalization parameters \(\phi\) are updated (Adam, lr \(10^{-5}\), 2 steps); all text-side and non-normalization visual parameters are frozen.

Key Contributions

• Diagnostic study isolating three conditions for TTA to help: (1) native-depth volumetric input, (2) transferable base representations, (3) an objective matched to prompt-pair Bernoulli multi-label structure.
• Identification of the entropy mismatch: standard cross-label entropy suppresses co-occurring abnormalities; per-label binary entropy degrades near the decision boundary.
• CARVE: first TTA method for zero-shot prompt-pair multi-label 3D CT, combining scan-specific cardinality estimation, retained-view selection, and memory-efficient volumetric adaptation.

Results

• CT-CLIP on internal CT-RATE (18-label, Zero-shot): CARVE AUROC 0.749 vs. No-TTA 0.713, TENT 0.712, RLCF 0.713, ML-TTA 0.712.
• CT-CLIP on external RAD-ChestCT (16-label, Zero-shot): CARVE AUROC 0.533 vs. No-TTA 0.499, TENT 0.508, RLCF 0.504, ML-TTA 0.508.
• CT-CLIP Vocab-FT on CT-RATE: CARVE AUROC 0.804 vs. No-TTA 0.763.
• Input depth ablation: reducing depth from \(z=240\) to \(z=40\) drops Zero-shot AUROC by 0.125 (0.713 → 0.588) on CT-RATE; on RAD-ChestCT, the unadapted base is near chance (0.499) at native depth, and no TTA objective rescues it — demonstrating that adaptation cannot substitute for representational transfer.
• CARVE is evaluated across CT-CLIP and fVLM, across multi-label, three-class (CC-CCII), and binary (LUNA16) settings; it provides the most consistent improvements across all.

Limitations

• Conditionality: CARVE only helps when the base model is already discriminative; it cannot recover from failed representation transfer under severe domain shift (e.g., CT-CLIP on RAD-ChestCT).
• Cardinality estimation noise: \(\hat{k}\) depends on prompt-pair probabilities that may be poorly calibrated, requiring hard bounds \([k_\text{min}, k_\text{max}]\).
• Normalization-only adaptation: updating only affine normalization parameters limits the expressiveness of adaptation; more severe shift may require broader parameter updates.
• No positive/negative prompt design ablation: the method inherits the prompt templates from base models; prompt quality is a hidden dependency not controlled in evaluation.
• Fixed hyperparameters across cohorts (\(V=8\), \(\rho=0.25\), \(\lambda_\text{neg}=0.8\)) may not be optimal for all target distributions.

Relevance to Vision-Language Models

This work directly targets the deployment gap of 3D CT VLMs: zero-shot models built on contrastive image-text pretraining degrade under scanner/protocol shift, yet standard VLM TTA methods (TPT, RLCF) assume softmax multi-class outputs that are incompatible with prompt-pair Bernoulli scoring. CARVE introduces a principled adaptation objective specifically for the prompt-pair paradigm used by CT-CLIP, fVLM, and similar architectures, making it a direct contribution to the deployability of medical VLMs. The diagnostic framing — separating input pipeline preconditions from objective design — provides a reusable analysis framework applicable to any domain where VLMs are adapted without labels or source data.