TCLA: Training-Free Class-wise Logit Adaptation for Medical Vision-Language Models

🕒 Published (v1): 2026-07-10 16:06 UTC · Source: Arxiv · link

Why this paper was selected

Training-free class-wise logit adaptation for OOD medical VLMs; lightweight, reproducible, practical

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Medical VLMs (CLIP-based models pretrained on biomedical image-text pairs) suffer OOD performance drops due to domain shift and class bias, and existing few-shot adaptation methods are unstable at 1-shot. TCLA corrects zero-shot logits at inference time using class-wise, multi-layer prototypes from a small support set — no gradient updates, no learnable parameters. On nine medical imaging datasets spanning five modalities, TCLA matches or outperforms both training-free and training-based baselines across 1–16 shot settings.

Problem

Pretrained medical VLMs (e.g., BioMedCLIP) degrade on OOD data because their large-scale pretraining distributions differ from target clinical datasets in scanner type, hospital, or patient demographics. Few-shot adaptation methods address this, but training-based approaches (prompt learning, feature adapters, trainable logit correction) are brittle in the 1-shot regime where parameter optimization is unreliable. A training-free alternative that is stable, fast, and model-agnostic is lacking.

Method

TCLA operates entirely at inference time via three components:

CLAP (Class-wise Layer-Adaptive Prototype): For each class \(c\) and visual encoder layer \(l\), a Mahalanobis-weighted prototype \(p_c^{(l)}\) is computed from support features. Inter-class separability \(\text{score}^{(l)} = 1 - \bar{s}^{(l)}\) (where \(\bar{s}^{(l)}\) is mean pairwise cosine similarity of class prototypes) selects the top-\(K\) discriminative layers \(L^*\) with normalized aggregation weights \(\omega^{(l)}\). For a test image \(x\), a sharpened prototype correction basis is computed as \(K_{\text{sharp}}^{(l)}(x) = \exp(\beta A^{(l)}(x)) - 1\), where \(A^{(l)}(x) = f^{(l)}(x)(P^{(l)})^\top\).

PDA (Prompt Distribution Alignment): Text prototypes for each class are reweighted across prompt templates by their compatibility with projected support image features at each selected layer, then aggregated using CLAP weights. This replaces generic prompt averaging with a support-image-guided text prototype.

RLC (Prototype-guided Residual Logit Correction): The support residual \(R_S = Y_S - Z^{\text{ZS}}_S\) (difference between one-hot labels and zero-shot logits on support images) is predicted via ridge regression on the prototype correction basis: $\(W_m = \left((K_S^m)^\top K_S^m + \lambda I\right)^{-1} (K_S^m)^\top R_S\)$ A scalar magnitude coefficient \(\eta_m\) calibrates residual strength. Final logits are: $\(Z_m^{\text{TCLA}}(x) = Z^{\text{ZS}}(x) + \mu_m(s)\,\eta_m\,K_m(x)\,W_m\)$ Two modes are defined: aggressive (pure sharp kernel) and conservative (interpolated with soft kernel weighted by per-class prototype reliability \(r_c^{(l)}\), which diminishes for 1-shot). CT and MRI use conservative mode; all others use aggressive mode.

Key Contributions

• A fully training-free few-shot logit adaptation framework for medical VLMs requiring no gradient updates, new parameters, or validation sets.
• CLAP: layer selection by inter-class prototype separability using Mahalanobis-weighted prototypes, generalizing beyond last-layer features.
• PDA: support-image-guided reweighting of text prompt templates to produce aligned text prototypes.
• RLC: closed-form ridge regression from prototype kernel responses to zero-shot logit residuals, with a scalar magnitude calibration step.
• Two correction modes (aggressive/conservative) tied to modality, with reliability-weighted interpolation that naturally degrades to conservative behavior at 1-shot.
• Evaluation across nine datasets, five modalities (X-ray, CT, MRI, ultrasound, histopathology), 1–16 shots, averaged over 50 random seeds.

Results

All figures are balanced accuracy (BACC) on BioMedCLIP as backbone, averaged over 50 runs.

• vs. training-free baselines (Table 1): TCLA achieves best BACC on most datasets/shot settings. At 16-shot: COVID 72.21 (vs. ProKeR 66.30), CXRP 93.48 (vs. ProKeR 89.20), TBSZ 81.70 (vs. ProKeR 76.11), LC25k 93.17 (vs. ProKeR 89.74). ProKeR is competitive on CTKi and SCCT.
• vs. training-based baselines (Table 2): TCLA equals or exceeds LP, CoOp, BioMedCoOp, Tip-Adapter-F in most settings. At 16-shot, LDC outperforms TCLA on CTKi (73.10 vs. 71.41) and SCCT (83.62 vs. 79.14) and LC25k (93.98 vs. 93.17), but TCLA wins on COVID, CXRP, TBSZ, BUSI.
• 1-shot extreme regime: TCLA outperforms LDC on X-ray, CT, and histopathology modalities; overall the most stable method across modalities.
• Ablation (1-shot, Table 3): Full TCLA (CLAP+PDA+RLC) reduces standard deviation vs. base on CT (±3.16 vs. ±5.27) and X-ray (±4.30 vs. ±5.16), indicating improved consistency. CT gains +5.94 BACC from adding RLC (setting c→d).

Limitations

• The modality-to-mode assignment (aggressive vs. conservative) is hand-specified and not learned from data; this may be suboptimal for edge cases within a modality.
• CLAP's multi-layer strategy showed negligible or slightly negative gains for MRI, suggesting modality-specific behavior that the method does not automatically adapt to.
• Evaluation is confined to BioMedCLIP as the backbone; generalization to other medical VLMs (e.g., PubMedCLIP, BIOMEDICA) is asserted but not empirically demonstrated.
• Ridge coefficient \(\lambda = 1.0\) and sharpness \(\beta\) are fixed hyperparameters; sensitivity analysis is not reported.
• Evaluation is limited to classification tasks; segmentation and report generation — other medical VLM use cases cited in the introduction — are not tested.

Relevance to Vision-Language Models

TCLA addresses a practical deployment bottleneck for CLIP-based medical VLMs: how to reliably adapt zero-shot models to OOD clinical distributions when labeled data is scarce and fine-tuning is unreliable. It positions training-free logit correction as a viable alternative to parameter-efficient fine-tuning methods (LoRA, adapters, prompt learning) in the few-shot regime, a finding relevant to the broader question of when inference-time adaptation suffices. The multi-layer prototype extraction strategy (selecting layers by inter-class separability rather than always using the final representation) is a transferable insight for any CLIP-style model operating on distribution-shifted data. For researchers tracking VLMs, this paper contributes to the growing body of work — alongside Tip-Adapter, APE, and ProKeR — establishing the trade-off frontier between training cost, data efficiency, and OOD robustness in vision-language adaptation.