Skip to content

Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation

🕒 Published (v1): 2026-07-20 08:41 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; memory-synergistic TTA for medical VLM segmentation, classification→seg bridge

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MSSA is a training-free test-time adaptation (TTA) framework for medical image segmentation that avoids updating VLM parameters — which degrades pretrained image–text alignment — and instead builds an online memory of high-quality predictions to guide subsequent segmentation via image-to-image prototype matching. It synergistically combines semantic grounding (image–text) with structural consistency (image–image) to achieve robust adaptation under domain shift. MSSA outperforms all fine-tuning-based TTA baselines by up to +12.2% DSC without any gradient updates.

Problem

Fine-tuning VLMs at test time with noisy pseudo-labels disrupts the pretrained image–text alignment space, causing unstable optimization and often negative performance compared to zero-shot inference. Existing TTA methods designed for Vision Foundation Models (VFMs) transfer poorly to VLM-based segmentation pipelines. No prior work had produced a training-free TTA strategy for medical image segmentation capable of leveraging VLM generalization without parameter corruption.

Method

MSSA operates in two sequential stages, both entirely parameter-update-free:

Stage 1 — Image-Text Coarse Grounding. The pipeline builds on MedCLIP-SAMv2 (BiomedCLIP + SAM). GPT-4 generates descriptive text prompts; BiomedCLIP produces a saliency map via M2IB. Two stabilization mechanisms improve robustness: (i) Gaussian Point Selection samples SAM foreground prompts weighted by \(\text{GM}(x,y) = \exp\!\left(-\frac{(x-x_0)^2+(y-y_0)^2}{2\sigma^2}\right)\) to favour centroid-proximal stable points; (ii) dual majority voting aggregates photometric-augmented saliency maps and geometric-augmented SAM outputs to yield a stable candidate mask \(Y_\text{cand}\).

Stage 2 — Noise-Aware Memory Construction + Prototype Matching. Candidates are scored by \(\mathcal{Q}_\text{total} = \mathcal{Q}_\text{sem} + \mathcal{Q}_\text{smo}\), where \(\mathcal{Q}_\text{sem} = \text{BiomedCLIP}(I_\text{masked}, T)\) measures semantic alignment and \(\mathcal{Q}_\text{smo}\) combines shape regularity and boundary smoothness. Admission to the memory bank \(\mathcal{B}\) requires \(\mathcal{Q}_\text{total}^t \ge \tau_t\), where \(\tau_t\) is an adaptive, monotonically non-decreasing percentile threshold (\(P=80\)) over the historical score buffer \(\mathcal{H}_t\). Memory entries are (DINOv2 feature \(F_j\), mask \(Y_j\)) pairs stored under FIFO up to 15 slots. At query time, the anchor with highest cosine similarity \(s_j = \frac{F_t \cdot F_j}{\|F_t\|\|F_j\|}\) is retrieved; foreground/background prototypes are extracted via sliding-window average pooling (ALPNet-style); and the final mask is decoded by \(\hat{Y}_t = \arg\max \operatorname{softmax}([\tilde{M}^1; \tilde{M}^0])\) where \(\tilde{M}^c = \max_l M_l^c\) is the element-wise max of per-prototype cosine similarity maps.

Key Contributions

  • First training-free TTA framework for VLM-based medical image segmentation, avoiding parameter corruption and error accumulation.
  • Noise-Aware Memory Construction (NMC): dual-criterion scoring (semantic alignment + spatial smoothness) with adaptive percentile thresholding to curate a high-fidelity online memory bank.
  • Relevance-driven image-to-image prototype matching using DINOv2 features and ALPNet-style non-parametric prototyping for parameter-free query segmentation.
  • Gaussian Point Selection for stable SAM prompt generation from coarse VLM saliency.
  • Dual majority voting over photometric and geometric augmentations to stabilize cross-modal predictions.

Results

  • Optic disc segmentation (5 domains: RIM-ONE-r3, REFUGE, ORIGA, REFUGE-Valid, Drishti-GS):
  • Average DSC 83.9%, mIoU 73.1%
  • +12.2% DSC over TTCS (best training-based TTA baseline)
  • +12.5% DSC over MedCLIPSAMv2 (best zero-shot VLM baseline)
  • +17.5% DSC over MedCLIPv2 (the base zero-shot model)
  • All six training-based TTA methods (TENT, EATA, CoTTA, DELTA, SVDP, TTCS) yielded \(\le\)+0.4% DSC gain or negative transfer relative to MedCLIPv2; MSSA achieves +17.5%
  • Lung segmentation (COVID-QU-Ex, MC, SZ datasets): results appear in Table 2 but are truncated in the provided text; quantitative headline numbers not fully available.

Limitations

  • Memory bank is capped at 15 samples; performance may degrade if the target stream is highly heterogeneous or if the warm-up window (\(N_\text{warm}=10\)) yields no high-scoring candidates.
  • Text prompts are generated offline by GPT-4, introducing a dependency on a closed-weight external model and fixed prompt quality.
  • Prototype matching relies on a single anchor retrieval (top-1 cosine similarity), which may fail when the most similar memory sample is still structurally dissimilar to the query.
  • Evaluation is limited to two task types (optic disc/cup, lung segmentation); generalizability to more complex multi-class or 3D segmentation tasks is not demonstrated.
  • Adaptive thresholding enforces monotonic non-decrease, meaning early noisy warm-up candidates can artificially cap the threshold and lock out later good-quality samples in pathological stream orderings.

Relevance to Vision-Language Models

This paper directly addresses a failure mode of VLMs when deployed under distribution shift: fine-tuning disrupts the image–text alignment that gives VLMs their zero-shot power, making standard TTA counterproductive. MSSA demonstrates that VLM representations are best treated as frozen priors to be exploited, not weights to be updated, which is a broadly applicable insight for deploying CLIP-family models in specialized domains. The memory-bank approach — curating reliable VLM outputs to bootstrap image-to-image consistency — offers a general recipe for training-free adaptation of any VLM-backbone segmentation pipeline. For VLM researchers, the dual-modality complementarity finding (image–text gives semantics; image–image gives structure) motivates hybrid inference-time strategies that could extend beyond medical imaging.