Topology-Driven Transferability Estimation of Medical Foundation Models for Segmentation

🕒 Published (v1): 2026-02-27 11:04 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Selecting the best pre-trained medical foundation model for a segmentation task requires expensive exhaustive fine-tuning across the model zoo. This paper proposes a training-free transferability estimation framework that uses graph-theoretic topology (Minimum Spanning Trees) rather than distributional statistics to predict post-fine-tuning segmentation performance. On the OpenMind benchmark it achieves ~31% relative improvement in weighted Kendall's τ over the best prior method.

Problem

Existing transferability estimation (TE) metrics were designed for classification and rely on global statistical assumptions (Gaussianity, linear separability) that misalign with the SSL-encoder + segmentation-decoder paradigm. They fail to capture the local geometric structure near anatomical boundaries that governs segmentation quality, yielding rankings inconsistent with actual fine-tuning outcomes, especially under distribution shift.

Method

The framework operates in three components, all training-free:

1. Global Representation Topology Divergence (GRTD): Constructs a Minimum Spanning Tree (MST) in the encoder's feature space (edge weights = L2 distances) and a second MST in semantic label space (intra-class edges forced to 0, inter-class edges capped at penalty λ). GRTD is the negative discrepancy between total MST weights across D decoder layers — smaller discrepancy (score closer to 0) indicates the feature manifold respects semantic boundaries.

2. Local Boundary-Aware Topological Consistency (LBTC): Detects anatomical boundary anchors via morphological gradient of ground-truth masks, extracts local patches around each anchor, and builds a local MST per patch. The "topological leakage rate" ρ_k counts the fraction of MST edges that cross class boundaries; LBTC = 1 − mean(ρ_k) over N sampled boundary patches, evaluated on the final encoder layers.

3. Task-Adaptive Fusion: A sigmoid-gated convex combination S_ϕ = α·N(GRTD) + (1−α)·N(LBTC), where α = σ(γ·log|C| + β) and |C| is the number of semantic classes. High class count (complex anatomy) weights GRTD; low class count (focal pathology) weights LBTC. Features are extracted via a randomly initialized nnU-Net decoder with stratified sampling to handle class imbalance.

Key Contributions

• Training-free TE metric specifically designed for SSL foundation model selection in medical segmentation.
• GRTD: MST-based global manifold isomorphism score between feature and label topologies.
• LBTC: Local boundary leakage metric focusing on the high-frequency transition zones where segmentation most often fails.
• Task-adaptive fusion gated by semantic cardinality, generalizing across multi-organ and focal-lesion regimes.
• Demonstration that topology-driven scoring substantially outperforms all prior statistical TE methods on a large 3D medical benchmark.

Results

• Primary metric: Weighted Kendall's τ on OpenMind benchmark (7 SSL models, 114k 3D volumes, 6 segmentation tasks).
• Ours avg τ = 0.723 vs. CCFV (best prior) = 0.552 — ~31% relative improvement.
• Per-task: MSF 0.705, ISL 0.814, HNT 0.905, TPC 0.664, ACD 0.671, KIT 0.578.
• CCFV degrades severely on OOD cross-modality task KIT (τ = 0.180); ours achieves 0.578.
• LogME and LEEP show negative correlations across most tasks, confirming inadequacy for dense prediction.
• Robustness: τ variance across Kaiming/Xavier/Gaussian decoder initializations is small (avg 0.680/0.659/0.674).
• Speed: Scoring all 7 models takes ~6–10 min (vs. 3000+ min for exhaustive fine-tuning; CCFV takes 22–1050 min depending on dataset).
• Ablation: GRTD alone avg 0.436; LBTC alone avg 0.475; adaptive fusion avg 0.714.

Limitations

• Gating hyperparameters γ and β in the task-adaptive fusion are not learned end-to-end; selection details are underspecified.
• The pilot set of K=10 cases used to estimate gating weights may be insufficient for highly heterogeneous datasets.
• Evaluated exclusively on 3D brain/head-and-neck MRI pre-trained models (ResEnc-L backbone, OpenMind zoo); generalizability to other pre-training domains (e.g., pathology, radiology reports, 2D modalities) is untested.
• MST construction scales as O(N² log N) in sample count; computational cost at large N (full dataset) is not analyzed.
• Only reconstruction and contrastive SSL paradigms are tested; masked-language-style or multi-modal pre-training are absent.

Relevance to Foundation Models in Medicine

This work directly addresses the practical deployment bottleneck of the growing zoo of medical SSL foundation models: which model to use for a new segmentation task without re-training all candidates. It introduces a principled, topology-aware selection criterion that is agnostic to the specific SSL pre-training objective (MAE, contrastive, etc.), making it broadly applicable as the field produces ever more specialist and generalist encoders. The finding that statistical TE metrics from classification fail catastrophically for segmentation is an important negative result that should inform future benchmark design for medical foundation model evaluation. For researchers tracking foundation models in medicine, this provides a practical tool for efficient model selection and a new lens—manifold topology rather than feature statistics—for understanding what makes a representation medically useful.