Skip to content

MFM-DA: Instance-Aware Adaptor and Hierarchical Alignment for Efficient Domain Adaptation in Medical Foundation Models

🕒 Published (v1): 2025-03-02 08:54 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MFM-DA is a two-stage few-shot unsupervised domain adaptation framework for medical foundation models (MFMs) that uses only 10 unlabeled target-domain images. It adapts a DDPM to generate target-style images via a dynamic instance-aware adaptor, then fine-tunes the MFM with pyramid hierarchical feature alignment and LoRA. The method addresses the persistent domain gap that MFMs exhibit even after source-domain fine-tuning.

Problem

MFMs fine-tuned on source-domain data still suffer significant performance degradation when applied to target domains due to domain shift. Existing UDA methods require large amounts of unlabeled target data, which is impractical in clinical settings. No prior work had proposed a domain adaptation framework specifically designed for MFMs under few-shot (≤10 image) constraints.

Method

Stage 1 — Image-level adaptation via adapted DDPM: - A DDPM is trained on source-domain data, then adapted to the target domain using only 10 unlabeled target images. - The Dynamic Instance-Aware Adaptor (DIAA) replaces the fixed static cross-domain direction vector (∆static, computed as the CLIP feature centroid difference) with a learned gated fusion: ∆ = g ⊙ ∆dynamic + (1−g) ⊙ ∆static. The dynamic component is a bottleneck MLP conditioned on per-batch source CLIP features, enabling instance-specific adaptation directions. - A distribution consistency loss (LDC) enforces that generated images CLIP(x^{S→T}) align with CLIP(x^S) + ∆, preserving source structure while shifting distribution toward target style.

Stage 2 — Feature-level adaptation of the MFM: - The MFM is fine-tuned on a mix of source images and DDPM-generated pseudo-target images. - Pyramid Hierarchical Alignment computes cosine similarity between source and target features at four pyramid levels (f^(k) ∈ R^{B×C×Hk×Wk}), enforcing position-wise channel and spatial alignment via L_align = (1/n)Σ(1 − Sk). - LoRA matrices (rank r ≪ c) are inserted into each attention layer; only 0.8% of MFM parameters are updated. - Total loss: L_total = L_BCE + L_align.

Key Contributions

  • First framework explicitly targeting few-shot UDA for MFMs (as opposed to general UDA or standard fine-tuning).
  • Dynamic Instance-Aware Adaptor (DIAA): learns per-batch dynamic adaptation directions via a gated static/dynamic fusion, improving coverage of the target distribution over fixed-direction DDC baseline.
  • Pyramid Hierarchical Alignment: multi-scale cosine-similarity loss over four feature levels applied within LoRA-adapted MFM fine-tuning.
  • Demonstrated generalizability across multiple MFM backbones (RETFound, DINOv2, CLIP).

Results

  • Generation quality (10-shot, average over 3 target domains): MFM-DA achieves FID 80.86 and IC-LPIPS 0.354, vs. DDC (FID 108.78, IC 0.300), FreezeD (FID 123.25, IC 0.242), and vanilla Syn (FID 138.65, IC 0.259).
  • Segmentation (Dice, average across 3 target domains, RETFound backbone):
  • RETFound baseline: 80.66%
  • RETFound + Coral: 81.41%; + FDA: 82.99%; + Reins: 81.18%
  • RETFound + MFM-DA (ours): 83.13% (best among RETFound variants)
  • UNet baseline: 76.24%; UNet + FDA: 81.81%
  • Ablation: Using all four pyramid alignment levels (L-All: 84.83 Dice) outperforms any single level (L1–L4: 83.39–84.17).
  • Few-shot robustness: Effective from 1-shot to 10-shot; 3-shot achieves IC 0.356, competitive with 10-shot (0.371).
  • Cross-backbone generalization: CLIP + MFM-DA: 83.46% vs. CLIP baseline 82.52%; DINOv2 + MFM-DA: 80.91% vs. DINOv2 baseline 79.90%.

Limitations

  • Evaluated exclusively on fundus optic cup/disc segmentation (RIGA+ and REFUGE datasets); generalization to other imaging modalities (CT, MRI, pathology) is untested.
  • Requires access to labeled source-domain data, raising privacy concerns when source data cannot be shared.
  • Stage 1 DDPM training is computationally expensive (63 GPU hours on 5× NVIDIA 4090s).
  • Performance improvement margins over best baselines are modest (~0.14 Dice points over FDA on RETFound).

Relevance to Foundation Models in Medicine

MFM-DA directly targets a practical deployment bottleneck for MFMs: domain shift persists even after fine-tuning, and collecting large unlabeled target datasets is often infeasible in clinical settings. The LoRA-based adaptation with only 0.8% parameter updates is well aligned with efficient fine-tuning strategies increasingly adopted in the MFM literature (e.g., SAM, RETFound). By combining generative image-domain transfer with feature-level hierarchical alignment, this work advances the toolkit for adapting task-specific MFMs to new acquisition sites or device settings without requiring labeled target data—a directly actionable technique for multi-site clinical deployment of MFMs.