Skip to content

MedFM-Robust: Benchmarking Robustness of Medical Foundation Models

🕒 Published (v1): 2026-05-18 18:50 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MedFM-Robust is a comprehensive robustness benchmark for medical foundation models covering 40 perturbation types (12 generic, 28 modality-specific) across 8 imaging modalities. It evaluates 5 VLMs on VQA/captioning/grounding and 2 SAM-based segmentation models under 5 fine-tuning strategies. The central finding is that fine-tuning strategy—not model architecture—is the primary determinant of robustness.

Problem

Existing medical AI benchmarks evaluate models on clean, curated images, creating a gap between reported performance and real-world reliability. Medical images are subject to acquisition artifacts (motion blur, metal streaks, MRI ghosting, stain variation) that generic natural-image corruption benchmarks (e.g., ImageNet-C) fail to capture. No prior work systematically evaluates both VLM and segmentation foundation models under domain-specific perturbations with matched fine-tuning strategies.

Method

The authors construct a modality-adaptive perturbation pipeline with: - 12 base perturbations: noise (Gaussian, salt-and-pepper, speckle), degradation (blur, brightness, contrast, JPEG, pixelation), and geometric transforms. - 28 modality-specific perturbations: e.g., CT metal streaks/beam hardening, MRI bias field/ghosting, ultrasound acoustic shadowing, pathology HSV stain variation, OCT blink/defocus, X-ray scatter/grid. - SSIM-guided severity calibration: binary search maps perturbation parameters to 5 severity bands (SSIM 0.50–0.98), ensuring consistent cross-modality degradation comparisons.

VLMs (LLaVA-Med, MedGemma, MedGemma-1.5, Gemini-2.5-flash, GPT-4o-mini) are evaluated zero-shot on VQA (OmniMedVQA, accuracy) and captioning (ROCOv2, BLEU/ROUGE-L/CIDEr), and with LoRA fine-tuning on visual grounding (MeCoVQA, IoU@0.5). Segmentation models (MedSAM, SAM-Med2D) are evaluated across 5 datasets (ISIC dermoscopy, Kvasir endoscopy, Brain Tumor MRI, Glaucoma Disc/Cup) under 5 fine-tuning strategies: full, encoder-only, decoder-only, LoRA, and adapter. Robustness is quantified as absolute metric drop (Δ) averaged over perturbation types per severity level.

Key Contributions

  • First benchmark covering 40 perturbations (12 base + 28 modality-specific) across 8 medical imaging modalities with SSIM-calibrated severity levels.
  • Unified evaluation spanning segmentation (IoU/Dice) and three VLM tasks (VQA, captioning, visual grounding) under matched perturbation conditions.
  • Systematic ablation of five fine-tuning strategies for SAM-based segmentation models, revealing LoRA as consistently the least robust.
  • Empirical deployment guidelines: full fine-tuning for robustness, SAM-Med2D adapter as efficiency-robustness trade-off, MedGemma as most stable medical VLM.

Results

Segmentation: - Full fine-tuning: mean IoU drop 0.025; LoRA: 0.048 (≈2× degradation); Dec-Only and Enc-Only: 0.029 each. - SAM-Med2D Adapter: mean IoU drop 0.036, favorable efficiency-robustness balance. - 9 of top 15 most damaging perturbations are modality-specific (Motion Artifacts OCT, Light Reflection, Specular Reflection, Bubbles, Stain Saturation, MRI Motion Artifacts, Shadow, Speckle Artifact, Blink Artifacts). - Most sensitive dataset: Kvasir endoscopy (IoU drop 0.050); most stable: Brain MRI (0.019). - LoRA severity curve: IoU drop rises from 0.028 (low) to 0.065 (high); full fine-tuning shows the flattest curve.

VLMs: - Visual grounding (LoRA fine-tuned): MedGemma 65.4%→22.3% (−43.1 pts); MedGemma-1.5 69.2%→29.0% (−40.2 pts). Zero-shot general-purpose models: 0–10% (fail entirely). - Zero-shot VQA: Gemini-2.5-flash achieves highest clean accuracy (67.0%) but drops 36.1 pts (54% relative); GPT-4o-mini 50.0%→~42% (12% relative drop); medical VLMs drop <20% relatively. - Captioning: all models drop <0.02 BLEU across all perturbations and severity levels; most resilient task. - Top VQA perturbations: Motion Blur and Gaussian Blur (~8 pt drop); top grounding perturbations: Compression Artifacts (32.9 drop) and Gaussian Noise (32.6 drop).

Limitations

  • VLM evaluation uses only 500 samples per task, which may limit statistical power for low-frequency perturbation types.
  • Visual grounding evaluated only under LoRA fine-tuning; other fine-tuning strategies for VLMs are not ablated.
  • Perturbation severity is SSIM-calibrated but SSIM may not perfectly correlate with clinical impact or diagnostic difficulty across all modalities.
  • No evaluation on prospective or longitudinal clinical data; all datasets are static benchmarks.
  • MedSAM and SAM-Med2D represent only one generation of SAM-based models; newer architectures (e.g., SAM 2) are not included.

Relevance to Foundation Models in Medicine

This work directly addresses a critical gap in medical foundation model evaluation: the disconnect between clean-benchmark performance and deployment-time reliability under realistic acquisition artifacts. For the foundation models in medicine space, the finding that fine-tuning strategy—not model scale or architecture—is the dominant robustness factor has immediate implications for how MedSAM, MedGemma, and similar models should be adapted for clinical use. The result that LoRA, the dominant PEFT approach in clinical adaptation literature, incurs nearly double the robustness degradation of full fine-tuning challenges prevailing deployment assumptions. The benchmark also provides a rare cross-task, cross-modality comparison that exposes qualitatively different failure modes: task-specific fine-tuning (grounding, segmentation) amplifies perturbation sensitivity, while zero-shot inference preserves pretrained stability at the cost of task performance.