Skip to content

MedForge: Building Medical Foundation Models Like Open Source Software Development

🕒 Published (v1): 2025-02-22 03:19 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MedForge proposes a collaborative, bottom-up framework for building medical foundation models by merging task-specific LoRA modules contributed asynchronously by distributed clinical centers, analogous to open-source software development. Branch contributors share only lightweight LoRA modules and privacy-preserving distilled datasets—never raw patient data—enabling multi-center cooperation without synchronization requirements. Two merging strategies (Fusion and Mixture) incrementally accumulate multi-task clinical knowledge into a shared CLIP-based model.

Problem

Top-down medical FM development is bottlenecked by data centralization: institutions cannot safely share raw patient data, and federated learning requires synchronous coordination that is impractical in real clinical settings. Existing model merging methods (e.g., LoRAHub, ModelSoup) either require raw data access, lack incremental update capability, or cannot handle the continuous, asynchronous contribution patterns of the medical community.

Method

MedForge mirrors open-source Git workflows. Each medical center (branch contributor) locally fine-tunes a LoRA plugin module on its private data and separately synthesizes a small distilled dataset via distribution matching (DM) with Differentiable Siamese Augmentation (DSA); the distilled images obfuscate patient-identifiable visual features. The frozen base module is CLIP ViT-B/16; LoRA is applied to query/value matrices and the visual projection. Two merging strategies operate on pushed LoRA modules and distilled datasets:

  • MedForge-Fusion: Weighted parameter summation of the current main LoRA and the incoming branch LoRA (θ_i = (w_i A_{i-1} + w'_i A'_i)(w_i B_{i-1} + w'_i B'_i)), producing an updated single main plugin module.
  • MedForge-Mixture: Coefficients are optimized per branch; merged output is a weighted sum of individual branch LoRA outputs (y_i = w_i A_{i-1}B_{i-1}x + w'_i A'_i B'_i x), preserving original parameter structures.

Coefficient optimization minimizes cross-entropy + L1 regularization on the distilled dataset using the gradient-free Shiwa/Cobyla optimizer.

Key Contributions

  • A Git-inspired asynchronous collaborative workflow for incremental medical FM development without raw data sharing.
  • Dataset distillation (DM+DSA) as a privacy-preserving communication medium between branch contributors and the main branch.
  • Two LoRA merging strategies: MedForge-Fusion (parameter-space weighted fusion) and MedForge-Mixture (output-space weighted aggregation), the latter offering better robustness.
  • Empirical demonstration of order-robustness in sequential multi-task merging across three histopathology cancer classification datasets.

Results

  • MedForge-Mixture (LoRA) achieves average ACC 0.909 / AUC 0.941 across BreakHis (breast), LC25000 (lung), and MedFMC-Colon datasets.
  • Outperforms all collaborative baselines:
  • LoRAHub w/o distill: avg ACC 0.830 / AUC 0.861
  • LoRAHub w/ distill: avg ACC 0.819 / AUC 0.912
  • ModelSoup: avg ACC 0.713 / AUC 0.879
  • Upper-bound single-task LoRA supervised: avg ACC 0.963 / AUC 0.972; MedForge-Mixture closes ~65% of the gap versus ModelSoup.
  • MedForge-Mixture is robust to merging order (3 permutations tested; avg AUC range 0.931–0.941); MedForge-Fusion shows higher order sensitivity.
  • DM+DSA distillation outperforms DC+DSA in MedForge-Mixture (avg AUC 0.941 vs 0.919).
  • Results replicate with DoRA substituted for LoRA: MedForge-Mixture with DoRA achieves avg ACC 0.906 / AUC 0.937, still outperforming all baselines.

Limitations

  • Evaluated exclusively on 2D histopathology classification (three cancer types); generalization to radiology (CT, MRI), multi-modal tasks, or regression/segmentation is unverified.
  • Distilled datasets still require transmission to the main branch, introducing non-zero leakage risk (though visual obfuscation is argued but not formally quantified via differential privacy or membership inference attacks).
  • MedForge-Fusion is sensitive to contributor ordering, limiting reliability in truly arbitrary asynchronous deployments.
  • Coefficient optimization via gradient-free Shiwa/Cobyla with 40 iterations may not scale to large contributor pools or high-dimensional LoRA ranks.
  • Experiments simulate multi-center scenarios using public datasets on a single machine; true federated deployment latency, heterogeneous data distributions, and network costs are not measured.
  • Gap to single-task supervised upper bound remains substantial on some tasks (e.g., BreakHis ACC 0.803 vs 0.914).

Relevance to Foundation Models in Medicine

MedForge directly addresses a critical deployment barrier for medical FMs: the inability to pool distributed clinical data due to privacy regulations. By framing LoRA module merging as open-source-style contribution, it offers a practical alternative to federated learning that removes synchronization constraints—a key friction point in multi-center AI development. The use of dataset distillation as a privacy-preserving surrogate dataset is a novel bridge between model merging and privacy-safe collaboration, relevant to any scenario where CLIP-style or adapter-based FMs need multi-institutional specialization. This work extends the growing literature on parameter-efficient adaptation and model merging (LoRAHub, ModelSoup) into the privacy-constrained medical domain, pointing toward community-driven FM improvement pipelines akin to Hugging Face model hubs.