Specializing Foundation Models via Mixture of Low-Rank Experts for Comprehensive Head CT Analysis

🕒 Published (v1): 2026-02-28 14:32 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MoLRE (Mixture of Low-Rank Experts) extends LoRA with K specialized low-rank adapters and an unsupervised soft router, enabling input-conditional PEFT for medical imaging foundation models. Benchmarked across six models on 72,756 head CTs with 75 neurological findings, MoLRE consistently improves multi-label detection AUC while adding <0.5% parameters. MedGemma+MoLRE achieves state-of-the-art 0.917 mean AUC across all 75 findings.

Problem

Standard LoRA applies a uniform low-rank weight update regardless of input, creating knowledge interference when a single adaptation capacity must serve highly heterogeneous pathologies (hemorrhage, ischemia, trauma, mass lesions, chronic changes). Comprehensive multi-label head CT analysis—75 findings across six pathology categories—is precisely this setting, yet systematic benchmarking of PEFT strategies across diverse foundation model architectures is lacking.

Method

MoLRE replaces the single LoRA adapter with K=6 parallel low-rank experts (rank r=8 each), combined via a two-layer MLP soft router trained end-to-end with the task loss (no explicit pathology supervision). The adapted output is h = W₀x + Σᵢ gᵢ(x)·ΔWᵢx, where gᵢ(x) = Softmax(W₂·ReLU(W₁x+b₁)+b₂). For 2D models, routing operates at the slice level before attention-weighted pooling aggregates slice features into a volume-level representation. For 3D models (Pillar0-HeadCT), routing is applied to the spatially pooled volumetric embedding. Training uses multi-label focal loss (γ=2.0), prevalence-based class weights, Repeat Factor Sampling for rare findings, and AdamW with separate learning rates for classification heads (1e-3) vs. LoRA/MoLRE parameters (1e-4).

Key Contributions

• MoLRE framework: conditionally routed, multi-expert LoRA extension adding <0.5% parameters, trained fully unsupervised via task loss.
• Largest-scale head CT multi-label benchmark: 72,756 NCCT scans, 75 neurological findings, six foundation models spanning 7M–431M parameters, 2D/3D, general/medical/CT-specific pretraining.
• LLM-assisted label generation: GPT-4-mini auto-labeling from radiology reports, validated at 0.986 finding-level accuracy (0.64 FP and 0.09 FN per case on 90 neuroradiologist-annotated cases).
• Empirical characterization of adaptation benefit as a function of pretraining domain, architecture (2D vs. 3D), and scale.

Results

• MedGemma+MoLRE: 0.917 mean AUC (best overall); baseline MedGemma: 0.874 (+4.3 pp).
• DINOv3-Base+MoLRE: 0.902 vs. 0.856 baseline (+4.6 pp, largest absolute gain).
• DINOv3-Large+MoLRE: 0.913 vs. 0.910 (+0.3 pp; diminishing returns at larger scale).
• MedImageInsight+MoLRE: 0.876 vs. 0.863 (+1.3 pp).
• Pillar0-HeadCT (3D)+MoLRE: 0.893 vs. 0.891 (+0.2 pp; minimal gain from volume-level routing).
• For DINOv3-Base, MoLRE increases findings with AUC ≥ 0.90 from 33 to 43; for MedGemma, from 35 to 48.
• Largest per-finding gains for visually subtle or rare entities: early ischemic signs, venous sinus thrombosis, lipomas, occult bone lesions.

Limitations

• MoLRE routing is slice-level for 2D models but collapses to volume-level for 3D models, eliminating spatial heterogeneity that drives effective expert specialization; this explains the near-zero gains on Pillar0-HeadCT.
• DeepCNTD (6.9M, smallest model) is excluded from MoLRE evaluation due to absence of attention pooling, leaving the small-model regime uncharacterized.
• Labels are automatically generated by a private GPT-4-mini model; while validated at 0.986 accuracy, this introduces potential systematic errors and prevents external label reproducibility.
• Dataset is non-contrast head CT only (Siemens, GE, Canon scanners, nine centers); generalizability to contrast-enhanced CT or other scanner ecosystems is untested.
• Research prototype; not commercially available.

Relevance to Foundation Models in Medicine

MoLRE directly addresses a core bottleneck in clinical deployment of medical foundation models: how to efficiently specialize a single pretrained backbone for high-dimensional, heterogeneous multi-label diagnostic tasks without full fine-tuning. The systematic benchmark across pretraining strategy (general vs. medical vs. task-specific), architecture (2D vs. 3D), and scale (7M–431M) provides actionable guidance—general-domain models gain most from MoE routing, while task-specific 3D models show diminishing returns. The finding that MedGemma (generative pretraining) benefits more than MedImageInsight (discriminative pretraining) challenges assumptions about which pretraining objective best prepares features for downstream clinical classification. This work contributes both a practical PEFT technique and a rigorous benchmarking methodology that the field needs as foundation models proliferate in radiology.