Skip to content

Model Merging for Medical LVLMs: A Benchmark and a Winner-Take-All Approach

🕒 Published (v1): 2026-07-17 06:13 UTC · Source: Arxiv · link

Why this paper was selected

Medical LVLM model-merging benchmark + WTA method; practical LoRA-expert fusion

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces MergeMedBench, the first systematic benchmark for merging LoRA-finetuned medical large vision-language models (LVLMs), covering 8 imaging modalities and 16 expert models. The authors also propose winner-take-all, a hyperparameter-free merging strategy that selects the most dominant parameter at each position across experts rather than averaging. Despite its simplicity, it consistently outperforms all baselines on the benchmark.

Problem

Deploying multiple LoRA-finetuned expert LVLMs—each specialized for a distinct medical imaging modality (CT, MRI, X-ray, etc.)—incurs high computational and operational overhead. Model merging could consolidate these experts into one model without retraining, but no benchmark or systematic study existed for this setting, and existing merging methods (designed for general NLP/CV) rely on averaging or subspace alignment that perturbs the small subset of parameters that critically govern task performance.

Method

The authors observe empirically that a small subset of LoRA parameters (dominant parameters, identified by normalized magnitude) disproportionately controls model behavior: rescaling them by even modest factors causes large accuracy swings, while rescaling non-dominant parameters has negligible effect.

Winner-take-all exploits this: for each position \((i,j)\) in each low-rank matrix \(\Phi \in \{A, B\}\) at layer \(l\) and weight matrix \(M\), it selects the parameter from the expert \(n^*\) with the highest normalized dominance score:

\[S_{l,M,i,j}^{(\Phi,n)} = \frac{\Phi_{l,M}^{(n)}[i,j] - \mathbb{E}[\Phi^{(n)}]}{\sqrt{\text{Var}[\Phi^{(n)}] + \epsilon}}\]
\[n^{*(\Phi)}_{l,M,i,j} = \arg\max_{n} S_{l,M,i,j}^{(\Phi,n)}\]

The merged parameter is taken directly from the winning expert with no rescaling:

\[\Phi^{\text{merged}}_{l,M}[i,j] = \Phi_{l,M}^{(n^*)}[i,j]\]

Z-score normalization across each expert's parameter distribution corrects for scale differences across independently trained LoRA modules before comparison. The merged LoRA update is then \(\Delta W^{\text{merged}}_{l,M} = B^{\text{merged}}_{l,M} A^{\text{merged}}_{l,M}\), applied on top of the frozen base model.

Key Contributions

  • MergeMedBench: first model-merging benchmark for medical LVLMs — 88,995 VQA samples across 8 modalities (CT, MRI, X-ray, ultrasound, dermoscopy, fundus photography, OCT, microscopy), 71,333 train / 17,662 test, 16 LoRA expert models on Qwen3-VL-2B-Instruct and InternVL2-1B.
  • Empirical dominant-parameter analysis: controlled rescaling experiments demonstrating that top-magnitude LoRA parameters are hypersensitive and non-dominant ones are near-insensitive.
  • Winner-take-all merging: hyperparameter-free, gradient-free method that preserves dominant parameters intact via position-wise expert selection with normalized magnitude scoring.
  • Extensive benchmark: evaluation of 13 baseline configurations (TA, TIES-Merging, DARE-TIES, Iso-C, Iso-CTS, STF, RobustMerge, KnOTS, KnOTS-TIES, KnOTS-DARE-TIES, Core, Core-TIES, Core-DARE-TIES) plus ablations on winner selection rank \(k\), normalization scheme, and subspace integration.

Results

  • Qwen3-VL-2B-Instruct: winner-take-all achieves 90.88% average normalized accuracy, vs. best baseline STF at 89.81% and TIES-Merging at 86.91%; simple averaging (TA) scores only 75.08%.
  • InternVL2-1B: winner-take-all achieves 84.64%, vs. best baseline STF at 82.91%; TA at 65.94%.
  • Winner-take-all clearly outperforms parameter averaging (90.88% vs. 72.18% on Qwen-VL).
  • Runtime is 0.21 s (Qwen-VL), faster than most baselines; subspace methods range from 64–650 s.
  • Combining winner-take-all with Core raises Core's accuracy by +14.55%; with KnOTS by +7.69% (Qwen-VL).
  • Winner selection at \(k=1\) dominates; accuracy degrades monotonically as \(k\) increases to 8.
  • Normalized z-score scoring outperforms unnormalized (+1.75% on InternVL2), min-max, and RMS variants.

Limitations

  • Evaluated on a single dataset (OmniMedVQA); generalization to other medical VQA distributions is untested.
  • Only two base LVLM architectures (Qwen3-VL-2B-Instruct, InternVL2-1B) are studied; behavior on larger-scale models (7B+) is unknown.
  • The dominance criterion is magnitude-based (gradient-free); gradient-based scoring is discussed but not empirically evaluated.
  • Winner-take-all selects independently at each parameter position, potentially breaking cross-position correlations within LoRA factors \(A\) and \(B\).
  • No evaluation of merging beyond 8 experts or on tasks requiring compositional cross-modality reasoning.

Relevance to Vision-Language Models

This work directly addresses efficient multi-task deployment of LoRA-finetuned LVLMs, a core practical challenge as VLM ecosystems accumulate many modality-specialized expert models. The finding that a small fraction of LoRA parameters critically governs task behavior is a transferable insight for understanding parameter importance in VLM adaptation more broadly. The proposed winner-take-all strategy offers a zero-overhead, hyperparameter-free alternative to existing merging methods, relevant to any VLM finetuning pipeline using LoRA. MergeMedBench also fills a notable gap as the first rigorous testbed for evaluating VLM merging in a high-stakes domain.