Model Merging for Medical LVLMs: A Benchmark and a Winner-Take-All Approach

• URL: http://arxiv.org/abs/2607.15661v1
• Date: Fri, 17 Jul 2026 06:13:13 GMT
• Title: Model Merging for Medical LVLMs: A Benchmark and a Winner-Take-All Approach
• Abstract summary: Large vision-language models (LVLMs) can be adapted to specialized medical imaging tasks.<n> deploying multiple expert LVLMs in practice incurs substantial computational and operational overhead.<n>We present the first systematic study of model merging for medical LVLMs.
• Score: 33.18889550881221
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large vision-language models (LVLMs) can be adapted to specialized medical imaging tasks via parameter-efficient fine-tuning approaches such as low-rank adaptation (LoRA), leading to a growing ecosystem of expert models tailored to specific imaging modalities and clinical scenarios. However, deploying multiple expert LVLMs in practice incurs substantial computational and operational overhead. Model merging provides a promising solution by consolidating multiple experts into a single model without retraining, yet it remains largely unexplored in the medical domain. In this work, we present the first systematic study of model merging for medical LVLMs. We introduce MergeMedBench, a comprehensive benchmark spanning eight imaging modalities and diverse clinical task types, comprising 16 LoRA fine-tuned models built upon two mainstream architectures. We conduct an extensive evaluation of existing merging methods and further propose winner-take-all, a simple and hyperparameter-free approach that retains only the most dominant parameters across expert models. By preserving the critical parameters that govern model behavior and discarding weaker ones, our method avoids the information dilution inherent in averaging- or alignment-based strategies. Despite its simplicity, winner-take-all consistently outperforms existing approaches, offering both a new perspective on LoRA merging and a strong practical baseline for future research.

Related papers

• MedSAM-Agent: Empowering Interactive Medical Image Segmentation with Multi-turn Agentic Reinforcement Learning [53.37068897861388]
  MedSAM-Agent is a framework that reformulates interactive segmentation as a multi-step autonomous decision-making process.<n>We develop a two-stage training pipeline that integrates multi-turn, end-to-end outcome verification.<n>Experiments across 6 medical modalities and 21 datasets demonstrate that MedSAM-Agent achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-02-03T09:47:49Z)
• MMedExpert-R1: Strengthening Multimodal Medical Reasoning via Domain-Specific Adaptation and Clinical Guideline Reinforcement [63.82954136824963]
  Medical Vision-Language Models excel at perception tasks with complex clinical reasoning required in real-world scenarios.<n>We propose a novel reasoning MedVLM that addresses these challenges through domain-specific adaptation and guideline reinforcement.
  arXiv  Detail & Related papers  (2026-01-16T02:32:07Z)
• MedAlign: A Synergistic Framework of Multimodal Preference Optimization and Federated Meta-Cognitive Reasoning [52.064286116035134]
  We develop MedAlign, a framework to ensure visually accurate LVLM responses for Medical Visual Question Answering (Med-VQA)<n>We first propose a multimodal Direct Preference Optimization (mDPO) objective to align preference learning with visual context.<n>We then design a Retrieval-Aware Mixture-of-Experts (RA-MoE) architecture that utilizes image and text similarity to route queries to a specialized and context-augmented LVLM.
  arXiv  Detail & Related papers  (2025-10-24T02:11:05Z)
• Merge and Guide: Unifying Model Merging and Guided Decoding for Controllable Multi-Objective Generation [49.98025799046136]
  We introduce Merge-And-GuidE, a two-stage framework that leverages model merging for guided decoding.<n>In Stage 1, MAGE resolves a compatibility problem between the guidance and base models.<n>In Stage 2, we merge explicit and implicit value models into a unified guidance proxy, which then steers the decoding of the base model from Stage 1.
  arXiv  Detail & Related papers  (2025-10-04T11:10:07Z)
• MedSAMix: A Training-Free Model Merging Approach for Medical Image Segmentation [21.766481181140527]
  We propose MedSAMix, a training-free model merging method for medical image segmentation.<n>We show that MedSAMix consistently improves performance in both domain-specific accuracy and generalization.<n>For clinical applications, we develop two regimes to meet the demand of domain-specificity and generalizability.
  arXiv  Detail & Related papers  (2025-08-14T19:35:57Z)
• Multimodal Masked Autoencoder Pre-training for 3D MRI-Based Brain Tumor Analysis with Missing Modalities [0.0]
  BM-MAE is a masked image modeling pre-training strategy tailored for multimodal MRI data.<n>It seamlessly adapts to any combination of available modalities, extracting rich representations that capture both intra- and inter-modal information.<n>It can quickly and efficiently reconstruct missing modalities, highlighting its practical value.
  arXiv  Detail & Related papers  (2025-05-01T14:51:30Z)
• MedMAP: Promoting Incomplete Multi-modal Brain Tumor Segmentation with Alignment [20.358300924109162]
  In clinical practice, certain modalities of MRI may be missing, which presents a more difficult scenario. Knowledge Distillation, Domain Adaption, and Shared Latent Space have emerged as commonly promising strategies. We propose a novel paradigm that aligns latent features of involved modalities to a well-defined distribution anchor as the substitution of the pre-trained model.
  arXiv  Detail & Related papers  (2024-08-18T13:16:30Z)
• Large Language Model Distilling Medication Recommendation Model [58.94186280631342]
  We harness the powerful semantic comprehension and input-agnostic characteristics of Large Language Models (LLMs)<n>Our research aims to transform existing medication recommendation methodologies using LLMs.<n>To mitigate this, we have developed a feature-level knowledge distillation technique, which transfers the LLM's proficiency to a more compact model.
  arXiv  Detail & Related papers  (2024-02-05T08:25:22Z)
• Ambiguous Medical Image Segmentation using Diffusion Models [60.378180265885945]
  We introduce a single diffusion model-based approach that produces multiple plausible outputs by learning a distribution over group insights. Our proposed model generates a distribution of segmentation masks by leveraging the inherent sampling process of diffusion. Comprehensive results show that our proposed approach outperforms existing state-of-the-art ambiguous segmentation networks.
  arXiv  Detail & Related papers  (2023-04-10T17:58:22Z)
• Few-shot segmentation of medical images based on meta-learning with implicit gradients [0.48861336570452174]
  We propose to exploit an optimization-based implicit model agnostic meta-learning iMAML algorithm in a few-shot setting for medical image segmentation. Our approach can leverage the learned weights from a diverse set of training samples and can be deployed on a new unseen dataset.
  arXiv  Detail & Related papers  (2021-06-06T19:52:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.