MedMIX: Modality-Internal Expert Fusion for Multimodal Medical Diagnosis

• URL: http://arxiv.org/abs/2605.16639v1
• Date: Fri, 15 May 2026 21:12:17 GMT
• Title: MedMIX: Modality-Internal Expert Fusion for Multimodal Medical Diagnosis
• Abstract summary: We introduce Medmix, a multimodal framework that combines intra-modality expert fusion, learned inter-modality fusion, and training-only large--small model collaboration.<n>Across three benchmarks, Medmix consistently achieves strong performance while remaining robust under controlled missing-modality perturbations.<n>These results highlight Medmix as a practical framework that unifies within-modality expert collaboration, sample-specific cross-modality fusion, and efficient large--small model collaboration.
• Score: 3.2131549954795453
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Multimodal clinical prediction faces three challenges: multiple foundation models (FMs) with complementary strengths per modality, pervasive missing modalities at training and test time, and sample-specific variation in modality contributions. We introduce MedMIX, a multimodal framework that combines intra-modality expert fusion, learned inter-modality fusion, and training-only large--small model collaboration for robust medical prediction under incomplete modalities. Within each modality, MedMIX aggregates complementary embeddings from multiple small expert models; across modalities, it performs learned fusion over available modalities; and during training, it leverages large teacher models to improve deployed representations without additional inference cost. Across three heterogeneous benchmarks (OpenI, MIMIC-IV-MM, and MMIST-ccRCC), MedMIX achieves consistently strong performance while remaining robust under controlled missing-modality perturbations, and further demonstrates sustained robustness under cross-cohort shift on MIMIC-III. These results highlight MedMIX as a practical framework that unifies within-modality expert collaboration, sample-specific cross-modality fusion, and efficient large--small model collaboration while remaining robust to incomplete modalities.

Related papers

• Vision-Core Guided Contrastive Learning for Balanced Multi-modal Prognosis Prediction of Stroke [4.272151451487239]
  This paper proposes a novel tri-modal fusion model for ischemic stroke prognosis.<n>We employ a Large Language Model (LLM) to automatically generate semi-structured diagnostic text from brain MRIs.<n>We also design a core component termed the Vision-Conditioned Dual Alignment Fusion Module.
  arXiv  Detail & Related papers  (2026-05-14T11:28:57Z)
• M-IDoL: Information Decomposition for Modality-Specific and Diverse Representation Learning in Medical Foundation Model [53.96788246923603]
  We propose M-IDoL, a self-supervised underlinetextitMFM that introduces Information Decomposition for multimodal representation Learning.<n>By pre-training on 1.15 million medical images, M-IDoL i) delivers superior generalization across 21 downstream clinical tasks.
  arXiv  Detail & Related papers  (2026-04-10T04:06:11Z)
• REMIND: Rethinking Medical High-Modality Learning under Missingness--A Long-Tailed Distribution Perspective [10.304297174213293]
  We propose a group-specialized Mixture-of-Experts architecture that learns group-specific multi-modal fusion functions for arbitrary modality combinations.<n>Our core idea is to propose a novel group-specialized Mixture-of-Experts architecture that scalably learns group-specific multi-modal fusion functions for arbitrary modality combinations.
  arXiv  Detail & Related papers  (2026-02-09T10:16:37Z)
• Cross-Modal Alignment via Variational Copula Modelling [54.25504956780864]
  It is essential to develop multimodal learning methods to aggregate various information from multiple modalities.<n>Existing methods mainly rely on concatenation or the Kronecker product, oversimplifying the interaction structure between modalities.<n>We propose a novel copula-driven multimodal learning framework, which focuses on learning the joint distribution of various modalities.
  arXiv  Detail & Related papers  (2025-11-05T05:28:28Z)
• ProbMed: A Probabilistic Framework for Medical Multimodal Binding [21.27709522688514]
  We present Probabilistic Modality-Enhanced Diagnosis (ProbMED)<n>ProbMED aligns four distinct modalities--chest X-rays, electrocardiograms, echocardiograms--into a unified probabilistic embedding space.<n>Our model outperforms current medical vision-language pretraining models in cross-modality retrieval, zero-shot, and few-shot classification.
  arXiv  Detail & Related papers  (2025-09-30T03:16:01Z)
• impuTMAE: Multi-modal Transformer with Masked Pre-training for Missing Modalities Imputation in Cancer Survival Prediction [75.43342771863837]
  We introduce impuTMAE, a novel transformer-based end-to-end approach with an efficient multimodal pre-training strategy.<n>It learns inter- and intra-modal interactions while simultaneously imputing missing modalities by reconstructing masked patches.<n>Our model is pre-trained on heterogeneous, incomplete data and fine-tuned for glioma survival prediction using TCGA-GBM/LGG and BraTS datasets.
  arXiv  Detail & Related papers  (2025-08-08T10:01:16Z)
• MedPatch: Confidence-Guided Multi-Stage Fusion for Multimodal Clinical Data [0.46040036610482665]
  Real-world medical data is heterogeneous in nature, limited in size, and sparse due to missing modalities.<n>Inspired by clinical prediction tasks, we introduce MedPatch, which seamlessly integrates multiple modalities via confidence-guided patching.<n>We evaluate MedPatch using real-world data consisting of clinical time-series data, chest X-ray images, radiology reports, and discharge notes extracted from the MIMIC-IV, MIMIC-CXR, and MIMIC-Notes datasets.
  arXiv  Detail & Related papers  (2025-08-07T12:46:26Z)
• MIND: Modality-Informed Knowledge Distillation Framework for Multimodal Clinical Prediction Tasks [50.98856172702256]
  We propose the Modality-INformed knowledge Distillation (MIND) framework, a multimodal model compression approach.<n>MIND transfers knowledge from ensembles of pre-trained deep neural networks of varying sizes into a smaller multimodal student.<n>We evaluate MIND on binary and multilabel clinical prediction tasks using time series data and chest X-ray images.
  arXiv  Detail & Related papers  (2025-02-03T08:50:00Z)
• Confidence-aware multi-modality learning for eye disease screening [58.861421804458395]
  We propose a novel multi-modality evidential fusion pipeline for eye disease screening. It provides a measure of confidence for each modality and elegantly integrates the multi-modality information. Experimental results on both public and internal datasets demonstrate that our model excels in robustness.
  arXiv  Detail & Related papers  (2024-05-28T13:27:30Z)
• Missing-modality Enabled Multi-modal Fusion Architecture for Medical Data [8.472576865966744]
  Fusing multi-modal data can improve the performance of deep learning models. Missing modalities are common for medical data due to patients' specificity. This study developed an efficient multi-modal fusion architecture for medical data that was robust to missing modalities.
  arXiv  Detail & Related papers  (2023-09-27T09:46:07Z)
• A Novel Unified Conditional Score-based Generative Framework for Multi-modal Medical Image Completion [54.512440195060584]
  We propose the Unified Multi-Modal Conditional Score-based Generative Model (UMM-CSGM) to take advantage of Score-based Generative Model (SGM) UMM-CSGM employs a novel multi-in multi-out Conditional Score Network (mm-CSN) to learn a comprehensive set of cross-modal conditional distributions. Experiments on BraTS19 dataset show that the UMM-CSGM can more reliably synthesize the heterogeneous enhancement and irregular area in tumor-induced lesions.
  arXiv  Detail & Related papers  (2022-07-07T16:57:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.