M$^3$Care: Learning with Missing Modalities in Multimodal Healthcare Data

• URL: http://arxiv.org/abs/2210.17292v1
• Date: Fri, 28 Oct 2022 02:38:40 GMT
• Title: M$^3$Care: Learning with Missing Modalities in Multimodal Healthcare Data
• Abstract summary: Missing modality caused by various clinical and social reasons is a common issue in real-world clinical scenarios. We propose a model solving a direct problem, dubbed learning with Missing Modalities in Multimodal healthcare data (M3Care) M3Care is an end-to-end model compensating the missing information of the patients with missing modalities to perform clinical analysis.
• Score: 17.479050575272147
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal electronic health record (EHR) data are widely used in clinical applications. Conventional methods usually assume that each sample (patient) is associated with the unified observed modalities, and all modalities are available for each sample. However, missing modality caused by various clinical and social reasons is a common issue in real-world clinical scenarios. Existing methods mostly rely on solving a generative model that learns a mapping from the latent space to the original input space, which is an unstable ill-posed inverse problem. To relieve the underdetermined system, we propose a model solving a direct problem, dubbed learning with Missing Modalities in Multimodal healthcare data (M3Care). M3Care is an end-to-end model compensating the missing information of the patients with missing modalities to perform clinical analysis. Instead of generating raw missing data, M3Care imputes the task-related information of the missing modalities in the latent space by the auxiliary information from each patient's similar neighbors, measured by a task-guided modality-adaptive similarity metric, and thence conducts the clinical tasks. The task-guided modality-adaptive similarity metric utilizes the uncensored modalities of the patient and the other patients who also have the same uncensored modalities to find similar patients. Experiments on real-world datasets show that M3Care outperforms the state-of-the-art baselines. Moreover, the findings discovered by M3Care are consistent with experts and medical knowledge, demonstrating the capability and the potential of providing useful insights and explanations.

Related papers

• Real-World Multi-Modal and Longitudinal Lung Cancer Dataset [9.050510996987436]
  We introduce a newly curated dataset designed to support the evaluation of a wide range of learning pipelines under realistic conditions.<n>The dataset comprises a total of 1,365 lung cancer patients and has three imaging modalities (whole-slide images, CT scans, and PET scans), structured clinical data, transcriptomic, and longitudinal follow-up and treatment information.<n>Our results show that, despite high levels of missingness, integrating complementary modalities consistently improves predictive performance over uni-modal approaches.
  arXiv  Detail & Related papers  (2026-09-04T14:35:10Z)
• MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models [70.34265674686516]
  Multimodal embedding models aim to map heterogeneous inputs, such as text, images, videos, and audio, into a shared semantic space.<n>We introduce MMEB-V3, a benchmark that evaluates embeddings across text, image, video, audio, as well as agent-centric scenarios.<n>We conduct a systematic analysis of full-modality embeddings and identify three key findings.
  arXiv  Detail & Related papers  (2026-04-25T14:15:05Z)
• MedForget: Hierarchy-Aware Multimodal Unlearning Testbed for Medical AI [66.0701326117134]
  MedForget is a hierarchy-aware multimodal unlearning testbed for building compliant medical AI systems.<n>We show that existing methods struggle to achieve complete, hierarchy-aware forgetting without reducing diagnostic performance.<n>We introduce a reconstruction attack that progressively adds hierarchical level context to prompts.
  arXiv  Detail & Related papers  (2025-12-10T17:55:06Z)
• Causal Representation Learning from Multimodal Clinical Records under Non-Random Modality Missingness [5.351519104745287]
  Clinical notes contain rich patient information, such as diagnoses or medications, making them valuable for patient representation learning.<n>Recent advances in large language models have further improved the ability to extract meaningful representations from clinical texts.<n>We propose a causal representation learning framework that leverages observed data and informative missingness in multimodal clinical records.
  arXiv  Detail & Related papers  (2025-09-21T20:34:52Z)
• Causal Debiasing Medical Multimodal Representation Learning with Missing Modalities [6.02318066285653]
  Real-world medical datasets often suffer from missing modalities due to cost, protocol, or patient-specific constraints.<n>Our method consists of two key components: (1) a missingness deconfounding module that approximates causal intervention based on backdoor adjustment and (2) a dual-branch neural network that explicitly disentangles causal features from spurious correlations.
  arXiv  Detail & Related papers  (2025-09-06T06:27:10Z)
• medDreamer: Model-Based Reinforcement Learning with Latent Imagination on Complex EHRs for Clinical Decision Support [3.8382507197481144]
  medDreamer is a novel model-based reinforcement learning framework for personalized treatment recommendation.<n>It simulates latent patient states from irregular data and a two-phase policy trained on a hybrid of real and imagined trajectories.<n>It significantly outperforms model-free and model-based baselines in both clinical outcomes and off-policy metrics.
  arXiv  Detail & Related papers  (2025-05-26T10:16:39Z)
• Enhancing Depression Detection via Question-wise Modality Fusion [47.45016610508853]
  Depression is a highly prevalent and disabling condition that incurs substantial personal and societal costs.<n>We propose a novel Question-wise Modality Fusion framework trained with a novel Imbalanced Ordinal Log-Loss function.
  arXiv  Detail & Related papers  (2025-03-26T12:34:34Z)
• A Learnable Multi-views Contrastive Framework with Reconstruction Discrepancy for Medical Time-Series [8.741139851597364]
  We propose incorporating external data from related tasks and leveraging AE-GAN to extract prior knowledge. We introduce LMCF, a framework that integrates a multi-head attention mechanism and adaptively learns representations from different views. Experiments on three target datasets demonstrate that our method consistently outperforms seven other baselines.
  arXiv  Detail & Related papers  (2025-01-30T14:20:11Z)
• Continually Evolved Multimodal Foundation Models for Cancer Prognosis [50.43145292874533]
  Cancer prognosis is a critical task that involves predicting patient outcomes and survival rates. Previous studies have integrated diverse data modalities, such as clinical notes, medical images, and genomic data, leveraging their complementary information. Existing approaches face two major limitations. First, they struggle to incorporate newly arrived data with varying distributions into training, such as patient records from different hospitals. Second, most multimodal integration methods rely on simplistic concatenation or task-specific pipelines, which fail to capture the complex interdependencies across modalities.
  arXiv  Detail & Related papers  (2025-01-30T06:49:57Z)
• DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency [18.291267748113142]
  We propose DrFuse to achieve effective clinical multi-modal fusion. We address the missing modality issue by disentangling the features shared across modalities and those unique within each modality. We validate the proposed method using real-world large-scale datasets, MIMIC-IV and MIMIC-CXR.
  arXiv  Detail & Related papers  (2024-03-10T12:41:34Z)
• Self-supervised 3D Patient Modeling with Multi-modal Attentive Fusion [32.71972792352939]
  3D patient body modeling is critical to the success of automated patient positioning for smart medical scanning and operating rooms. Existing CNN-based end-to-end patient modeling solutions typically require customized network designs demanding large amount of relevant training data. We propose a generic modularized 3D patient modeling method consists of (a) a multi-modal keypoint detection module with attentive fusion for 2D patient joint localization. We demonstrate the efficacy of the proposed method by extensive patient positioning experiments on both public and clinical data.
  arXiv  Detail & Related papers  (2024-03-05T18:58:55Z)
• The Relevance Feature and Vector Machine for health applications [0.11538034264098687]
  This paper presents a novel model that addresses the challenges of the fat-data problem when dealing with clinical prospective studies. The model capabilities are tested against state-of-the-art models in several medical datasets with fat-data problems.
  arXiv  Detail & Related papers  (2024-02-11T01:21:56Z)
• DCID: Deep Canonical Information Decomposition [84.59396326810085]
  We consider the problem of identifying the signal shared between two one-dimensional target variables. We propose ICM, an evaluation metric which can be used in the presence of ground-truth labels. We also propose Deep Canonical Information Decomposition (DCID) - a simple, yet effective approach for learning the shared variables.
  arXiv  Detail & Related papers  (2023-06-27T16:59:06Z)
• Cross-Modal Vertical Federated Learning for MRI Reconstruction [42.527873703840996]
  Federated learning enables multiple hospitals to cooperatively learn a shared model without privacy disclosure. We develop a novel framework, namely Federated Consistent Regularization constrained Feature Disentanglement (Fed-CRFD), for boosting MRI reconstruction. Our method can fully exploit the multi-source data from hospitals while alleviating the domain shift problem.
  arXiv  Detail & Related papers  (2023-06-05T08:07:01Z)
• Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study [60.56194508762205]
  We reproduce, compare, and analyze state-of-the-art automated medical coding machine learning models. We show that several models underperform due to weak configurations, poorly sampled train-test splits, and insufficient evaluation. We present the first comprehensive results on the newly released MIMIC-IV dataset using the reproduced models.
  arXiv  Detail & Related papers  (2023-04-21T11:54:44Z)
• SPeC: A Soft Prompt-Based Calibration on Performance Variability of Large Language Model in Clinical Notes Summarization [50.01382938451978]
  We introduce a model-agnostic pipeline that employs soft prompts to diminish variance while preserving the advantages of prompt-based summarization. Experimental findings indicate that our method not only bolsters performance but also effectively curbs variance for various language models.
  arXiv  Detail & Related papers  (2023-03-23T04:47:46Z)
• Modelling Patient Trajectories Using Multimodal Information [0.0]
  We propose a solution to model patient trajectories that combines different types of information and considers the temporal aspect of clinical data. The developed solution was evaluated on two different clinical outcomes, unexpected patient readmission and disease progression.
  arXiv  Detail & Related papers  (2022-09-09T10:20:54Z)
• Learning Multitask Gaussian Bayesian Networks [11.745963019193955]
  Major depressive disorder (MDD) requires study of brain functional connectivity alterations for patients. The amount of data collected during an fMRI scan is too limited to provide sufficient information for individual analysis. We propose a multitask Gaussian Bayesian network framework capable for identifying individual disease-induced alterations for MDD patients.
  arXiv  Detail & Related papers  (2022-05-11T08:38:00Z)
• MIMO: Mutual Integration of Patient Journey and Medical Ontology for Healthcare Representation Learning [49.57261599776167]
  We propose an end-to-end robust Transformer-based solution, Mutual Integration of patient journey and Medical Ontology (MIMO) for healthcare representation learning and predictive analytics.
  arXiv  Detail & Related papers  (2021-07-20T07:04:52Z)
• Adversarial Sample Enhanced Domain Adaptation: A Case Study on Predictive Modeling with Electronic Health Records [57.75125067744978]
  We propose a data augmentation method to facilitate domain adaptation. adversarially generated samples are used during domain adaptation. Results confirm the effectiveness of our method and the generality on different tasks.
  arXiv  Detail & Related papers  (2021-01-13T03:20:20Z)
• Hemogram Data as a Tool for Decision-making in COVID-19 Management: Applications to Resource Scarcity Scenarios [62.997667081978825]
  COVID-19 pandemics has challenged emergency response systems worldwide, with widespread reports of essential services breakdown and collapse of health care structure. This work describes a machine learning model derived from hemogram exam data performed in symptomatic patients. Proposed models can predict COVID-19 qRT-PCR results in symptomatic individuals with high accuracy, sensitivity and specificity.
  arXiv  Detail & Related papers  (2020-05-10T01:45:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.