Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects

• URL: http://arxiv.org/abs/2311.02332v5
• Date: Sat, 20 Jan 2024 04:36:50 GMT
• Title: Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects
• Authors: Elisa Warner, Joonsang Lee, William Hsu, Tanveer Syeda-Mahmood, Charles Kahn, Olivier Gevaert and Arvind Rao
• Abstract summary: The paper explores the transformative potential of multimodal models for clinical predictions. Despite advancements, challenges such as data biases and the scarcity of "big data" in many biomedical domains persist.
• Score: 2.1070612998322438
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Machine learning (ML) applications in medical artificial intelligence (AI) systems have shifted from traditional and statistical methods to increasing application of deep learning models. This survey navigates the current landscape of multimodal ML, focusing on its profound impact on medical image analysis and clinical decision support systems. Emphasizing challenges and innovations in addressing multimodal representation, fusion, translation, alignment, and co-learning, the paper explores the transformative potential of multimodal models for clinical predictions. It also highlights the need for principled assessments and practical implementation of such models, bringing attention to the dynamics between decision support systems and healthcare providers and personnel. Despite advancements, challenges such as data biases and the scarcity of "big data" in many biomedical domains persist. We conclude with a discussion on principled innovation and collaborative efforts to further the mission of seamless integration of multimodal ML models into biomedical practice.

Related papers

• A Survey of Medical Vision-and-Language Applications and Their Techniques [48.268198631277315]
  Medical vision-and-language models (MVLMs) have attracted substantial interest due to their capability to offer a natural language interface for interpreting complex medical data. Here, we provide a comprehensive overview of MVLMs and the various medical tasks to which they have been applied. We also examine the datasets used for these tasks and compare the performance of different models based on standardized evaluation metrics.
  arXiv  Detail & Related papers  (2024-11-19T03:27:05Z)
• Navigating the landscape of multimodal AI in medicine: a scoping review on technical challenges and clinical applications [2.3754862363513523]
  This review examines the landscape of deep learning-based multimodal AI applications across the medical domain. multimodal AI models consistently outperform their unimodal counterparts, with an average improvement of 6.2 percentage points in AUC. We identify key factors driving multimodal AI development and propose recommendations to accelerate the field's maturation.
  arXiv  Detail & Related papers  (2024-11-06T09:18:05Z)
• From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practice [12.390859712280328]
  Large Language Models (LLMs) have rapidly evolved from text-based systems to multimodal platforms. We examine the current landscape of MLLMs in healthcare, analyzing their applications across clinical decision support, medical imaging, patient engagement, and research.
  arXiv  Detail & Related papers  (2024-09-14T02:35:29Z)
• A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical Applications [0.7831774233149619]
  This systematic review aims to analyze and formalize current intermediate fusion methods in biomedical applications. We introduce a structured notation to enhance the understanding and application of these methods beyond the biomedical domain. Our findings are intended to support researchers, healthcare professionals, and the broader deep learning community in developing more sophisticated and insightful multimodal models.
  arXiv  Detail & Related papers  (2024-08-02T11:48:04Z)
• Automated Ensemble Multimodal Machine Learning for Healthcare [52.500923923797835]
  We introduce a multimodal framework, AutoPrognosis-M, that enables the integration of structured clinical (tabular) data and medical imaging using automated machine learning. AutoPrognosis-M incorporates 17 imaging models, including convolutional neural networks and vision transformers, and three distinct multimodal fusion strategies.
  arXiv  Detail & Related papers  (2024-07-25T17:46:38Z)
• Review of multimodal machine learning approaches in healthcare [0.0]
  Clinicians rely on a variety of data sources to make informed decisions. Recent advances in machine learning have facilitated the more efficient incorporation of multimodal data.
  arXiv  Detail & Related papers  (2024-02-04T12:21:38Z)
• Incomplete Multimodal Learning for Complex Brain Disorders Prediction [65.95783479249745]
  We propose a new incomplete multimodal data integration approach that employs transformers and generative adversarial networks. We apply our new method to predict cognitive degeneration and disease outcomes using the multimodal imaging genetic data from Alzheimer's Disease Neuroimaging Initiative cohort.
  arXiv  Detail & Related papers  (2023-05-25T16:29:16Z)
• DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations [119.1953397679783]
  We focus on advancing the state-of-the-art in interpreting multimodal models. Our proposed approach, DIME, enables accurate and fine-grained analysis of multimodal models.
  arXiv  Detail & Related papers  (2022-03-03T20:52:47Z)
• Domain Shift in Computer Vision models for MRI data analysis: An Overview [64.69150970967524]
  Machine learning and computer vision methods are showing good performance in medical imagery analysis. Yet only a few applications are now in clinical use. Poor transferability of themodels to data from different sources or acquisition domains is one of the reasons for that.
  arXiv  Detail & Related papers  (2020-10-14T16:34:21Z)
• Machine Learning in Nano-Scale Biomedical Engineering [77.75587007080894]
  We review the existing research regarding the use of machine learning in nano-scale biomedical engineering. The main challenges that can be formulated as ML problems are classified into the three main categories. For each of the presented methodologies, special emphasis is given to its principles, applications, and limitations.
  arXiv  Detail & Related papers  (2020-08-05T15:45:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.