Continually Evolved Multimodal Foundation Models for Cancer Prognosis

• URL: http://arxiv.org/abs/2501.18170v2
• Date: Sat, 01 Feb 2025 03:19:00 GMT
• Title: Continually Evolved Multimodal Foundation Models for Cancer Prognosis
• Authors: Jie Peng, Shuang Zhou, Longwei Yang, Yiran Song, Mohan Zhang, Kaixiong Zhou, Feng Xie, Mingquan Lin, Rui Zhang, Tianlong Chen,
• Abstract summary: Cancer prognosis is a critical task that involves predicting patient outcomes and survival rates. Previous studies have integrated diverse data modalities, such as clinical notes, medical images, and genomic data, leveraging their complementary information. Existing approaches face two major limitations. First, they struggle to incorporate newly arrived data with varying distributions into training, such as patient records from different hospitals. Second, most multimodal integration methods rely on simplistic concatenation or task-specific pipelines, which fail to capture the complex interdependencies across modalities.
• Score: 50.43145292874533
• License:
• Abstract: Cancer prognosis is a critical task that involves predicting patient outcomes and survival rates. To enhance prediction accuracy, previous studies have integrated diverse data modalities, such as clinical notes, medical images, and genomic data, leveraging their complementary information. However, existing approaches face two major limitations. First, they struggle to incorporate newly arrived data with varying distributions into training, such as patient records from different hospitals, thus rendering sub-optimal generalizability and limited utility in real-world applications. Second, most multimodal integration methods rely on simplistic concatenation or task-specific pipelines, which fail to capture the complex interdependencies across modalities. To address these, we propose a continually evolving multi-modal foundation model. Extensive experiments on the TCGA dataset demonstrate the effectiveness of our approach, highlighting its potential to advance cancer prognosis by enabling robust and adaptive multimodal integration.

Related papers

• Survival Prediction in Lung Cancer through Multi-Modal Representation Learning [9.403446155541346]
  This paper presents a novel approach to survival prediction by harnessing comprehensive information from CT and PET scans, along with associated Genomic data. We aim to develop a robust predictive model for survival outcomes by integrating multi-modal imaging data with genetic information.
  arXiv  Detail & Related papers  (2024-09-30T10:42:20Z)
• M2EF-NNs: Multimodal Multi-instance Evidence Fusion Neural Networks for Cancer Survival Prediction [24.323961146023358]
  We propose a neural network model called M2EF-NNs for accurate cancer survival prediction. To capture global information in the images, we use a pre-trained Vision Transformer (ViT) model. We are the first to apply the Dempster-Shafer evidence theory (DST) to cancer survival prediction.
  arXiv  Detail & Related papers  (2024-08-08T02:31:04Z)
• Multi-modal Data Binding for Survival Analysis Modeling with Incomplete Data and Annotations [19.560652381770243]
  We introduce a novel framework that simultaneously handles incomplete data across modalities and censored survival labels. Our approach employs advanced foundation models to encode individual modalities and align them into a universal representation space. The proposed method demonstrates outstanding prediction accuracy in two survival analysis tasks on both employed datasets.
  arXiv  Detail & Related papers  (2024-07-25T02:55:39Z)
• Confidence-aware multi-modality learning for eye disease screening [58.861421804458395]
  We propose a novel multi-modality evidential fusion pipeline for eye disease screening. It provides a measure of confidence for each modality and elegantly integrates the multi-modality information. Experimental results on both public and internal datasets demonstrate that our model excels in robustness.
  arXiv  Detail & Related papers  (2024-05-28T13:27:30Z)
• XAI for In-hospital Mortality Prediction via Multimodal ICU Data [57.73357047856416]
  We propose an efficient, explainable AI solution for predicting in-hospital mortality via multimodal ICU data. We employ multimodal learning in our framework, which can receive heterogeneous inputs from clinical data and make decisions. Our framework can be easily transferred to other clinical tasks, which facilitates the discovery of crucial factors in healthcare research.
  arXiv  Detail & Related papers  (2023-12-29T14:28:04Z)
• Cross-modality Attention-based Multimodal Fusion for Non-small Cell Lung Cancer (NSCLC) Patient Survival Prediction [0.6476298550949928]
  We propose a cross-modality attention-based multimodal fusion pipeline designed to integrate modality-specific knowledge for patient survival prediction in non-small cell lung cancer (NSCLC) Compared with single modality, which achieved c-index of 0.5772 and 0.5885 using solely tissue image data or RNA-seq data, respectively, the proposed fusion approach achieved c-index 0.6587 in our experiment.
  arXiv  Detail & Related papers  (2023-08-18T21:42:52Z)
• MaxCorrMGNN: A Multi-Graph Neural Network Framework for Generalized Multimodal Fusion of Medical Data for Outcome Prediction [3.2889220522843625]
  We develop an innovative fusion approach called MaxCorr MGNN that models non-linear modality correlations within and across patients. We then design, for the first time, a generalized multi-layered graph neural network (MGNN) for task-informed reasoning in multi-layered graphs. We evaluate our model an outcome prediction task on a Tuberculosis dataset consistently outperforming several state-of-the-art neural, graph-based and traditional fusion techniques.
  arXiv  Detail & Related papers  (2023-07-13T23:52:41Z)
• Incomplete Multimodal Learning for Complex Brain Disorders Prediction [65.95783479249745]
  We propose a new incomplete multimodal data integration approach that employs transformers and generative adversarial networks. We apply our new method to predict cognitive degeneration and disease outcomes using the multimodal imaging genetic data from Alzheimer's Disease Neuroimaging Initiative cohort.
  arXiv  Detail & Related papers  (2023-05-25T16:29:16Z)
• MIMO: Mutual Integration of Patient Journey and Medical Ontology for Healthcare Representation Learning [49.57261599776167]
  We propose an end-to-end robust Transformer-based solution, Mutual Integration of patient journey and Medical Ontology (MIMO) for healthcare representation learning and predictive analytics.
  arXiv  Detail & Related papers  (2021-07-20T07:04:52Z)
• M2Net: Multi-modal Multi-channel Network for Overall Survival Time Prediction of Brain Tumor Patients [151.4352001822956]
  Early and accurate prediction of overall survival (OS) time can help to obtain better treatment planning for brain tumor patients. Existing prediction methods rely on radiomic features at the local lesion area of a magnetic resonance (MR) volume. We propose an end-to-end OS time prediction model; namely, Multi-modal Multi-channel Network (M2Net)
  arXiv  Detail & Related papers  (2020-06-01T05:21:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.