Personalized 2D Binary Patient Codes of Tissue Images and Immunogenomic Data Through Multimodal Self-Supervised Fusion

• URL: http://arxiv.org/abs/2409.13115v1
• Date: Thu, 19 Sep 2024 22:49:27 GMT
• Title: Personalized 2D Binary Patient Codes of Tissue Images and Immunogenomic Data Through Multimodal Self-Supervised Fusion
• Authors: Areej Alsaafin, Abubakr Shafique, Saghir Alfasly, H. R. Tizhoosh,
• Abstract summary: MarbliX is an innovative framework that integrates histopathology images with immunogenomic sequencing data, encapsulating them into a concise binary patient code. The experimental results demonstrate the potential of MarbliX to empower healthcare professionals with in-depth insights.
• Score: 0.9374652839580183
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The field of medical diagnostics has witnessed a transformative convergence of artificial intelligence (AI) and healthcare data, offering promising avenues for enhancing patient care and disease comprehension. However, this integration of multimodal data, specifically histopathology whole slide images (WSIs) and genetic sequencing data, presents unique challenges due to modality disparities and the need for scalable computational solutions. This paper addresses the scarcity of multimodal solutions, primarily centered around unimodal data solutions, thus limiting the realization of the rich insights that can be derived from integrating images and genomic data. Here, we introduce MarbliX ``Multimodal Association and Retrieval with Binary Latent Indexed matriX,'' an innovative multimodal framework that integrates histopathology images with immunogenomic sequencing data, encapsulating them into a concise binary patient code, referred to as ``monogram.'' This binary representation facilitates the establishment of a comprehensive archive, enabling clinicians to match similar cases. The experimental results demonstrate the potential of MarbliX to empower healthcare professionals with in-depth insights, leading to more precise diagnoses, reduced variability, and expanded personalized treatment options, particularly in the context of cancer.

Related papers

• FORESEE: Multimodal and Multi-view Representation Learning for Robust Prediction of Cancer Survival [3.4686401890974197]
  We propose a new end-to-end framework, FORESEE, for robustly predicting patient survival by mining multimodal information. Cross-fusion transformer effectively utilizes features at the cellular level, tissue level, and tumor heterogeneity level to correlate prognosis. The hybrid attention encoder (HAE) uses the denoising contextual attention module to obtain the contextual relationship features. We also propose an asymmetrically masked triplet masked autoencoder to reconstruct lost information within modalities.
  arXiv  Detail & Related papers  (2024-05-13T12:39:08Z)
• MMIST-ccRCC: A Real World Medical Dataset for the Development of Multi-Modal Systems [12.914295902429]
  We introduce a real world multi-modal dataset called MMIST-CCRCC. This dataset comprises 2 radiology modalities (CT and MRI), histopathology, genomics, and clinical data from 618 patients with clear cell renal cell carcinoma (ccRCC) We show that even with such severe missing rates the fusion of modalities leads to improvements in the survival forecasting.
  arXiv  Detail & Related papers  (2024-05-02T18:29:05Z)
• HyperFusion: A Hypernetwork Approach to Multimodal Integration of Tabular and Medical Imaging Data for Predictive Modeling [4.44283662576491]
  We present a novel framework based on hypernetworks to fuse clinical imaging and tabular data by conditioning the image processing on the EHR's values and measurements. We show that our framework outperforms both single-modality models and state-of-the-art MRI-tabular data fusion methods.
  arXiv  Detail & Related papers  (2024-03-20T05:50:04Z)
• XAI for In-hospital Mortality Prediction via Multimodal ICU Data [57.73357047856416]
  We propose an efficient, explainable AI solution for predicting in-hospital mortality via multimodal ICU data. We employ multimodal learning in our framework, which can receive heterogeneous inputs from clinical data and make decisions. Our framework can be easily transferred to other clinical tasks, which facilitates the discovery of crucial factors in healthcare research.
  arXiv  Detail & Related papers  (2023-12-29T14:28:04Z)
• HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data [10.774128925670183]
  This paper presents the Hybrid Early-fusion Attention Learning Network (HEALNet), a flexible multimodal fusion architecture. We conduct multimodal survival analysis on Whole Slide Images and Multi-omic data on four cancer datasets from The Cancer Genome Atlas (TCGA) HEALNet achieves state-of-the-art performance compared to other end-to-end trained fusion models.
  arXiv  Detail & Related papers  (2023-11-15T17:06:26Z)
• MaxCorrMGNN: A Multi-Graph Neural Network Framework for Generalized Multimodal Fusion of Medical Data for Outcome Prediction [3.2889220522843625]
  We develop an innovative fusion approach called MaxCorr MGNN that models non-linear modality correlations within and across patients. We then design, for the first time, a generalized multi-layered graph neural network (MGNN) for task-informed reasoning in multi-layered graphs. We evaluate our model an outcome prediction task on a Tuberculosis dataset consistently outperforming several state-of-the-art neural, graph-based and traditional fusion techniques.
  arXiv  Detail & Related papers  (2023-07-13T23:52:41Z)
• Incomplete Multimodal Learning for Complex Brain Disorders Prediction [65.95783479249745]
  We propose a new incomplete multimodal data integration approach that employs transformers and generative adversarial networks. We apply our new method to predict cognitive degeneration and disease outcomes using the multimodal imaging genetic data from Alzheimer's Disease Neuroimaging Initiative cohort.
  arXiv  Detail & Related papers  (2023-05-25T16:29:16Z)
• Cross-Modality Deep Feature Learning for Brain Tumor Segmentation [158.8192041981564]
  This paper proposes a novel cross-modality deep feature learning framework to segment brain tumors from the multi-modality MRI data. The core idea is to mine rich patterns across the multi-modality data to make up for the insufficient data scale. Comprehensive experiments are conducted on the BraTS benchmarks, which show that the proposed cross-modality deep feature learning framework can effectively improve the brain tumor segmentation performance.
  arXiv  Detail & Related papers  (2022-01-07T07:46:01Z)
• Deep Co-Attention Network for Multi-View Subspace Learning [73.3450258002607]
  We propose a deep co-attention network for multi-view subspace learning. It aims to extract both the common information and the complementary information in an adversarial setting. In particular, it uses a novel cross reconstruction loss and leverages the label information to guide the construction of the latent representation.
  arXiv  Detail & Related papers  (2021-02-15T18:46:44Z)
• G-MIND: An End-to-End Multimodal Imaging-Genetics Framework for Biomarker Identification and Disease Classification [49.53651166356737]
  We propose a novel deep neural network architecture to integrate imaging and genetics data, as guided by diagnosis, that provides interpretable biomarkers. We have evaluated our model on a population study of schizophrenia that includes two functional MRI (fMRI) paradigms and Single Nucleotide Polymorphism (SNP) data.
  arXiv  Detail & Related papers  (2021-01-27T19:28:04Z)
• Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion [71.87627318863612]
  We propose a novel multimodal segmentation framework which is robust to the absence of imaging modalities. Our network uses feature disentanglement to decompose the input modalities into the modality-specific appearance code. We validate our method on the important yet challenging multimodal brain tumor segmentation task with the BRATS challenge dataset.
  arXiv  Detail & Related papers  (2020-02-22T14:32:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.