Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Prediction

• URL: http://arxiv.org/abs/2401.01646v2
• Date: Mon, 26 Feb 2024 08:21:24 GMT
• Title: Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Prediction
• Authors: Yilan Zhang, Yingxue Xu, Jianqi Chen, Fengying Xie, Hao Chen
• Abstract summary: Multimodal learning significantly benefits cancer survival prediction. Massive redundancy in multimodal data prevents it from extracting discriminative and compact information. We propose a new framework, Prototypical Information Bottlenecking and Disentangling.
• Score: 7.772854118416362
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal learning significantly benefits cancer survival prediction, especially the integration of pathological images and genomic data. Despite advantages of multimodal learning for cancer survival prediction, massive redundancy in multimodal data prevents it from extracting discriminative and compact information: (1) An extensive amount of intra-modal task-unrelated information blurs discriminability, especially for gigapixel whole slide images (WSIs) with many patches in pathology and thousands of pathways in genomic data, leading to an ``intra-modal redundancy" issue. (2) Duplicated information among modalities dominates the representation of multimodal data, which makes modality-specific information prone to being ignored, resulting in an ``inter-modal redundancy" issue. To address these, we propose a new framework, Prototypical Information Bottlenecking and Disentangling (PIBD), consisting of Prototypical Information Bottleneck (PIB) module for intra-modal redundancy and Prototypical Information Disentanglement (PID) module for inter-modal redundancy. Specifically, a variant of information bottleneck, PIB, is proposed to model prototypes approximating a bunch of instances for different risk levels, which can be used for selection of discriminative instances within modality. PID module decouples entangled multimodal data into compact distinct components: modality-common and modality-specific knowledge, under the guidance of the joint prototypical distribution. Extensive experiments on five cancer benchmark datasets demonstrated our superiority over other methods.

Related papers

• Incomplete Modality Disentangled Representation for Ophthalmic Disease Grading and Diagnosis [16.95583564875497]
  We propose an Incomplete Modality Disentangled Representation (IMDR) strategy to disentangle features into explicit independent modal-common and modal-specific features. Experiments on four multimodal datasets demonstrate that the proposed IMDR outperforms the state-of-the-art methods significantly.
  arXiv  Detail & Related papers  (2025-02-17T12:10:35Z)
• MIND: Modality-Informed Knowledge Distillation Framework for Multimodal Clinical Prediction Tasks [50.98856172702256]
  We propose the Modality-INformed knowledge Distillation (MIND) framework, a multimodal model compression approach. MIND transfers knowledge from ensembles of pre-trained deep neural networks of varying sizes into a smaller multimodal student. We evaluate MIND on binary and multilabel clinical prediction tasks using time series data and chest X-ray images.
  arXiv  Detail & Related papers  (2025-02-03T08:50:00Z)
• Continually Evolved Multimodal Foundation Models for Cancer Prognosis [50.43145292874533]
  Cancer prognosis is a critical task that involves predicting patient outcomes and survival rates. Previous studies have integrated diverse data modalities, such as clinical notes, medical images, and genomic data, leveraging their complementary information. Existing approaches face two major limitations. First, they struggle to incorporate newly arrived data with varying distributions into training, such as patient records from different hospitals. Second, most multimodal integration methods rely on simplistic concatenation or task-specific pipelines, which fail to capture the complex interdependencies across modalities.
  arXiv  Detail & Related papers  (2025-01-30T06:49:57Z)
• FORESEE: Multimodal and Multi-view Representation Learning for Robust Prediction of Cancer Survival [3.4686401890974197]
  We propose a new end-to-end framework, FORESEE, for robustly predicting patient survival by mining multimodal information. Cross-fusion transformer effectively utilizes features at the cellular level, tissue level, and tumor heterogeneity level to correlate prognosis. The hybrid attention encoder (HAE) uses the denoising contextual attention module to obtain the contextual relationship features. We also propose an asymmetrically masked triplet masked autoencoder to reconstruct lost information within modalities.
  arXiv  Detail & Related papers  (2024-05-13T12:39:08Z)
• HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data [10.774128925670183]
  This paper presents the Hybrid Early-fusion Attention Learning Network (HEALNet), a flexible multimodal fusion architecture. We conduct multimodal survival analysis on Whole Slide Images and Multi-omic data on four cancer datasets from The Cancer Genome Atlas (TCGA) HEALNet achieves state-of-the-art performance compared to other end-to-end trained fusion models.
  arXiv  Detail & Related papers  (2023-11-15T17:06:26Z)
• Dynamic Multimodal Information Bottleneck for Multimodality Classification [26.65073424377933]
  We propose a dynamic multimodal information bottleneck framework for attaining a robust fused feature representation. Specifically, our information bottleneck module serves to filter out the task-irrelevant information and noises in the fused feature. Our method surpasses the state-of-the-art and is significantly more robust, being the only method to remain performance when large-scale noisy channels exist.
  arXiv  Detail & Related papers  (2023-11-02T08:34:08Z)
• Align and Attend: Multimodal Summarization with Dual Contrastive Losses [57.83012574678091]
  The goal of multimodal summarization is to extract the most important information from different modalities to form output summaries. Existing methods fail to leverage the temporal correspondence between different modalities and ignore the intrinsic correlation between different samples. We introduce Align and Attend Multimodal Summarization (A2Summ), a unified multimodal transformer-based model which can effectively align and attend the multimodal input.
  arXiv  Detail & Related papers  (2023-03-13T17:01:42Z)
• Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework [89.8609061423685]
  We propose an information-theoretic approach to quantify the degree of redundancy, uniqueness, and synergy relating input modalities with an output task. To validate PID estimation, we conduct extensive experiments on both synthetic datasets where the PID is known and on large-scale multimodal benchmarks. We demonstrate their usefulness in (1) quantifying interactions within multimodal datasets, (2) quantifying interactions captured by multimodal models, (3) principled approaches for model selection, and (4) three real-world case studies.
  arXiv  Detail & Related papers  (2023-02-23T18:59:05Z)
• Lung Cancer Risk Estimation with Incomplete Data: A Joint Missing Imputation Perspective [5.64530854079352]
  We address imputation of missing data by modeling the joint distribution of multi-modal data. Motivated by partial bidirectional generative adversarial net (PBiGAN), we propose a new Conditional PBiGAN (C-PBiGAN) method. C-PBiGAN achieves significant improvements in lung cancer risk estimation compared with representative imputation methods.
  arXiv  Detail & Related papers  (2021-07-25T20:15:16Z)
• Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Models [86.9292779620645]
  We develop a contrastive framework for generative model learning, allowing us to train the model not just by the commonality between modalities, but by the distinction between "related" and "unrelated" multimodal data. Under our proposed framework, the generative model can accurately identify related samples from unrelated ones, making it possible to make use of the plentiful unlabeled, unpaired multimodal data.
  arXiv  Detail & Related papers  (2020-07-02T15:08:11Z)
• M2Net: Multi-modal Multi-channel Network for Overall Survival Time Prediction of Brain Tumor Patients [151.4352001822956]
  Early and accurate prediction of overall survival (OS) time can help to obtain better treatment planning for brain tumor patients. Existing prediction methods rely on radiomic features at the local lesion area of a magnetic resonance (MR) volume. We propose an end-to-end OS time prediction model; namely, Multi-modal Multi-channel Network (M2Net)
  arXiv  Detail & Related papers  (2020-06-01T05:21:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.