AdaMHF: Adaptive Multimodal Hierarchical Fusion for Survival Prediction

• URL: http://arxiv.org/abs/2503.21124v1
• Date: Thu, 27 Mar 2025 03:27:55 GMT
• Title: AdaMHF: Adaptive Multimodal Hierarchical Fusion for Survival Prediction
• Authors: Shuaiyu Zhang, Xun Lin, Rongxiang Zhang, Yu Bai, Yong Xu, Tao Tan, Xunbin Zheng, Zitong Yu,
• Abstract summary: AdaMHF is a framework designed for efficient, comprehensive, and tailored feature extraction and fusion.<n>It is specifically adapted to the uniqueness of medical data, enabling accurate predictions with minimal resource consumption.<n>Extensive experiments on TCGA datasets demonstrate that AdaMHF surpasses current state-of-the-art methods.
• Score: 29.11927806428086
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The integration of pathologic images and genomic data for survival analysis has gained increasing attention with advances in multimodal learning. However, current methods often ignore biological characteristics, such as heterogeneity and sparsity, both within and across modalities, ultimately limiting their adaptability to clinical practice. To address these challenges, we propose AdaMHF: Adaptive Multimodal Hierarchical Fusion, a framework designed for efficient, comprehensive, and tailored feature extraction and fusion. AdaMHF is specifically adapted to the uniqueness of medical data, enabling accurate predictions with minimal resource consumption, even under challenging scenarios with missing modalities. Initially, AdaMHF employs an experts expansion and residual structure to activate specialized experts for extracting heterogeneous and sparse features. Extracted tokens undergo refinement via selection and aggregation, reducing the weight of non-dominant features while preserving comprehensive information. Subsequently, the encoded features are hierarchically fused, allowing multi-grained interactions across modalities to be captured. Furthermore, we introduce a survival prediction benchmark designed to resolve scenarios with missing modalities, mirroring real-world clinical conditions. Extensive experiments on TCGA datasets demonstrate that AdaMHF surpasses current state-of-the-art (SOTA) methods, showcasing exceptional performance in both complete and incomplete modality settings.

Related papers

• MIRROR: Multi-Modal Pathological Self-Supervised Representation Learning via Modality Alignment and Retention [52.106879463828044]
  Histopathology and transcriptomics are fundamental modalities in oncology, encapsulating the morphological and molecular aspects of the disease. We present MIRROR, a novel multi-modal representation learning method designed to foster both modality alignment and retention. Extensive evaluations on TCGA cohorts for cancer subtyping and survival analysis highlight MIRROR's superior performance.
  arXiv  Detail & Related papers  (2025-03-01T07:02:30Z)
• Continually Evolved Multimodal Foundation Models for Cancer Prognosis [50.43145292874533]
  Cancer prognosis is a critical task that involves predicting patient outcomes and survival rates.<n>Previous studies have integrated diverse data modalities, such as clinical notes, medical images, and genomic data, leveraging their complementary information.<n>Existing approaches face two major limitations. First, they struggle to incorporate newly arrived data with varying distributions into training, such as patient records from different hospitals.<n>Second, most multimodal integration methods rely on simplistic concatenation or task-specific pipelines, which fail to capture the complex interdependencies across modalities.
  arXiv  Detail & Related papers  (2025-01-30T06:49:57Z)
• AMM-Diff: Adaptive Multi-Modality Diffusion Network for Missing Modality Imputation [2.8498944632323755]
  In clinical practice, full imaging is not always feasible, often due to complex acquisition protocols, stringent privacy regulations, or specific clinical needs.<n>A promising solution is missing data imputation, where absent modalities are generated from available ones.<n>We propose an Adaptive Multi-Modality Diffusion Network (AMM-Diff), a novel diffusion-based generative model capable of handling any number of input modalities and generating the missing ones.
  arXiv  Detail & Related papers  (2025-01-22T12:29:33Z)
• MoME: Mixture of Multimodal Experts for Cancer Survival Prediction [46.520971457396726]
  Survival analysis, as a challenging task, requires integrating Whole Slide Images (WSIs) and genomic data for comprehensive decision-making. Previous approaches utilize co-attention methods, which fuse features from both modalities only once after separate encoding. We propose a Biased Progressive Clever (BPE) paradigm, performing encoding and fusion simultaneously.
  arXiv  Detail & Related papers  (2024-06-14T03:44:33Z)
• Unveiling Incomplete Modality Brain Tumor Segmentation: Leveraging Masked Predicted Auto-Encoder and Divergence Learning [6.44069573245889]
  Brain tumor segmentation remains a significant challenge, particularly in the context of multi-modal magnetic resonance imaging (MRI) We propose a novel strategy, which is called masked predicted pre-training, enabling robust feature learning from incomplete modality data. In the fine-tuning phase, we utilize a knowledge distillation technique to align features between complete and missing modality data, simultaneously enhancing model robustness.
  arXiv  Detail & Related papers  (2024-06-12T20:35:16Z)
• FORESEE: Multimodal and Multi-view Representation Learning for Robust Prediction of Cancer Survival [3.4686401890974197]
  We propose a new end-to-end framework, FORESEE, for robustly predicting patient survival by mining multimodal information. Cross-fusion transformer effectively utilizes features at the cellular level, tissue level, and tumor heterogeneity level to correlate prognosis. The hybrid attention encoder (HAE) uses the denoising contextual attention module to obtain the contextual relationship features. We also propose an asymmetrically masked triplet masked autoencoder to reconstruct lost information within modalities.
  arXiv  Detail & Related papers  (2024-05-13T12:39:08Z)
• Seeing Unseen: Discover Novel Biomedical Concepts via Geometry-Constrained Probabilistic Modeling [53.7117640028211]
  We present a geometry-constrained probabilistic modeling treatment to resolve the identified issues. We incorporate a suite of critical geometric properties to impose proper constraints on the layout of constructed embedding space. A spectral graph-theoretic method is devised to estimate the number of potential novel classes.
  arXiv  Detail & Related papers  (2024-03-02T00:56:05Z)
• HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data [10.774128925670183]
  This paper presents the Hybrid Early-fusion Attention Learning Network (HEALNet), a flexible multimodal fusion architecture. We conduct multimodal survival analysis on Whole Slide Images and Multi-omic data on four cancer datasets from The Cancer Genome Atlas (TCGA) HEALNet achieves state-of-the-art performance compared to other end-to-end trained fusion models.
  arXiv  Detail & Related papers  (2023-11-15T17:06:26Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)
• Differentiable Agent-based Epidemiology [71.81552021144589]
  We introduce GradABM: a scalable, differentiable design for agent-based modeling that is amenable to gradient-based learning with automatic differentiation. GradABM can quickly simulate million-size populations in few seconds on commodity hardware, integrate with deep neural networks and ingest heterogeneous data sources.
  arXiv  Detail & Related papers  (2022-07-20T07:32:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.