MM-GTUNets: Unified Multi-Modal Graph Deep Learning for Brain Disorders Prediction

• URL: http://arxiv.org/abs/2406.14455v1
• Date: Thu, 20 Jun 2024 16:14:43 GMT
• Title: MM-GTUNets: Unified Multi-Modal Graph Deep Learning for Brain Disorders Prediction
• Authors: Luhui Cai, Weiming Zeng, Hongyu Chen, Hua Zhang, Yueyang Li, Hongjie Yan, Lingbin Bian, Nizhuan Wang,
• Abstract summary: We propose MM-GTUNets, an end-to-end graph transformer based multi-modal graph deep learning framework for brain disorders prediction. We introduce Modality Reward Representation Learning (MRRL) which adaptively constructs population graphs using a reward system. We also propose Adaptive Cross-Modal Graph Learning (ACMGL), which captures critical modality-specific and modality-shared features.
• Score: 8.592259720470697
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Graph deep learning (GDL) has demonstrated impressive performance in predicting population-based brain disorders (BDs) through the integration of both imaging and non-imaging data. However, the effectiveness of GDL based methods heavily depends on the quality of modeling the multi-modal population graphs and tends to degrade as the graph scale increases. Furthermore, these methods often constrain interactions between imaging and non-imaging data to node-edge interactions within the graph, overlooking complex inter-modal correlations, leading to suboptimal outcomes. To overcome these challenges, we propose MM-GTUNets, an end-to-end graph transformer based multi-modal graph deep learning (MMGDL) framework designed for brain disorders prediction at large scale. Specifically, to effectively leverage rich multi-modal information related to diseases, we introduce Modality Reward Representation Learning (MRRL) which adaptively constructs population graphs using a reward system. Additionally, we employ variational autoencoder to reconstruct latent representations of non-imaging features aligned with imaging features. Based on this, we propose Adaptive Cross-Modal Graph Learning (ACMGL), which captures critical modality-specific and modality-shared features through a unified GTUNet encoder taking advantages of Graph UNet and Graph Transformer, and feature fusion module. We validated our method on two public multi-modal datasets ABIDE and ADHD-200, demonstrating its superior performance in diagnosing BDs. Our code is available at https://github.com/NZWANG/MM-GTUNets.

Related papers

• InstructG2I: Synthesizing Images from Multimodal Attributed Graphs [50.852150521561676]
  We propose a graph context-conditioned diffusion model called InstructG2I. InstructG2I first exploits the graph structure and multimodal information to conduct informative neighbor sampling. A Graph-QFormer encoder adaptively encodes the graph nodes into an auxiliary set of graph prompts to guide the denoising process.
  arXiv  Detail & Related papers  (2024-10-09T17:56:15Z)
• GIF: A General Graph Unlearning Strategy via Influence Function [63.52038638220563]
  Graph Influence Function (GIF) is a model-agnostic unlearning method that can efficiently and accurately estimate parameter changes in response to a $epsilon$-mass perturbation in deleted data. We conduct extensive experiments on four representative GNN models and three benchmark datasets to justify GIF's superiority in terms of unlearning efficacy, model utility, and unlearning efficiency.
  arXiv  Detail & Related papers  (2023-04-06T03:02:54Z)
• Multi-modal Multi-kernel Graph Learning for Autism Prediction and Biomarker Discovery [29.790200009136825]
  We propose a novel method to offset the negative impact between modalities in the process of multi-modal integration and extract heterogeneous information from graphs. Our method is evaluated on the benchmark Autism Brain Imaging Data Exchange (ABIDE) dataset and outperforms the state-of-the-art methods. In addition, discriminative brain regions associated with autism are identified by our model, providing guidance for the study of autism pathology.
  arXiv  Detail & Related papers  (2023-03-03T07:09:17Z)
• MMGA: Multimodal Learning with Graph Alignment [8.349066399479938]
  We propose MMGA, a novel multimodal pre-training framework to incorporate information from graph (social network), image and text modalities on social media. In MMGA, a multi-step graph alignment mechanism is proposed to add the self-supervision from graph modality to optimize the image and text encoders. We release our dataset, the first social media multimodal dataset with graph, of 60,000 users labeled with specific topics based on 2 million posts to facilitate future research.
  arXiv  Detail & Related papers  (2022-10-18T15:50:31Z)
• Multi-modal Graph Learning for Disease Prediction [35.156975779372836]
  We propose an end-to-end Multi-modal Graph Learning framework (MMGL) for disease prediction with multi-modality. Instead of defining the graph manually, the latent graph structure is captured through an effective way of adaptive graph learning. An extensive group of experiments on two disease prediction tasks demonstrates that the proposed MMGL achieves more favorable performance.
  arXiv  Detail & Related papers  (2022-03-11T12:33:20Z)
• Towards Graph Self-Supervised Learning with Contrastive Adjusted Zooming [48.99614465020678]
  We introduce a novel self-supervised graph representation learning algorithm via Graph Contrastive Adjusted Zooming. This mechanism enables G-Zoom to explore and extract self-supervision signals from a graph from multiple scales. We have conducted extensive experiments on real-world datasets, and the results demonstrate that our proposed model outperforms state-of-the-art methods consistently.
  arXiv  Detail & Related papers  (2021-11-20T22:45:53Z)
• Multi-modal Graph Learning for Disease Prediction [35.4310911850558]
  We propose an end-to-end Multimodal Graph Learning framework (MMGL) for disease prediction. Instead of defining the adjacency matrix manually as existing methods, the latent graph structure can be captured through a novel way of adaptive graph learning.
  arXiv  Detail & Related papers  (2021-07-01T03:59:22Z)
• Diversified Multiscale Graph Learning with Graph Self-Correction [55.43696999424127]
  We propose a diversified multiscale graph learning model equipped with two core ingredients. A graph self-correction (GSC) mechanism to generate informative embedded graphs, and a diversity boosting regularizer (DBR) to achieve a comprehensive characterization of the input graph. Experiments on popular graph classification benchmarks show that the proposed GSC mechanism leads to significant improvements over state-of-the-art graph pooling methods.
  arXiv  Detail & Related papers  (2021-03-17T16:22:24Z)
• Multilayer Clustered Graph Learning [66.94201299553336]
  We use contrastive loss as a data fidelity term, in order to properly aggregate the observed layers into a representative graph. Experiments show that our method leads to a clustered clusters w.r.t. We learn a clustering algorithm for solving clustering problems.
  arXiv  Detail & Related papers  (2020-10-29T09:58:02Z)
• Graph Representation Learning via Graphical Mutual Information Maximization [86.32278001019854]
  We propose a novel concept, Graphical Mutual Information (GMI), to measure the correlation between input graphs and high-level hidden representations. We develop an unsupervised learning model trained by maximizing GMI between the input and output of a graph neural encoder.
  arXiv  Detail & Related papers  (2020-02-04T08:33:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.