Revolutionizing Disease Diagnosis with simultaneous functional PET/MR and Deeply Integrated Brain Metabolic, Hemodynamic, and Perfusion Networks

• URL: http://arxiv.org/abs/2403.20058v2
• Date: Sat, 14 Sep 2024 04:18:52 GMT
• Title: Revolutionizing Disease Diagnosis with simultaneous functional PET/MR and Deeply Integrated Brain Metabolic, Hemodynamic, and Perfusion Networks
• Authors: Luoyu Wang, Yitian Tao, Qing Yang, Yan Liang, Siwei Liu, Hongcheng Shi, Dinggang Shen, Han Zhang,
• Abstract summary: We propose MX-ARM, a multimodal MiXture-of-experts Alignment Reconstruction and Model. It is modality detachable and exchangeable, allocating different multi-layer perceptrons dynamically ("mixture of experts") through learnable weights to learn respective representations from different modalities.
• Score: 40.986069119392944
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Simultaneous functional PET/MR (sf-PET/MR) presents a cutting-edge multimodal neuroimaging technique. It provides an unprecedented opportunity for concurrently monitoring and integrating multifaceted brain networks built by spatiotemporally covaried metabolic activity, neural activity, and cerebral blood flow (perfusion). Albeit high scientific/clinical values, short in hardware accessibility of PET/MR hinders its applications, let alone modern AI-based PET/MR fusion models. Our objective is to develop a clinically feasible AI-based disease diagnosis model trained on comprehensive sf-PET/MR data with the power of, during inferencing, allowing single modality input (e.g., PET only) as well as enforcing multimodal-based accuracy. To this end, we propose MX-ARM, a multimodal MiXture-of-experts Alignment and Reconstruction Model. It is modality detachable and exchangeable, allocating different multi-layer perceptrons dynamically ("mixture of experts") through learnable weights to learn respective representations from different modalities. Such design will not sacrifice model performance in uni-modal situation. To fully exploit the inherent complex and nonlinear relation among modalities while producing fine-grained representations for uni-modal inference, we subsequently add a modal alignment module to line up a dominant modality (e.g., PET) with representations of auxiliary modalities (MR). We further adopt multimodal reconstruction to promote the quality of learned features. Experiments on precious multimodal sf-PET/MR data for Mild Cognitive Impairment diagnosis showcase the efficacy of our model toward clinically feasible precision medicine.

Related papers

• Multi-modal Evidential Fusion Network for Trusted PET/CT Tumor Segmentation [5.839660501978193]
  The quality of PET and CT images varies widely in clinical settings, which leads to uncertainty in the modality information extracted by networks. This paper proposes a novel Multi-modal Evidential Fusion Network (MEFN) comprising a Cross-Modal Feature Learning (CFL) module and a Multi-modal Trusted Fusion (MTF) module. Our model can provide radiologists with credible uncertainty of the segmentation results for their decision in accepting or rejecting the automatic segmentation results.
  arXiv  Detail & Related papers  (2024-06-26T13:14:24Z)
• PASTA: Pathology-Aware MRI to PET Cross-Modal Translation with Diffusion Models [7.6672160690646445]
  We introduce PASTA, a novel pathology-aware image translation framework based on conditional diffusion models. A cycle exchange consistency and volumetric generation strategy elevate PASTA's capability to produce high-quality 3D PET scans. For Alzheimer's classification, the performance of synthesized scans improves over MRI by 4%, almost reaching the performance of actual PET.
  arXiv  Detail & Related papers  (2024-05-27T08:33:24Z)
• Functional Imaging Constrained Diffusion for Brain PET Synthesis from Structural MRI [5.190302448685122]
  We propose a framework for 3D brain PET image synthesis with paired structural MRI as input condition, through a new constrained diffusion model (CDM) The FICD introduces noise to PET and then progressively removes it with CDM, ensuring high output fidelity throughout a stable training phase. The CDM learns to predict denoised PET with a functional imaging constraint introduced to ensure voxel-wise alignment between each denoised PET and its ground truth.
  arXiv  Detail & Related papers  (2024-05-03T22:33:46Z)
• Physical formula enhanced multi-task learning for pharmacokinetics prediction [54.13787789006417]
  A major challenge for AI-driven drug discovery is the scarcity of high-quality data. We develop a formula enhanced mul-ti-task learning (PEMAL) method that predicts four key parameters of pharmacokinetics simultaneously. Our experiments reveal that PEMAL significantly lowers the data demand, compared to typical Graph Neural Networks.
  arXiv  Detail & Related papers  (2024-04-16T07:42:55Z)
• MMA-DFER: MultiModal Adaptation of unimodal models for Dynamic Facial Expression Recognition in-the-wild [81.32127423981426]
  Multimodal emotion recognition based on audio and video data is important for real-world applications. Recent methods have focused on exploiting advances of self-supervised learning (SSL) for pre-training of strong multimodal encoders. We propose a different perspective on the problem and investigate the advancement of multimodal DFER performance by adapting SSL-pre-trained disjoint unimodal encoders.
  arXiv  Detail & Related papers  (2024-04-13T13:39:26Z)
• Joint Diffusion: Mutual Consistency-Driven Diffusion Model for PET-MRI Co-Reconstruction [19.790873500057355]
  The study aims to accelerate MRI and enhance PET image quality. Conventional approaches involve the separate reconstruction of each modality within PET-MRI systems. We propose a novel PET-MRI joint reconstruction model employing a mutual consistency-driven diffusion mode, namely MC-Diffusion.
  arXiv  Detail & Related papers  (2023-11-24T13:26:53Z)
• Contrastive Diffusion Model with Auxiliary Guidance for Coarse-to-Fine PET Reconstruction [62.29541106695824]
  This paper presents a coarse-to-fine PET reconstruction framework that consists of a coarse prediction module (CPM) and an iterative refinement module (IRM) By delegating most of the computational overhead to the CPM, the overall sampling speed of our method can be significantly improved. Two additional strategies, i.e., an auxiliary guidance strategy and a contrastive diffusion strategy, are proposed and integrated into the reconstruction process.
  arXiv  Detail & Related papers  (2023-08-20T04:10:36Z)
• Incomplete Multimodal Learning for Complex Brain Disorders Prediction [65.95783479249745]
  We propose a new incomplete multimodal data integration approach that employs transformers and generative adversarial networks. We apply our new method to predict cognitive degeneration and disease outcomes using the multimodal imaging genetic data from Alzheimer's Disease Neuroimaging Initiative cohort.
  arXiv  Detail & Related papers  (2023-05-25T16:29:16Z)
• Multimodal foundation models are better simulators of the human brain [65.10501322822881]
  We present a newly-designed multimodal foundation model pre-trained on 15 million image-text pairs. We find that both visual and lingual encoders trained multimodally are more brain-like compared with unimodal ones.
  arXiv  Detail & Related papers  (2022-08-17T12:36:26Z)
• Synthesizing Multi-Tracer PET Images for Alzheimer's Disease Patients using a 3D Unified Anatomy-aware Cyclic Adversarial Network [9.406405460188818]
  Positron Emission Tomography (PET) is an important tool for studying Alzheimer's disease (AD) Previous works on medical image synthesis focus on one-to-one fixed domain translations, and cannot simultaneously learn the feature from multi-tracer domains. We propose a 3D unified anatomy-aware cyclic adversarial network (UCAN) for translating multi-tracer PET volumes with one unified generative model.
  arXiv  Detail & Related papers  (2021-07-12T15:10:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.