MultiViT2: A Data-augmented Multimodal Neuroimaging Prediction Framework via Latent Diffusion Model

• URL: http://arxiv.org/abs/2506.13667v1
• Date: Mon, 16 Jun 2025 16:25:13 GMT
• Title: MultiViT2: A Data-augmented Multimodal Neuroimaging Prediction Framework via Latent Diffusion Model
• Authors: Bi Yuda, Jia Sihan, Gao Yutong, Abrol Anees, Fu Zening, Calhoun Vince,
• Abstract summary: Multimodal medical imaging integrates diverse data types, such as structural and functional neuroimaging.<n>This study focuses on a neuroimaging prediction framework based on both structural and functional neuroimaging data.<n>We show that MultiViT2 significantly outperforms the first-generation model in schizophrenia classification accuracy.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal medical imaging integrates diverse data types, such as structural and functional neuroimaging, to provide complementary insights that enhance deep learning predictions and improve outcomes. This study focuses on a neuroimaging prediction framework based on both structural and functional neuroimaging data. We propose a next-generation prediction model, \textbf{MultiViT2}, which combines a pretrained representative learning base model with a vision transformer backbone for prediction output. Additionally, we developed a data augmentation module based on the latent diffusion model that enriches input data by generating augmented neuroimaging samples, thereby enhancing predictive performance through reduced overfitting and improved generalizability. We show that MultiViT2 significantly outperforms the first-generation model in schizophrenia classification accuracy and demonstrates strong scalability and portability.

Related papers

• MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings [75.0617088717528]
  MoCa is a framework for transforming pre-trained VLM backbones into effective bidirectional embedding models.<n>MoCa consistently improves performance across MMEB and ViDoRe-v2 benchmarks, achieving new state-of-the-art results.
  arXiv  Detail & Related papers  (2025-06-29T06:41:00Z)
• Beyond DAGs: A Latent Partial Causal Model for Multimodal Learning [80.44084021062105]
  We propose a novel latent partial causal model for multimodal data, featuring two latent coupled variables, connected by an undirected edge, to represent the transfer of knowledge across modalities.<n>Under specific statistical assumptions, we establish an identifiability result, demonstrating that representations learned by multimodal contrastive learning correspond to the latent coupled variables up to a trivial transformation.<n>Experiments on a pre-trained CLIP model embodies disentangled representations, enabling few-shot learning and improving domain generalization across diverse real-world datasets.
  arXiv  Detail & Related papers  (2024-02-09T07:18:06Z)
• MPRE: Multi-perspective Patient Representation Extractor for Disease Prediction [3.914545513460964]
  We propose the Multi-perspective Patient Representation Extractor (MPRE) for disease prediction. Specifically, we propose Frequency Transformation Module (FTM) to extract the trend and variation information of dynamic features. In the 2D Multi-Extraction Network (2D MEN), we form the 2D temporal tensor based on trend and variation. We also propose the First-Order Difference Attention Mechanism (FODAM) to calculate the contributions of differences in adjacent variations to the disease diagnosis.
  arXiv  Detail & Related papers  (2024-01-01T13:52:05Z)
• PheME: A deep ensemble framework for improving phenotype prediction from multi-modal data [42.56953523499849]
  We present PheME, an Ensemble framework using Multi-modality data of structured EHRs and unstructured clinical notes for accurate Phenotype prediction. We leverage ensemble learning to combine outputs from single-modal models and multi-modal models to improve phenotype predictions.
  arXiv  Detail & Related papers  (2023-03-19T23:41:04Z)
• Two-step interpretable modeling of Intensive Care Acquired Infections [0.0]
  We present a novel methodology for integrating high resolution longitudinal data with the dynamic prediction capabilities of survival models. The aim is two-fold: to improve the predictive power while maintaining interpretability of the models.
  arXiv  Detail & Related papers  (2023-01-26T14:54:17Z)
• On the Generalization and Adaption Performance of Causal Models [99.64022680811281]
  Differentiable causal discovery has proposed to factorize the data generating process into a set of modules. We study the generalization and adaption performance of such modular neural causal models. Our analysis shows that the modular neural causal models outperform other models on both zero and few-shot adaptation in low data regimes.
  arXiv  Detail & Related papers  (2022-06-09T17:12:32Z)
• DeepAD: A Robust Deep Learning Model of Alzheimer's Disease Progression for Real-World Clinical Applications [0.9999629695552196]
  We propose a novel multi-task deep learning model to predict Alzheimer's disease progression. Our model integrates high dimensional MRI features from a 3D convolutional neural network with other data modalities.
  arXiv  Detail & Related papers  (2022-03-17T05:42:00Z)
• EINNs: Epidemiologically-Informed Neural Networks [75.34199997857341]
  We introduce a new class of physics-informed neural networks-EINN-crafted for epidemic forecasting. We investigate how to leverage both the theoretical flexibility provided by mechanistic models as well as the data-driven expressability afforded by AI models.
  arXiv  Detail & Related papers  (2022-02-21T18:59:03Z)
• A multi-stage machine learning model on diagnosis of esophageal manometry [50.591267188664666]
  The framework includes deep-learning models at the swallow-level stage and feature-based machine learning models at the study-level stage. This is the first artificial-intelligence-style model to automatically predict CC diagnosis of HRM study from raw multi-swallow data.
  arXiv  Detail & Related papers  (2021-06-25T20:09:23Z)
• DeepCOVIDNet: An Interpretable Deep Learning Model for Predictive Surveillance of COVID-19 Using Heterogeneous Features and their Interactions [2.30238915794052]
  We propose a deep learning model to forecast the range of increase in COVID-19 infected cases in future days. Using data collected from various sources, we estimate the range of increase in infected cases seven days into the future for all U.S. counties.
  arXiv  Detail & Related papers  (2020-07-31T23:37:38Z)
• Ensemble Transfer Learning for the Prediction of Anti-Cancer Drug Response [49.86828302591469]
  In this paper, we apply transfer learning to the prediction of anti-cancer drug response. We apply the classic transfer learning framework that trains a prediction model on the source dataset and refines it on the target dataset. The ensemble transfer learning pipeline is implemented using LightGBM and two deep neural network (DNN) models with different architectures.
  arXiv  Detail & Related papers  (2020-05-13T20:29:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.