Cross-Modal Synthesis of Structural MRI and Functional Connectivity Networks via Conditional ViT-GANs

• URL: http://arxiv.org/abs/2309.08160v1
• Date: Fri, 15 Sep 2023 05:03:08 GMT
• Title: Cross-Modal Synthesis of Structural MRI and Functional Connectivity Networks via Conditional ViT-GANs
• Authors: Yuda Bi, Anees Abrol, Jing Sui, and Vince Calhoun
• Abstract summary: Cross-modal synthesis between structural magnetic resonance imaging (sMRI) and functional network connectivity (FNC) is relatively unexplored in medical imaging. This study employs conditional Vision Transformer Geneversarative Adrial Networks (cViT-GANs) to generate FNC data based on sMRI inputs.
• Score: 0.8778841570220198
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The cross-modal synthesis between structural magnetic resonance imaging (sMRI) and functional network connectivity (FNC) is a relatively unexplored area in medical imaging, especially with respect to schizophrenia. This study employs conditional Vision Transformer Generative Adversarial Networks (cViT-GANs) to generate FNC data based on sMRI inputs. After training on a comprehensive dataset that included both individuals with schizophrenia and healthy control subjects, our cViT-GAN model effectively synthesized the FNC matrix for each subject, and then formed a group difference FNC matrix, obtaining a Pearson correlation of 0.73 with the actual FNC matrix. In addition, our FNC visualization results demonstrate significant correlations in particular subcortical brain regions, highlighting the model's capability of capturing detailed structural-functional associations. This performance distinguishes our model from conditional CNN-based GAN alternatives such as Pix2Pix. Our research is one of the first attempts to link sMRI and FNC synthesis, setting it apart from other cross-modal studies that concentrate on T1- and T2-weighted MR images or the fusion of MRI and CT scans.

Related papers

• Copula-Linked Parallel ICA: A Method for Coupling Structural and Functional MRI brain Networks [0.5277756703318045]
  Prior studies fusing functional MRI (fMRI) and structural MRI (sMRI) have shown the benefits of this approach. We developed a novel fusion method, by combining deep learning frameworks, copulas and independent component analysis (ICA), named copula linked parallel ICA (CLiP-ICA) CLiP-ICA effectively captures both strongly and weakly linked sMRI and fMRI networks, including the cerebellum, sensorimotor, visual, cognitive control, and default mode networks.
  arXiv  Detail & Related papers  (2024-10-14T01:35:41Z)
• MindFormer: Semantic Alignment of Multi-Subject fMRI for Brain Decoding [50.55024115943266]
  We introduce a novel semantic alignment method of multi-subject fMRI signals using so-called MindFormer. This model is specifically designed to generate fMRI-conditioned feature vectors that can be used for conditioning Stable Diffusion model for fMRI- to-image generation or large language model (LLM) for fMRI-to-text generation. Our experimental results demonstrate that MindFormer generates semantically consistent images and text across different subjects.
  arXiv  Detail & Related papers  (2024-05-28T00:36:25Z)
• NeuroPictor: Refining fMRI-to-Image Reconstruction via Multi-individual Pretraining and Multi-level Modulation [55.51412454263856]
  This paper proposes to directly modulate the generation process of diffusion models using fMRI signals. By training with about 67,000 fMRI-image pairs from various individuals, our model enjoys superior fMRI-to-image decoding capacity.
  arXiv  Detail & Related papers  (2024-03-27T02:42:52Z)
• SDR-Former: A Siamese Dual-Resolution Transformer for Liver Lesion Classification Using 3D Multi-Phase Imaging [59.78761085714715]
  This study proposes a novel Siamese Dual-Resolution Transformer (SDR-Former) framework for liver lesion classification. The proposed framework has been validated through comprehensive experiments on two clinical datasets. To support the scientific community, we are releasing our extensive multi-phase MR dataset for liver lesion analysis to the public.
  arXiv  Detail & Related papers  (2024-02-27T06:32:56Z)
• DiffGAN-F2S: Symmetric and Efficient Denoising Diffusion GANs for Structural Connectivity Prediction from Brain fMRI [15.40111168345568]
  It is challenging to bridge the reliable non-linear mapping relations between structural connectivity (SC) and functional magnetic resonance imaging (fMRI) A novel diffusision generative adversarial network-based fMRI-to-SC model is proposed to predict SC from brain fMRI in an end-to-end manner.
  arXiv  Detail & Related papers  (2023-09-28T06:55:50Z)
• Controllable Mind Visual Diffusion Model [58.83896307930354]
  Brain signal visualization has emerged as an active research area, serving as a critical interface between the human visual system and computer vision models. We propose a novel approach, referred to as Controllable Mind Visual Model Diffusion (CMVDM) CMVDM extracts semantic and silhouette information from fMRI data using attribute alignment and assistant networks. We then leverage a control model to fully exploit the extracted information for image synthesis, resulting in generated images that closely resemble the visual stimuli in terms of semantics and silhouette.
  arXiv  Detail & Related papers  (2023-05-17T11:36:40Z)
• High-fidelity Direct Contrast Synthesis from Magnetic Resonance Fingerprinting [28.702553164811473]
  We propose a supervised learning-based method that directly synthesizes contrast-weighted images from the MRF data without going through the quantitative mapping and spin-dynamics simulation. In-vivo experiments demonstrate excellent image quality compared to simulation-based contrast synthesis and previous DCS methods, both visually as well as by quantitative metrics.
  arXiv  Detail & Related papers  (2022-12-21T07:11:39Z)
• Confidence-guided Lesion Mask-based Simultaneous Synthesis of Anatomic and Molecular MR Images in Patients with Post-treatment Malignant Gliomas [65.64363834322333]
  Confidence Guided SAMR (CG-SAMR) synthesizes data from lesion information to multi-modal anatomic sequences. module guides the synthesis based on confidence measure about the intermediate results. experiments on real clinical data demonstrate that the proposed model can perform better than the state-of-theart synthesis methods.
  arXiv  Detail & Related papers  (2020-08-06T20:20:22Z)
• Lesion Mask-based Simultaneous Synthesis of Anatomic and MolecularMR Images using a GAN [59.60954255038335]
  The proposed framework consists of a stretch-out up-sampling module, a brain atlas encoder, a segmentation consistency module, and multi-scale label-wise discriminators. Experiments on real clinical data demonstrate that the proposed model can perform significantly better than the state-of-the-art synthesis methods.
  arXiv  Detail & Related papers  (2020-06-26T02:50:09Z)
• Multi-Modality Generative Adversarial Networks with Tumor Consistency Loss for Brain MR Image Synthesis [30.64847799586407]
  We propose a multi-modality generative adversarial network (MGAN) to synthesize three high-quality MR modalities (FLAIR, T1 and T1ce) from one MR modality T2 simultaneously. The experimental results show that the quality of the synthesized images is better than the one synthesized by the baseline model, pix2pix.
  arXiv  Detail & Related papers  (2020-05-02T21:33:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.