Towards General Text-guided Image Synthesis for Customized Multimodal Brain MRI Generation

• URL: http://arxiv.org/abs/2409.16818v1
• Date: Wed, 25 Sep 2024 11:14:47 GMT
• Title: Towards General Text-guided Image Synthesis for Customized Multimodal Brain MRI Generation
• Authors: Yulin Wang, Honglin Xiong, Kaicong Sun, Shuwei Bai, Ling Dai, Zhongxiang Ding, Jiameng Liu, Qian Wang, Qian Liu, Dinggang Shen,
• Abstract summary: Multimodal brain magnetic resonance (MR) imaging is indispensable in neuroscience and neurology. Current MR image synthesis approaches are typically trained on independent datasets for specific tasks. We present TUMSyn, a Text-guided Universal MR image Synthesis model, which can flexibly generate brain MR images.
• Score: 51.28453192441364
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal brain magnetic resonance (MR) imaging is indispensable in neuroscience and neurology. However, due to the accessibility of MRI scanners and their lengthy acquisition time, multimodal MR images are not commonly available. Current MR image synthesis approaches are typically trained on independent datasets for specific tasks, leading to suboptimal performance when applied to novel datasets and tasks. Here, we present TUMSyn, a Text-guided Universal MR image Synthesis generalist model, which can flexibly generate brain MR images with demanded imaging metadata from routinely acquired scans guided by text prompts. To ensure TUMSyn's image synthesis precision, versatility, and generalizability, we first construct a brain MR database comprising 31,407 3D images with 7 MRI modalities from 13 centers. We then pre-train an MRI-specific text encoder using contrastive learning to effectively control MR image synthesis based on text prompts. Extensive experiments on diverse datasets and physician assessments indicate that TUMSyn can generate clinically meaningful MR images with specified imaging metadata in supervised and zero-shot scenarios. Therefore, TUMSyn can be utilized along with acquired MR scan(s) to facilitate large-scale MRI-based screening and diagnosis of brain diseases.

Related papers

• Synthetic Brain Images: Bridging the Gap in Brain Mapping With Generative Adversarial Model [0.0]
  This work investigates the use of Deep Convolutional Generative Adversarial Networks (DCGAN) for producing high-fidelity and realistic MRI image slices. While the discriminator network discerns between created and real slices, the generator network learns to synthesise realistic MRI image slices. The generator refines its capacity to generate slices that closely mimic real MRI data through an adversarial training approach.
  arXiv  Detail & Related papers  (2024-04-11T05:06:51Z)
• Disentangled Multimodal Brain MR Image Translation via Transformer-based Modality Infuser [12.402947207350394]
  We propose a transformer-based modality infuser designed to synthesize multimodal brain MR images. In our method, we extract modality-agnostic features from the encoder and then transform them into modality-specific features. We carried out experiments on the BraTS 2018 dataset, translating between four MR modalities.
  arXiv  Detail & Related papers  (2024-02-01T06:34:35Z)
• fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decoding [54.17776744076334]
  We propose fMRI-PTE, an innovative auto-encoder approach for fMRI pre-training. Our approach involves transforming fMRI signals into unified 2D representations, ensuring consistency in dimensions and preserving brain activity patterns. Our contributions encompass introducing fMRI-PTE, innovative data transformation, efficient training, a novel learning strategy, and the universal applicability of our approach.
  arXiv  Detail & Related papers  (2023-11-01T07:24:22Z)
• Style transfer between Microscopy and Magnetic Resonance Imaging via Generative Adversarial Network in small sample size settings [49.84018914962972]
  Cross-modal augmentation of Magnetic Resonance Imaging (MRI) and microscopic imaging based on the same tissue samples is promising. We tested a method for generating microscopic histological images from MRI scans of the corpus callosum using conditional generative adversarial network (cGAN) architecture.
  arXiv  Detail & Related papers  (2023-10-16T13:58:53Z)
• CMRxRecon: An open cardiac MRI dataset for the competition of accelerated image reconstruction [62.61209705638161]
  There has been growing interest in deep learning-based CMR imaging algorithms. Deep learning methods require large training datasets. This dataset includes multi-contrast, multi-view, multi-slice and multi-coil CMR imaging data from 300 subjects.
  arXiv  Detail & Related papers  (2023-09-19T15:14:42Z)
• The Brain Tumor Segmentation (BraTS) Challenge 2023: Brain MR Image Synthesis for Tumor Segmentation (BraSyn) [5.399839183476989]
  We present the establishment of the Brain MR Image Synthesis Benchmark (BraSyn) in conjunction with the Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2023. The primary objective of this challenge is to evaluate image synthesis methods that can realistically generate missing MRI modalities when multiple available images are provided.
  arXiv  Detail & Related papers  (2023-05-15T20:49:58Z)
• BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decoding [51.911473457195555]
  BrainCLIP is a task-agnostic fMRI-based brain decoding model. It bridges the modality gap between brain activity, image, and text. BrainCLIP can reconstruct visual stimuli with high semantic fidelity.
  arXiv  Detail & Related papers  (2023-02-25T03:28:54Z)
• Multi-modal Aggregation Network for Fast MR Imaging [85.25000133194762]
  We propose a novel Multi-modal Aggregation Network, named MANet, which is capable of discovering complementary representations from a fully sampled auxiliary modality. In our MANet, the representations from the fully sampled auxiliary and undersampled target modalities are learned independently through a specific network. Our MANet follows a hybrid domain learning framework, which allows it to simultaneously recover the frequency signal in the $k$-space domain.
  arXiv  Detail & Related papers  (2021-10-15T13:16:59Z)
• Enhanced Magnetic Resonance Image Synthesis with Contrast-Aware Generative Adversarial Networks [5.3580471186206005]
  We trained a generative adversarial network (GAN) to generate synthetic MR knee images conditioned on various acquisition parameters. In a Turing test, two experts mislabeled 40.5% of real and synthetic MR images, demonstrating that the image quality of the generated synthetic and real MR images is comparable.
  arXiv  Detail & Related papers  (2021-02-17T11:39:36Z)
• Self-Supervised Ultrasound to MRI Fetal Brain Image Synthesis [20.53251934808636]
  Fetal brain magnetic resonance imaging (MRI) offers exquisite images of the developing brain but is not suitable for second-trimester anomaly screening. In this paper we propose to generate MR-like images directly from clinical US images. The proposed model is end-to-end trainable and self-supervised without any external annotations.
  arXiv  Detail & Related papers  (2020-08-19T22:56:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.