Make-A-Volume: Leveraging Latent Diffusion Models for Cross-Modality 3D Brain MRI Synthesis

• URL: http://arxiv.org/abs/2307.10094v1
• Date: Wed, 19 Jul 2023 16:01:09 GMT
• Title: Make-A-Volume: Leveraging Latent Diffusion Models for Cross-Modality 3D Brain MRI Synthesis
• Authors: Lingting Zhu, Zeyue Xue, Zhenchao Jin, Xian Liu, Jingzhen He, Ziwei Liu, Lequan Yu
• Abstract summary: Cross-modality medical image synthesis is a critical topic and has the potential to facilitate numerous applications in the medical imaging field. Most current medical image synthesis methods rely on generative adversarial networks and suffer from notorious mode collapse and unstable training. We introduce a new paradigm for volumetric medical data synthesis by leveraging 2D backbones and present a diffusion-based framework, Make-A-Volume.
• Score: 35.45013834475523
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Cross-modality medical image synthesis is a critical topic and has the potential to facilitate numerous applications in the medical imaging field. Despite recent successes in deep-learning-based generative models, most current medical image synthesis methods rely on generative adversarial networks and suffer from notorious mode collapse and unstable training. Moreover, the 2D backbone-driven approaches would easily result in volumetric inconsistency, while 3D backbones are challenging and impractical due to the tremendous memory cost and training difficulty. In this paper, we introduce a new paradigm for volumetric medical data synthesis by leveraging 2D backbones and present a diffusion-based framework, Make-A-Volume, for cross-modality 3D medical image synthesis. To learn the cross-modality slice-wise mapping, we employ a latent diffusion model and learn a low-dimensional latent space, resulting in high computational efficiency. To enable the 3D image synthesis and mitigate volumetric inconsistency, we further insert a series of volumetric layers in the 2D slice-mapping model and fine-tune them with paired 3D data. This paradigm extends the 2D image diffusion model to a volumetric version with a slightly increasing number of parameters and computation, offering a principled solution for generic cross-modality 3D medical image synthesis. We showcase the effectiveness of our Make-A-Volume framework on an in-house SWI-MRA brain MRI dataset and a public T1-T2 brain MRI dataset. Experimental results demonstrate that our framework achieves superior synthesis results with volumetric consistency.

Related papers

• Cross-D Conv: Cross-Dimensional Transferable Knowledge Base via Fourier Shifting Operation [3.69758875412828]
  Cross-D Conv operation bridges the dimensional gap by learning the phase shifting in the Fourier domain. Our method enables seamless weight transfer between 2D and 3D convolution operations, effectively facilitating cross-dimensional learning.
  arXiv  Detail & Related papers  (2024-11-02T13:03:44Z)
• Generative Enhancement for 3D Medical Images [74.17066529847546]
  We propose GEM-3D, a novel generative approach to the synthesis of 3D medical images. Our method begins with a 2D slice, noted as the informed slice to serve the patient prior, and propagates the generation process using a 3D segmentation mask. By decomposing the 3D medical images into masks and patient prior information, GEM-3D offers a flexible yet effective solution for generating versatile 3D images.
  arXiv  Detail & Related papers  (2024-03-19T15:57:04Z)
• SDR-Former: A Siamese Dual-Resolution Transformer for Liver Lesion Classification Using 3D Multi-Phase Imaging [59.78761085714715]
  This study proposes a novel Siamese Dual-Resolution Transformer (SDR-Former) framework for liver lesion classification. The proposed framework has been validated through comprehensive experiments on two clinical datasets. To support the scientific community, we are releasing our extensive multi-phase MR dataset for liver lesion analysis to the public.
  arXiv  Detail & Related papers  (2024-02-27T06:32:56Z)
• Interpretable 2D Vision Models for 3D Medical Images [47.75089895500738]
  This study proposes a simple approach of adapting 2D networks with an intermediate feature representation for processing 3D images. We show on all 3D MedMNIST datasets as benchmark and two real-world datasets consisting of several hundred high-resolution CT or MRI scans that our approach performs on par with existing methods.
  arXiv  Detail & Related papers  (2023-07-13T08:27:09Z)
• Improving 3D Imaging with Pre-Trained Perpendicular 2D Diffusion Models [52.529394863331326]
  We propose a novel approach using two perpendicular pre-trained 2D diffusion models to solve the 3D inverse problem. Our method is highly effective for 3D medical image reconstruction tasks, including MRI Z-axis super-resolution, compressed sensing MRI, and sparse-view CT.
  arXiv  Detail & Related papers  (2023-03-15T08:28:06Z)
• Pathology Synthesis of 3D-Consistent Cardiac MR Images using 2D VAEs and GANs [0.5039813366558306]
  We propose a method for generating labeled data for the application of supervised deep-learning (DL) training. The image synthesis consists of label deformation and label-to-image translation tasks. We demonstrate that such an approach could provide a solution to diversify and enrich an available database of cardiac MR images.
  arXiv  Detail & Related papers  (2022-09-09T10:17:49Z)
• Medical Transformer: Universal Brain Encoder for 3D MRI Analysis [1.6287500717172143]
  Existing 3D-based methods have transferred the pre-trained models to downstream tasks. They demand a massive amount of parameters to train the model for 3D medical imaging. We propose a novel transfer learning framework, called Medical Transformer, that effectively models 3D volumetric images in the form of a sequence of 2D image slices.
  arXiv  Detail & Related papers  (2021-04-28T08:34:21Z)
• Hierarchical Amortized Training for Memory-efficient High Resolution 3D GAN [52.851990439671475]
  We propose a novel end-to-end GAN architecture that can generate high-resolution 3D images. We achieve this goal by using different configurations between training and inference. Experiments on 3D thorax CT and brain MRI demonstrate that our approach outperforms state of the art in image generation.
  arXiv  Detail & Related papers  (2020-08-05T02:33:04Z)
• Modelling the Distribution of 3D Brain MRI using a 2D Slice VAE [66.63629641650572]
  We propose a method to model 3D MR brain volumes distribution by combining a 2D slice VAE with a Gaussian model that captures the relationships between slices. We also introduce a novel evaluation method for generated volumes that quantifies how well their segmentations match those of true brain anatomy.
  arXiv  Detail & Related papers  (2020-07-09T13:23:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.