MedGen3D: A Deep Generative Framework for Paired 3D Image and Mask Generation

• URL: http://arxiv.org/abs/2304.04106v2
• Date: Tue, 4 Jul 2023 23:06:50 GMT
• Title: MedGen3D: A Deep Generative Framework for Paired 3D Image and Mask Generation
• Authors: Kun Han, Yifeng Xiong, Chenyu You, Pooya Khosravi, Shanlin Sun, Xiangyi Yan, James Duncan, Xiaohui Xie
• Abstract summary: We present MedGen3D, a framework that can generate paired 3D medical images and masks. Our proposed framework guarantees accurate alignment between synthetic images and segmentation maps.
• Score: 17.373961762646356
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Acquiring and annotating sufficient labeled data is crucial in developing accurate and robust learning-based models, but obtaining such data can be challenging in many medical image segmentation tasks. One promising solution is to synthesize realistic data with ground-truth mask annotations. However, no prior studies have explored generating complete 3D volumetric images with masks. In this paper, we present MedGen3D, a deep generative framework that can generate paired 3D medical images and masks. First, we represent the 3D medical data as 2D sequences and propose the Multi-Condition Diffusion Probabilistic Model (MC-DPM) to generate multi-label mask sequences adhering to anatomical geometry. Then, we use an image sequence generator and semantic diffusion refiner conditioned on the generated mask sequences to produce realistic 3D medical images that align with the generated masks. Our proposed framework guarantees accurate alignment between synthetic images and segmentation maps. Experiments on 3D thoracic CT and brain MRI datasets show that our synthetic data is both diverse and faithful to the original data, and demonstrate the benefits for downstream segmentation tasks. We anticipate that MedGen3D's ability to synthesize paired 3D medical images and masks will prove valuable in training deep learning models for medical imaging tasks.

Related papers

• Discriminative Hamiltonian Variational Autoencoder for Accurate Tumor Segmentation in Data-Scarce Regimes [2.8498944632323755]
  We propose an end-to-end hybrid architecture for medical image segmentation. We use Hamiltonian Variational Autoencoders (HVAE) and a discriminative regularization to improve the quality of generated images. Our architecture operates on a slice-by-slice basis to segment 3D volumes, capitilizing on the richly augmented dataset.
  arXiv  Detail & Related papers  (2024-06-17T15:42:08Z)
• ToNNO: Tomographic Reconstruction of a Neural Network's Output for Weakly Supervised Segmentation of 3D Medical Images [6.035125735474387]
  ToNNO is based on the Tomographic reconstruction of a Neural Network's Output. It extracts stacks of slices with different angles from the input 3D volume, feeds these slices to a 2D encoder, and applies the inverse Radon transform in order to reconstruct a 3D heatmap of the encoder's predictions. We apply it to weakly supervised medical image segmentation by training the 2D encoder to output high values for slices containing the regions of interest.
  arXiv  Detail & Related papers  (2024-04-19T11:27:56Z)
• M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models [49.5030774873328]
  Previous research has primarily focused on 2D medical images, leaving 3D images under-explored, despite their richer spatial information. We present a large-scale 3D multi-modal medical dataset, M3D-Data, comprising 120K image-text pairs and 662K instruction-response pairs. We also introduce a new 3D multi-modal medical benchmark, M3D-Bench, which facilitates automatic evaluation across eight tasks.
  arXiv  Detail & Related papers  (2024-03-31T06:55:12Z)
• Generative Enhancement for 3D Medical Images [74.17066529847546]
  We propose GEM-3D, a novel generative approach to the synthesis of 3D medical images. Our method begins with a 2D slice, noted as the informed slice to serve the patient prior, and propagates the generation process using a 3D segmentation mask. By decomposing the 3D medical images into masks and patient prior information, GEM-3D offers a flexible yet effective solution for generating versatile 3D images.
  arXiv  Detail & Related papers  (2024-03-19T15:57:04Z)
• 3D Human Reconstruction in the Wild with Synthetic Data Using Generative Models [52.96248836582542]
  We propose an effective approach based on recent diffusion models, termed HumanWild, which can effortlessly generate human images and corresponding 3D mesh annotations. By exclusively employing generative models, we generate large-scale in-the-wild human images and high-quality annotations, eliminating the need for real-world data collection.
  arXiv  Detail & Related papers  (2024-03-17T06:31:16Z)
• Disruptive Autoencoders: Leveraging Low-level features for 3D Medical Image Pre-training [51.16994853817024]
  This work focuses on designing an effective pre-training framework for 3D radiology images. We introduce Disruptive Autoencoders, a pre-training framework that attempts to reconstruct the original image from disruptions created by a combination of local masking and low-level perturbations. The proposed pre-training framework is tested across multiple downstream tasks and achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-07-31T17:59:42Z)
• 3D Matting: A Benchmark Study on Soft Segmentation Method for Pulmonary Nodules Applied in Computed Tomography [32.775884701366465]
  We introduce the image matting into the 3D scenes and use the alpha matte, i.e., a soft mask, to describe lesions in a 3D medical image. To address this issue, we conduct a comprehensive study of 3D matting, including both traditional and deep-learning-based methods. We propose the first end-to-end deep 3D matting network and implement a solid 3D medical image matting benchmark.
  arXiv  Detail & Related papers  (2022-10-11T02:40:18Z)
• 3D Matting: A Soft Segmentation Method Applied in Computed Tomography [26.25446145993599]
  Three-dimensional (3D) images, such as CT, MRI, and PET, are common in medical imaging applications and important in clinical diagnosis. Semantic ambiguity is a typical feature of many medical image labels. In 2D medical images, using soft masks instead of binary masks generated by image matting to characterize lesions can provide rich semantic information.
  arXiv  Detail & Related papers  (2022-09-16T10:18:59Z)
• 3D-Aware Semantic-Guided Generative Model for Human Synthesis [67.86621343494998]
  This paper proposes a 3D-aware Semantic-Guided Generative Model (3D-SGAN) for human image synthesis. Our experiments on the DeepFashion dataset show that 3D-SGAN significantly outperforms the most recent baselines.
  arXiv  Detail & Related papers  (2021-12-02T17:10:53Z)
• Fed-Sim: Federated Simulation for Medical Imaging [131.56325440976207]
  We introduce a physics-driven generative approach that consists of two learnable neural modules. We show that our data synthesis framework improves the downstream segmentation performance on several datasets.
  arXiv  Detail & Related papers  (2020-09-01T19:17:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.