MediSyn: A Generalist Text-Guided Latent Diffusion Model For Diverse Medical Image Synthesis

• URL: http://arxiv.org/abs/2405.09806v4
• Date: Mon, 10 Feb 2025 20:00:24 GMT
• Title: MediSyn: A Generalist Text-Guided Latent Diffusion Model For Diverse Medical Image Synthesis
• Authors: Joseph Cho, Mrudang Mathur, Cyril Zakka, Dhamanpreet Kaur, Matthew Leipzig, Alex Dalal, Aravind Krishnan, Eubee Koo, Karen Wai, Cindy S. Zhao, Rohan Shad, Robyn Fong, Ross Wightman, Akshay Chaudhari, William Hiesinger,
• Abstract summary: MediSyn is a text-guided latent diffusion model capable of generating synthetic images from 6 medical specialties and 10 image types.<n>A direct comparison of the synthetic images against the real images confirms that our model synthesizes novel images and, crucially, may preserve patient privacy.<n>Our findings highlight the immense potential for generalist image generative models to accelerate algorithmic research and development in medicine.
• Score: 4.541407789437896
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Deep learning algorithms require extensive data to achieve robust performance. However, data availability is often restricted in the medical domain due to patient privacy concerns. Synthetic data presents a possible solution to these challenges. Recently, image generative models have found increasing use for medical applications but are often designed for singular medical specialties and imaging modalities, thus limiting their broader utility. To address this, we introduce MediSyn: a text-guided, latent diffusion model capable of generating synthetic images from 6 medical specialties and 10 image types. The synthetic images are validated by expert clinicians for alignment with their corresponding text prompts. Furthermore, a direct comparison of the synthetic images against the real images confirms that our model synthesizes novel images and, crucially, may preserve patient privacy. Finally, classifiers trained on a mixture of synthetic and real data achieve similar performance to those trained on twice the amount of real data. Our findings highlight the immense potential for generalist image generative models to accelerate algorithmic research and development in medicine.

Related papers

• Using Synthetic Images to Augment Small Medical Image Datasets [3.7522420000453]
  We have developed a novel conditional variant of a current GAN method, the StyleGAN2, to generate high-resolution medical images. We use the synthetic and real images from six datasets to train models for the downstream task of semantic segmentation. The quality of the generated medical images and the effect of this augmentation on the segmentation performance were evaluated afterward.
  arXiv  Detail & Related papers  (2025-03-02T17:02:11Z)
• Latent Drifting in Diffusion Models for Counterfactual Medical Image Synthesis [55.959002385347645]
  Latent Drifting enables diffusion models to be conditioned for medical images fitted for the complex task of counterfactual image generation. We evaluate our method on three public longitudinal benchmark datasets of brain MRI and chest X-rays for counterfactual image generation.
  arXiv  Detail & Related papers  (2024-12-30T01:59:34Z)
• MRGen: Segmentation Data Engine For Underrepresented MRI Modalities [59.61465292965639]
  Training medical image segmentation models for rare yet clinically significant imaging modalities is challenging due to the scarcity of annotated data. This paper investigates leveraging generative models to synthesize training data, to train segmentation models for underrepresented modalities.
  arXiv  Detail & Related papers  (2024-12-04T16:34:22Z)
• Deep Generative Models for 3D Medical Image Synthesis [1.931185411277237]
  Deep generative modeling has emerged as a powerful tool for synthesizing realistic medical images. This chapter explores various deep generative models for 3D medical image synthesis.
  arXiv  Detail & Related papers  (2024-10-23T08:33:23Z)
• Autoregressive Sequence Modeling for 3D Medical Image Representation [48.706230961589924]
  We introduce a pioneering method for learning 3D medical image representations through an autoregressive sequence pre-training framework. Our approach various 3D medical images based on spatial, contrast, and semantic correlations, treating them as interconnected visual tokens within a token sequence.
  arXiv  Detail & Related papers  (2024-09-13T10:19:10Z)
• Vision-Language Synthetic Data Enhances Echocardiography Downstream Tasks [4.1942958779358674]
  This paper utilizes recent vision-language models to produce diverse and realistic synthetic echocardiography image data. We show that the rich contextual information present in the synthesized data potentially enhances the accuracy and interpretability of downstream tasks.
  arXiv  Detail & Related papers  (2024-03-28T23:26:45Z)
• Is Synthetic Image Useful for Transfer Learning? An Investigation into Data Generation, Volume, and Utilization [62.157627519792946]
  We introduce a novel framework called bridged transfer, which initially employs synthetic images for fine-tuning a pre-trained model to improve its transferability. We propose dataset style inversion strategy to improve the stylistic alignment between synthetic and real images. Our proposed methods are evaluated across 10 different datasets and 5 distinct models, demonstrating consistent improvements.
  arXiv  Detail & Related papers  (2024-03-28T22:25:05Z)
• Generative Enhancement for 3D Medical Images [74.17066529847546]
  We propose GEM-3D, a novel generative approach to the synthesis of 3D medical images. Our method begins with a 2D slice, noted as the informed slice to serve the patient prior, and propagates the generation process using a 3D segmentation mask. By decomposing the 3D medical images into masks and patient prior information, GEM-3D offers a flexible yet effective solution for generating versatile 3D images.
  arXiv  Detail & Related papers  (2024-03-19T15:57:04Z)
• VolumeDiffusion: Flexible Text-to-3D Generation with Efficient Volumetric Encoder [56.59814904526965]
  This paper introduces a pioneering 3D encoder designed for text-to-3D generation. A lightweight network is developed to efficiently acquire feature volumes from multi-view images. The 3D volumes are then trained on a diffusion model for text-to-3D generation using a 3D U-Net.
  arXiv  Detail & Related papers  (2023-12-18T18:59:05Z)
• Adaptive Latent Diffusion Model for 3D Medical Image to Image Translation: Multi-modal Magnetic Resonance Imaging Study [4.3536336830666755]
  Multi-modal images play a crucial role in comprehensive evaluations in medical image analysis. In clinical practice, acquiring multiple modalities can be challenging due to reasons such as scan cost, limited scan time, and safety considerations. We propose a model that leverages switchable blocks for image-to-image translation in 3D medical images without patch cropping.
  arXiv  Detail & Related papers  (2023-11-01T03:22:57Z)
• EMIT-Diff: Enhancing Medical Image Segmentation via Text-Guided Diffusion Model [4.057796755073023]
  We develop controllable diffusion models for medical image synthesis, called EMIT-Diff. We leverage recent diffusion probabilistic models to generate realistic and diverse synthetic medical image data. In our approach, we ensure that the synthesized samples adhere to medically relevant constraints.
  arXiv  Detail & Related papers  (2023-10-19T16:18:02Z)
• MedSyn: Text-guided Anatomy-aware Synthesis of High-Fidelity 3D CT Images [22.455833806331384]
  This paper introduces an innovative methodology for producing high-quality 3D lung CT images guided by textual information. Current state-of-the-art approaches are limited to low-resolution outputs and underutilize radiology reports' abundant information.
  arXiv  Detail & Related papers  (2023-10-05T14:16:22Z)
• Augmenting medical image classifiers with synthetic data from latent diffusion models [12.077733447347592]
  We show that latent diffusion models can scalably generate images of skin disease. We generate and analyze a new dataset of 458,920 synthetic images produced using several generation strategies.
  arXiv  Detail & Related papers  (2023-08-23T22:34:49Z)
• IT3D: Improved Text-to-3D Generation with Explicit View Synthesis [71.68595192524843]
  This study presents a novel strategy that leverages explicitly synthesized multi-view images to address these issues. Our approach involves the utilization of image-to-image pipelines, empowered by LDMs, to generate posed high-quality images. For the incorporated discriminator, the synthesized multi-view images are considered real data, while the renderings of the optimized 3D models function as fake data.
  arXiv  Detail & Related papers  (2023-08-22T14:39:17Z)
• Make-A-Volume: Leveraging Latent Diffusion Models for Cross-Modality 3D Brain MRI Synthesis [35.45013834475523]
  Cross-modality medical image synthesis is a critical topic and has the potential to facilitate numerous applications in the medical imaging field. Most current medical image synthesis methods rely on generative adversarial networks and suffer from notorious mode collapse and unstable training. We introduce a new paradigm for volumetric medical data synthesis by leveraging 2D backbones and present a diffusion-based framework, Make-A-Volume.
  arXiv  Detail & Related papers  (2023-07-19T16:01:09Z)
• Image Captions are Natural Prompts for Text-to-Image Models [70.30915140413383]
  We analyze the relationship between the training effect of synthetic data and the synthetic data distribution induced by prompts. We propose a simple yet effective method that prompts text-to-image generative models to synthesize more informative and diverse training data. Our method significantly improves the performance of models trained on synthetic training data.
  arXiv  Detail & Related papers  (2023-07-17T14:38:11Z)
• Medical Diffusion -- Denoising Diffusion Probabilistic Models for 3D Medical Image Generation [0.6486409713123691]
  We show that diffusion probabilistic models can synthesize high quality medical imaging data. We provide quantitative measurements of their performance through a reader study with two medical experts. We demonstrate that synthetic images can be used in a self-supervised pre-training and improve the performance of breast segmentation models when data is scarce.
  arXiv  Detail & Related papers  (2022-11-07T08:37:48Z)
• Is synthetic data from generative models ready for image recognition? [69.42645602062024]
  We study whether and how synthetic images generated from state-of-the-art text-to-image generation models can be used for image recognition tasks. We showcase the powerfulness and shortcomings of synthetic data from existing generative models, and propose strategies for better applying synthetic data for recognition tasks.
  arXiv  Detail & Related papers  (2022-10-14T06:54:24Z)
• Generative Adversarial U-Net for Domain-free Medical Image Augmentation [49.72048151146307]
  The shortage of annotated medical images is one of the biggest challenges in the field of medical image computing. In this paper, we develop a novel generative method named generative adversarial U-Net. Our newly designed model is domain-free and generalizable to various medical images.
  arXiv  Detail & Related papers  (2021-01-12T23:02:26Z)
• Modelling the Distribution of 3D Brain MRI using a 2D Slice VAE [66.63629641650572]
  We propose a method to model 3D MR brain volumes distribution by combining a 2D slice VAE with a Gaussian model that captures the relationships between slices. We also introduce a novel evaluation method for generated volumes that quantifies how well their segmentations match those of true brain anatomy.
  arXiv  Detail & Related papers  (2020-07-09T13:23:15Z)
• Diffusion-Weighted Magnetic Resonance Brain Images Generation with Generative Adversarial Networks and Variational Autoencoders: A Comparison Study [55.78588835407174]
  We show that high quality, diverse and realistic-looking diffusion-weighted magnetic resonance images can be synthesized using deep generative models. We present two networks, the Introspective Variational Autoencoder and the Style-Based GAN, that qualify for data augmentation in the medical field.
  arXiv  Detail & Related papers  (2020-06-24T18:00:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.