Panoptic Segmentation of Mammograms with Text-To-Image Diffusion Model

• URL: http://arxiv.org/abs/2407.14326v1
• Date: Fri, 19 Jul 2024 14:04:05 GMT
• Title: Panoptic Segmentation of Mammograms with Text-To-Image Diffusion Model
• Authors: Kun Zhao, Jakub Prokop, Javier Montalt Tordera, Sadegh Mohammadi,
• Abstract summary: Vision-language diffusion models demonstrated outstanding performance in image generation and transferability to various downstream tasks. We propose leveraging pretrained features from a Stable Diffusion model as inputs to a state-of-the-art panoptic segmentation architecture.
• Score: 1.2130800774416757
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Mammography is crucial for breast cancer surveillance and early diagnosis. However, analyzing mammography images is a demanding task for radiologists, who often review hundreds of mammograms daily, leading to overdiagnosis and overtreatment. Computer-Aided Diagnosis (CAD) systems have been developed to assist in this process, but their capabilities, particularly in lesion segmentation, remained limited. With the contemporary advances in deep learning their performance may be improved. Recently, vision-language diffusion models emerged, demonstrating outstanding performance in image generation and transferability to various downstream tasks. We aim to harness their capabilities for breast lesion segmentation in a panoptic setting, which encompasses both semantic and instance-level predictions. Specifically, we propose leveraging pretrained features from a Stable Diffusion model as inputs to a state-of-the-art panoptic segmentation architecture, resulting in accurate delineation of individual breast lesions. To bridge the gap between natural and medical imaging domains, we incorporated a mammography-specific MAM-E diffusion model and BiomedCLIP image and text encoders into this framework. We evaluated our approach on two recently published mammography datasets, CDD-CESM and VinDr-Mammo. For the instance segmentation task, we noted 40.25 AP0.1 and 46.82 AP0.05, as well as 25.44 PQ0.1 and 26.92 PQ0.05. For the semantic segmentation task, we achieved Dice scores of 38.86 and 40.92, respectively.

Related papers

• Deep BI-RADS Network for Improved Cancer Detection from Mammograms [3.686808512438363]
  We introduce a novel multi-modal approach that combines textual BI-RADS lesion descriptors with visual mammogram content. Our method employs iterative attention layers to effectively fuse these different modalities. Experiments on the CBIS-DDSM dataset demonstrate substantial improvements across all metrics.
  arXiv  Detail & Related papers  (2024-11-16T21:32:51Z)
• Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports [51.45762396192655]
  Multimodal large language models (MLLMs) have recently transformed many domains, significantly affecting the medical field. Notably, Gemini-Vision-series (Gemini) and GPT-4-series (GPT-4) models have epitomized a paradigm shift in Artificial General Intelligence for computer vision. This study evaluated the performance of the Gemini, GPT-4, and 4 popular large models for an exhaustive evaluation across 14 medical imaging datasets.
  arXiv  Detail & Related papers  (2024-07-08T09:08:42Z)
• QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge [93.61262892578067]
  Uncertainty in medical image segmentation tasks, especially inter-rater variability, presents a significant challenge. This variability directly impacts the development and evaluation of automated segmentation algorithms. We report the set-up and summarize the benchmark results of the Quantification of Uncertainties in Biomedical Image Quantification Challenge (QUBIQ)
  arXiv  Detail & Related papers  (2024-03-19T17:57:24Z)
• VALD-MD: Visual Attribution via Latent Diffusion for Medical Diagnostics [0.0]
  Visual attribution in medical imaging seeks to make evident the diagnostically-relevant components of a medical image. We here present a novel generative visual attribution technique, one that leverages latent diffusion models in combination with domain-specific large language models. The resulting system also exhibits a range of latent capabilities including zero-shot localized disease induction.
  arXiv  Detail & Related papers  (2024-01-02T19:51:49Z)
• Beyond Images: An Integrative Multi-modal Approach to Chest X-Ray Report Generation [47.250147322130545]
  Image-to-text radiology report generation aims to automatically produce radiology reports that describe the findings in medical images. Most existing methods focus solely on the image data, disregarding the other patient information accessible to radiologists. We present a novel multi-modal deep neural network framework for generating chest X-rays reports by integrating structured patient data, such as vital signs and symptoms, alongside unstructured clinical notes.
  arXiv  Detail & Related papers  (2023-11-18T14:37:53Z)
• Introducing Shape Prior Module in Diffusion Model for Medical Image Segmentation [7.7545714516743045]
  We propose an end-to-end framework called VerseDiff-UNet, which leverages the denoising diffusion probabilistic model (DDPM) Our approach integrates the diffusion model into a standard U-shaped architecture. We evaluate our method on a single dataset of spine images acquired through X-ray imaging.
  arXiv  Detail & Related papers  (2023-09-12T03:05:00Z)
• LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching [59.01894976615714]
  We introduce LVM-Med, the first family of deep networks trained on large-scale medical datasets. We have collected approximately 1.3 million medical images from 55 publicly available datasets. LVM-Med empirically outperforms a number of state-of-the-art supervised, self-supervised, and foundation models.
  arXiv  Detail & Related papers  (2023-06-20T22:21:34Z)
• XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models [60.437091462613544]
  We introduce XrayGPT, a novel conversational medical vision-language model. It can analyze and answer open-ended questions about chest radiographs. We generate 217k interactive and high-quality summaries from free-text radiology reports.
  arXiv  Detail & Related papers  (2023-06-13T17:59:59Z)
• FetReg2021: A Challenge on Placental Vessel Segmentation and Registration in Fetoscopy [52.3219875147181]
  Fetoscopic laser photocoagulation is a widely adopted procedure for treating Twin-to-Twin Transfusion Syndrome (TTTS) The procedure is particularly challenging due to the limited field of view, poor manoeuvrability of the fetoscope, poor visibility, and variability in illumination. Computer-assisted intervention (CAI) can provide surgeons with decision support and context awareness by identifying key structures in the scene and expanding the fetoscopic field of view through video mosaicking. Seven teams participated in this challenge and their model performance was assessed on an unseen test dataset of 658 pixel-annotated images from 6 fet
  arXiv  Detail & Related papers  (2022-06-24T23:44:42Z)
• VinDr-Mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography [0.5452925161262461]
  VinDr-Mammo is a new benchmark dataset of full-field digital mammography (FFDM) The dataset consists of 5,000 mammography exams, each of which has four standard views and is double read with disagreement. It is created for the assessment of Breast Imaging Reporting and Data System (BI-RADS) and density at the breast level.
  arXiv  Detail & Related papers  (2022-03-20T18:17:42Z)
• MammoGANesis: Controlled Generation of High-Resolution Mammograms for Radiology Education [0.0]
  We train a generative adversarial network (GAN) to synthesize 512 x 512 high-resolution mammograms. The resulting model leads to the unsupervised separation of high-level features. We demonstrate the model's ability to generate and medically relevant mammograms by achieving an average AUC of 0.54 in a double-blind study.
  arXiv  Detail & Related papers  (2020-10-11T06:47:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.