Label-Efficient Semantic Segmentation with Diffusion Models

• URL: http://arxiv.org/abs/2112.03126v1
• Date: Mon, 6 Dec 2021 15:55:30 GMT
• Title: Label-Efficient Semantic Segmentation with Diffusion Models
• Authors: Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, Artem Babenko
• Abstract summary: We demonstrate that diffusion models can also serve as an instrument for semantic segmentation. In particular, for several pretrained diffusion models, we investigate the intermediate activations from the networks that perform the Markov step of the reverse diffusion process. We show that these activations effectively capture the semantic information from an input image and appear to be excellent pixel-level representations for the segmentation problem.
• Score: 27.01899943738203
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Denoising diffusion probabilistic models have recently received much research attention since they outperform alternative approaches, such as GANs, and currently provide state-of-the-art generative performance. The superior performance of diffusion models has made them an appealing tool in several applications, including inpainting, super-resolution, and semantic editing. In this paper, we demonstrate that diffusion models can also serve as an instrument for semantic segmentation, especially in the setup when labeled data is scarce. In particular, for several pretrained diffusion models, we investigate the intermediate activations from the networks that perform the Markov step of the reverse diffusion process. We show that these activations effectively capture the semantic information from an input image and appear to be excellent pixel-level representations for the segmentation problem. Based on these observations, we describe a simple segmentation method, which can work even if only a few training images are provided. Our approach significantly outperforms the existing alternatives on several datasets for the same amount of human supervision.

Related papers

• Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation [56.87049651707208]
  Few-shot Semantic has evolved into In-context tasks, morphing into a crucial element in assessing generalist segmentation models. Our initial focus lies in understanding how to facilitate interaction between the query image and the support image, resulting in the proposal of a KV fusion method within the self-attention framework. Based on our analysis, we establish a simple and effective framework named DiffewS, maximally retaining the original Latent Diffusion Model's generative framework.
  arXiv  Detail & Related papers  (2024-10-03T10:33:49Z)
• Enhancing Label-efficient Medical Image Segmentation with Text-guided Diffusion Models [5.865983529245793]
  TextDiff improves semantic representation through inexpensive medical text annotations. We show that TextDiff is significantly superior to the state-of-the-art multi-modal segmentation methods with only a few training samples.
  arXiv  Detail & Related papers  (2024-07-07T10:21:08Z)
• SatSynth: Augmenting Image-Mask Pairs through Diffusion Models for Aerial Semantic Segmentation [69.42764583465508]
  We explore the potential of generative image diffusion to address the scarcity of annotated data in earth observation tasks. To the best of our knowledge, we are the first to generate both images and corresponding masks for satellite segmentation.
  arXiv  Detail & Related papers  (2024-03-25T10:30:22Z)
• Leveraging Diffusion Disentangled Representations to Mitigate Shortcuts in Underspecified Visual Tasks [92.32670915472099]
  We propose an ensemble diversification framework exploiting the generation of synthetic counterfactuals using Diffusion Probabilistic Models (DPMs) We show that diffusion-guided diversification can lead models to avert attention from shortcut cues, achieving ensemble diversity performance comparable to previous methods requiring additional data collection.
  arXiv  Detail & Related papers  (2023-10-03T17:37:52Z)
• Prompting Diffusion Representations for Cross-Domain Semantic Segmentation [101.04326113360342]
  diffusion-pretraining achieves extraordinary domain generalization results for semantic segmentation. We introduce a scene prompt and a prompt randomization strategy to help further disentangle the domain-invariant information when training the segmentation head.
  arXiv  Detail & Related papers  (2023-07-05T09:28:25Z)
• Conditional Diffusion Models for Weakly Supervised Medical Image Segmentation [18.956306942099097]
  Conditional diffusion models (CDM) is capable of generating images subject to specific distributions. We utilize category-aware semantic information underlied in CDM to get the prediction mask of the target object. Our method outperforms state-of-the-art CAM and diffusion model methods on two public medical image segmentation datasets.
  arXiv  Detail & Related papers  (2023-06-06T17:29:26Z)
• Denoising Diffusion Semantic Segmentation with Mask Prior Modeling [61.73352242029671]
  We propose to ameliorate the semantic segmentation quality of existing discriminative approaches with a mask prior modeled by a denoising diffusion generative model. We evaluate the proposed prior modeling with several off-the-shelf segmentors, and our experimental results on ADE20K and Cityscapes demonstrate that our approach could achieve competitively quantitative performance.
  arXiv  Detail & Related papers  (2023-06-02T17:47:01Z)
• Denoising Diffusion Probabilistic Models for Generation of Realistic Fully-Annotated Microscopy Image Data Sets [1.07539359851877]
  In this study, we demonstrate that diffusion models can effectively generate fully-annotated microscopy image data sets. The proposed pipeline helps to reduce the reliance on manual annotations when training deep learning-based segmentation approaches.
  arXiv  Detail & Related papers  (2023-01-02T14:17:08Z)
• Semantic Image Synthesis via Diffusion Models [159.4285444680301]
  Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in various image generation tasks. Recent work on semantic image synthesis mainly follows the emphde facto Generative Adversarial Nets (GANs)
  arXiv  Detail & Related papers  (2022-06-30T18:31:51Z)
• Diffusion Models for Implicit Image Segmentation Ensembles [1.444701913511243]
  We present a novel semantic segmentation method based on diffusion models. By modifying the training and sampling scheme, we show that diffusion models can perform lesion segmentation of medical images. Compared to state-of-the-art segmentation models, our approach yields good segmentation results and, additionally, meaningful uncertainty maps.
  arXiv  Detail & Related papers  (2021-12-06T16:28:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.