Label-Efficient Semantic Segmentation with Diffusion Models
- URL: http://arxiv.org/abs/2112.03126v1
- Date: Mon, 6 Dec 2021 15:55:30 GMT
- Title: Label-Efficient Semantic Segmentation with Diffusion Models
- Authors: Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov,
Artem Babenko
- Abstract summary: We demonstrate that diffusion models can also serve as an instrument for semantic segmentation.
In particular, for several pretrained diffusion models, we investigate the intermediate activations from the networks that perform the Markov step of the reverse diffusion process.
We show that these activations effectively capture the semantic information from an input image and appear to be excellent pixel-level representations for the segmentation problem.
- Score: 27.01899943738203
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Denoising diffusion probabilistic models have recently received much research
attention since they outperform alternative approaches, such as GANs, and
currently provide state-of-the-art generative performance. The superior
performance of diffusion models has made them an appealing tool in several
applications, including inpainting, super-resolution, and semantic editing. In
this paper, we demonstrate that diffusion models can also serve as an
instrument for semantic segmentation, especially in the setup when labeled data
is scarce. In particular, for several pretrained diffusion models, we
investigate the intermediate activations from the networks that perform the
Markov step of the reverse diffusion process. We show that these activations
effectively capture the semantic information from an input image and appear to
be excellent pixel-level representations for the segmentation problem. Based on
these observations, we describe a simple segmentation method, which can work
even if only a few training images are provided. Our approach significantly
outperforms the existing alternatives on several datasets for the same amount
of human supervision.
Related papers
- Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation [56.87049651707208]
Few-shot Semantic has evolved into In-context tasks, morphing into a crucial element in assessing generalist segmentation models.
Our initial focus lies in understanding how to facilitate interaction between the query image and the support image, resulting in the proposal of a KV fusion method within the self-attention framework.
Based on our analysis, we establish a simple and effective framework named DiffewS, maximally retaining the original Latent Diffusion Model's generative framework.
arXiv Detail & Related papers (2024-10-03T10:33:49Z) - Enhancing Label-efficient Medical Image Segmentation with Text-guided Diffusion Models [5.865983529245793]
TextDiff improves semantic representation through inexpensive medical text annotations.
We show that TextDiff is significantly superior to the state-of-the-art multi-modal segmentation methods with only a few training samples.
arXiv Detail & Related papers (2024-07-07T10:21:08Z) - Diffusion Features to Bridge Domain Gap for Semantic Segmentation [2.8616666231199424]
This paper investigates the approach that leverages the sampling and fusion techniques to harness the features of diffusion models efficiently.
By leveraging the strength of text-to-image generation capability, we introduce a new training framework designed to implicitly learn posterior knowledge from it.
arXiv Detail & Related papers (2024-06-02T15:33:46Z) - SatSynth: Augmenting Image-Mask Pairs through Diffusion Models for Aerial Semantic Segmentation [69.42764583465508]
We explore the potential of generative image diffusion to address the scarcity of annotated data in earth observation tasks.
To the best of our knowledge, we are the first to generate both images and corresponding masks for satellite segmentation.
arXiv Detail & Related papers (2024-03-25T10:30:22Z) - Few-shot Multispectral Segmentation with Representations Generated by
Reinforcement Learning [0.0]
We propose a novel approach for improving few-shot segmentation performance on multispectral images using reinforcement learning.
Our methodology involves training an agent to identify the most informative expressions, updating the dataset using these expressions, and then using the updated dataset to perform segmentation.
We evaluate the effectiveness of our approach on several multispectral datasets and demonstrate its effectiveness in boosting the performance of segmentation algorithms.
arXiv Detail & Related papers (2023-11-20T15:04:16Z) - Leveraging Diffusion Disentangled Representations to Mitigate Shortcuts
in Underspecified Visual Tasks [92.32670915472099]
We propose an ensemble diversification framework exploiting the generation of synthetic counterfactuals using Diffusion Probabilistic Models (DPMs)
We show that diffusion-guided diversification can lead models to avert attention from shortcut cues, achieving ensemble diversity performance comparable to previous methods requiring additional data collection.
arXiv Detail & Related papers (2023-10-03T17:37:52Z) - Prompting Diffusion Representations for Cross-Domain Semantic
Segmentation [101.04326113360342]
diffusion-pretraining achieves extraordinary domain generalization results for semantic segmentation.
We introduce a scene prompt and a prompt randomization strategy to help further disentangle the domain-invariant information when training the segmentation head.
arXiv Detail & Related papers (2023-07-05T09:28:25Z) - Conditional Diffusion Models for Weakly Supervised Medical Image
Segmentation [18.956306942099097]
Conditional diffusion models (CDM) is capable of generating images subject to specific distributions.
We utilize category-aware semantic information underlied in CDM to get the prediction mask of the target object.
Our method outperforms state-of-the-art CAM and diffusion model methods on two public medical image segmentation datasets.
arXiv Detail & Related papers (2023-06-06T17:29:26Z) - Denoising Diffusion Semantic Segmentation with Mask Prior Modeling [61.73352242029671]
We propose to ameliorate the semantic segmentation quality of existing discriminative approaches with a mask prior modeled by a denoising diffusion generative model.
We evaluate the proposed prior modeling with several off-the-shelf segmentors, and our experimental results on ADE20K and Cityscapes demonstrate that our approach could achieve competitively quantitative performance.
arXiv Detail & Related papers (2023-06-02T17:47:01Z) - Semantic Image Synthesis via Diffusion Models [159.4285444680301]
Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in various image generation tasks.
Recent work on semantic image synthesis mainly follows the emphde facto Generative Adversarial Nets (GANs)
arXiv Detail & Related papers (2022-06-30T18:31:51Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.