EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models

• URL: http://arxiv.org/abs/2401.11739v1
• Date: Mon, 22 Jan 2024 07:34:06 GMT
• Title: EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models
• Authors: Koichi Namekata, Amirmojtaba Sabour, Sanja Fidler, Seung Wook Kim
• Abstract summary: We develop an image segmentor capable of generating fine-grained segmentation maps without any additional training. Our framework identifies semantic correspondences between image pixels and spatial locations of low-dimensional feature maps. In extensive experiments, the produced segmentation maps are demonstrated to be well delineated and capture detailed parts of the images.
• Score: 52.3015009878545
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion models have recently received increasing research attention for their remarkable transfer abilities in semantic segmentation tasks. However, generating fine-grained segmentation masks with diffusion models often requires additional training on annotated datasets, leaving it unclear to what extent pre-trained diffusion models alone understand the semantic relations of their generated images. To address this question, we leverage the semantic knowledge extracted from Stable Diffusion (SD) and aim to develop an image segmentor capable of generating fine-grained segmentation maps without any additional training. The primary difficulty stems from the fact that semantically meaningful feature maps typically exist only in the spatially lower-dimensional layers, which poses a challenge in directly extracting pixel-level semantic relations from these feature maps. To overcome this issue, our framework identifies semantic correspondences between image pixels and spatial locations of low-dimensional feature maps by exploiting SD's generation process and utilizes them for constructing image-resolution segmentation maps. In extensive experiments, the produced segmentation maps are demonstrated to be well delineated and capture detailed parts of the images, indicating the existence of highly accurate pixel-level semantic knowledge in diffusion models.

Related papers

• Decoding Diffusion: A Scalable Framework for Unsupervised Analysis of Latent Space Biases and Representations Using Natural Language Prompts [68.48103545146127]
  This paper proposes a novel framework for unsupervised exploration of diffusion latent spaces. We directly leverage natural language prompts and image captions to map latent directions. Our method provides a more scalable and interpretable understanding of the semantic knowledge encoded within diffusion models.
  arXiv  Detail & Related papers  (2024-10-25T21:44:51Z)
• Enhancing Label-efficient Medical Image Segmentation with Text-guided Diffusion Models [5.865983529245793]
  TextDiff improves semantic representation through inexpensive medical text annotations. We show that TextDiff is significantly superior to the state-of-the-art multi-modal segmentation methods with only a few training samples.
  arXiv  Detail & Related papers  (2024-07-07T10:21:08Z)
• SatSynth: Augmenting Image-Mask Pairs through Diffusion Models for Aerial Semantic Segmentation [69.42764583465508]
  We explore the potential of generative image diffusion to address the scarcity of annotated data in earth observation tasks. To the best of our knowledge, we are the first to generate both images and corresponding masks for satellite segmentation.
  arXiv  Detail & Related papers  (2024-03-25T10:30:22Z)
• Learned representation-guided diffusion models for large-image generation [58.192263311786824]
  We introduce a novel approach that trains diffusion models conditioned on embeddings from self-supervised learning (SSL) Our diffusion models successfully project these features back to high-quality histopathology and remote sensing images. Augmenting real data by generating variations of real images improves downstream accuracy for patch-level and larger, image-scale classification tasks.
  arXiv  Detail & Related papers  (2023-12-12T14:45:45Z)
• Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmenter [47.29967666846132]
  generative text-to-image diffusion models are highly efficient open-vocabulary semantic segmenters. We introduce a novel training-free approach named DiffSegmenter to generate realistic objects that are semantically faithful to the input text. Extensive experiments on three benchmark datasets show that the proposed DiffSegmenter achieves impressive results for open-vocabulary semantic segmentation.
  arXiv  Detail & Related papers  (2023-09-06T06:31:08Z)
• Zero-shot spatial layout conditioning for text-to-image diffusion models [52.24744018240424]
  Large-scale text-to-image diffusion models have significantly improved the state of the art in generative image modelling. We consider image generation from text associated with segments on the image canvas, which combines an intuitive natural language interface with precise spatial control over the generated content. We propose ZestGuide, a zero-shot segmentation guidance approach that can be plugged into pre-trained text-to-image diffusion models.
  arXiv  Detail & Related papers  (2023-06-23T19:24:48Z)
• Semantic Segmentation by Semantic Proportions [6.171990546748665]
  We propose a novel approach for semantic segmentation, requiring the rough information of individual semantic class proportions. This greatly simplifies the data annotation process and thus will significantly reduce the annotation time, cost and storage space.
  arXiv  Detail & Related papers  (2023-05-24T22:51:52Z)
• Unsupervised Semantic Correspondence Using Stable Diffusion [27.355330079806027]
  We show that one can leverage this semantic knowledge within diffusion models to find semantic correspondences. We optimize the prompt embeddings of these models for maximum attention on the regions of interest. We significantly outperform any existing weakly or unsupervised method on PF-Willow, CUB-200 and SPair-71k datasets.
  arXiv  Detail & Related papers  (2023-05-24T21:34:34Z)
• Stochastic Segmentation with Conditional Categorical Diffusion Models [3.8168879948759953]
  We propose a conditional categorical diffusion model (CCDM) for semantic segmentation based on Denoising Diffusion Probabilistic Models. Our results show that CCDM achieves state-of-the-art performance on LIDC, and outperforms established baselines on the classical segmentation dataset Cityscapes.
  arXiv  Detail & Related papers  (2023-03-15T19:16:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.