Unsupervised Region-Based Image Editing of Denoising Diffusion Models

• URL: http://arxiv.org/abs/2412.12912v1
• Date: Tue, 17 Dec 2024 13:46:12 GMT
• Title: Unsupervised Region-Based Image Editing of Denoising Diffusion Models
• Authors: Zixiang Li, Yue Song, Renshuai Tao, Xiaohong Jia, Yao Zhao, Wei Wang,
• Abstract summary: We propose a method to identify semantic attributes in the latent space of pre-trained diffusion models without any further training. Our approach facilitates precise semantic discovery and control over local masked areas, eliminating the need for annotations.
• Score: 50.005612464340246
• License:
• Abstract: Although diffusion models have achieved remarkable success in the field of image generation, their latent space remains under-explored. Current methods for identifying semantics within latent space often rely on external supervision, such as textual information and segmentation masks. In this paper, we propose a method to identify semantic attributes in the latent space of pre-trained diffusion models without any further training. By projecting the Jacobian of the targeted semantic region into a low-dimensional subspace which is orthogonal to the non-masked regions, our approach facilitates precise semantic discovery and control over local masked areas, eliminating the need for annotations. We conducted extensive experiments across multiple datasets and various architectures of diffusion models, achieving state-of-the-art performance. In particular, for some specific face attributes, the performance of our proposed method even surpasses that of supervised approaches, demonstrating its superior ability in editing local image properties.

Related papers

• Exploring the latent space of diffusion models directly through singular value decomposition [31.900933527692846]
  We propose a novel image editing framework that is capable of learning arbitrary attributes from one pair of latent codes destined by text prompts in Diffusion Models. We will release our codes soon to foster further research and applications in this area.
  arXiv  Detail & Related papers  (2025-02-04T11:04:36Z)
• Decoding Diffusion: A Scalable Framework for Unsupervised Analysis of Latent Space Biases and Representations Using Natural Language Prompts [68.48103545146127]
  This paper proposes a novel framework for unsupervised exploration of diffusion latent spaces. We directly leverage natural language prompts and image captions to map latent directions. Our method provides a more scalable and interpretable understanding of the semantic knowledge encoded within diffusion models.
  arXiv  Detail & Related papers  (2024-10-25T21:44:51Z)
• Enabling Local Editing in Diffusion Models by Joint and Individual Component Analysis [18.755311950243737]
  The latent space of Diffusion Models (DMs) is not as well understood as that of Generative Adversarial Networks (GANs) Recent research has focused on unsupervised semantic discovery in the latent space of DMs. We introduce an unsupervised method to factorize the latent semantics learned by the denoising network of pre-trained DMs.
  arXiv  Detail & Related papers  (2024-08-29T18:21:50Z)
• Diffusion Features to Bridge Domain Gap for Semantic Segmentation [2.8616666231199424]
  This paper investigates the approach that leverages the sampling and fusion techniques to harness the features of diffusion models efficiently. By leveraging the strength of text-to-image generation capability, we introduce a new training framework designed to implicitly learn posterior knowledge from it.
  arXiv  Detail & Related papers  (2024-06-02T15:33:46Z)
• EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models [52.3015009878545]
  We develop an image segmentor capable of generating fine-grained segmentation maps without any additional training. Our framework identifies semantic correspondences between image pixels and spatial locations of low-dimensional feature maps. In extensive experiments, the produced segmentation maps are demonstrated to be well delineated and capture detailed parts of the images.
  arXiv  Detail & Related papers  (2024-01-22T07:34:06Z)
• NoiseCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions in Diffusion Models [6.254873489691852]
  We propose an unsupervised method to discover latent semantics in text-to-image diffusion models without relying on text prompts. Our method achieves highly disentangled edits, outperforming existing approaches in both diffusion-based and GAN-based latent space editing methods.
  arXiv  Detail & Related papers  (2023-12-08T22:04:53Z)
• Denoising Diffusion Semantic Segmentation with Mask Prior Modeling [61.73352242029671]
  We propose to ameliorate the semantic segmentation quality of existing discriminative approaches with a mask prior modeled by a denoising diffusion generative model. We evaluate the proposed prior modeling with several off-the-shelf segmentors, and our experimental results on ADE20K and Cityscapes demonstrate that our approach could achieve competitively quantitative performance.
  arXiv  Detail & Related papers  (2023-06-02T17:47:01Z)
• Region-Based Semantic Factorization in GANs [67.90498535507106]
  We present a highly efficient algorithm to factorize the latent semantics learned by Generative Adversarial Networks (GANs) concerning an arbitrary image region. Through an appropriately defined generalized Rayleigh quotient, we solve such a problem without any annotations or training. Experimental results on various state-of-the-art GAN models demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2022-02-19T17:46:02Z)
• Self-supervised Segmentation via Background Inpainting [96.10971980098196]
  We introduce a self-supervised detection and segmentation approach that can work with single images captured by a potentially moving camera. We exploit a self-supervised loss function that we exploit to train a proposal-based segmentation network. We apply our method to human detection and segmentation in images that visually depart from those of standard benchmarks and outperform existing self-supervised methods.
  arXiv  Detail & Related papers  (2020-11-11T08:34:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.