Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing

• URL: http://arxiv.org/abs/2403.03431v1
• Date: Wed, 6 Mar 2024 03:32:56 GMT
• Title: Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing
• Authors: Bingyan Liu, Chengyu Wang, Tingfeng Cao, Kui Jia, Jun Huang
• Abstract summary: tuning-free Text-guided Image Editing (TIE) is of greater importance for application developers. We conduct an in-depth probing analysis and demonstrate that cross-attention maps in Stable Diffusion often contain object attribution information. In contrast, self-attention maps play a crucial role in preserving the geometric and shape details of the source image.
• Score: 47.71851180196975
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep Text-to-Image Synthesis (TIS) models such as Stable Diffusion have recently gained significant popularity for creative Text-to-image generation. Yet, for domain-specific scenarios, tuning-free Text-guided Image Editing (TIE) is of greater importance for application developers, which modify objects or object properties in images by manipulating feature components in attention layers during the generation process. However, little is known about what semantic meanings these attention layers have learned and which parts of the attention maps contribute to the success of image editing. In this paper, we conduct an in-depth probing analysis and demonstrate that cross-attention maps in Stable Diffusion often contain object attribution information that can result in editing failures. In contrast, self-attention maps play a crucial role in preserving the geometric and shape details of the source image during the transformation to the target image. Our analysis offers valuable insights into understanding cross and self-attention maps in diffusion models. Moreover, based on our findings, we simplify popular image editing methods and propose a more straightforward yet more stable and efficient tuning-free procedure that only modifies self-attention maps of the specified attention layers during the denoising process. Experimental results show that our simplified method consistently surpasses the performance of popular approaches on multiple datasets.

Related papers

• DiffUHaul: A Training-Free Method for Object Dragging in Images [78.93531472479202]
  We propose a training-free method, dubbed DiffUHaul, for the object dragging task. We first apply attention masking in each denoising step to make the generation more disentangled across different objects. In the early denoising steps, we interpolate the attention features between source and target images to smoothly fuse new layouts with the original appearance.
  arXiv  Detail & Related papers  (2024-06-03T17:59:53Z)
• LIME: Localized Image Editing via Attention Regularization in Diffusion Models [74.3811832586391]
  This paper introduces LIME for localized image editing in diffusion models that do not require user-specified regions of interest (RoI) or additional text input. Our method employs features from pre-trained methods and a simple clustering technique to obtain precise semantic segmentation maps. We propose a novel cross-attention regularization technique that penalizes unrelated cross-attention scores in the RoI during the denoising steps, ensuring localized edits.
  arXiv  Detail & Related papers  (2023-12-14T18:59:59Z)
• Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editing [23.00202969969574]
  We propose Dynamic Prompt Learning (DPL) to force cross-attention maps to focus on correct noun words in the text prompt. We show improved prompt editing results for Word-Swap, Prompt Refinement, and Attention Re-weighting, especially for complex multi-object scenes.
  arXiv  Detail & Related papers  (2023-09-27T13:55:57Z)
• PFB-Diff: Progressive Feature Blending Diffusion for Text-driven Image Editing [8.19063619210761]
  PFB-Diff is a Progressive Feature Blending method for Diffusion-based image editing. Our method demonstrates its superior performance in terms of image fidelity, editing accuracy, efficiency, and faithfulness to the original image.
  arXiv  Detail & Related papers  (2023-06-28T11:10:20Z)
• DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing [94.24479528298252]
  DragGAN is an interactive point-based image editing framework that achieves impressive editing results with pixel-level precision. By harnessing large-scale pretrained diffusion models, we greatly enhance the applicability of interactive point-based editing on both real and diffusion-generated images. We present a challenging benchmark dataset called DragBench to evaluate the performance of interactive point-based image editing methods.
  arXiv  Detail & Related papers  (2023-06-26T06:04:09Z)
• LayerDiffusion: Layered Controlled Image Editing with Diffusion Models [5.58892860792971]
  LayerDiffusion is a semantic-based layered controlled image editing method. We leverage a large-scale text-to-image model and employ a layered controlled optimization strategy. Experimental results demonstrate the effectiveness of our method in generating highly coherent images.
  arXiv  Detail & Related papers  (2023-05-30T01:26:41Z)
• Localizing Object-level Shape Variations with Text-to-Image Diffusion Models [60.422435066544814]
  We present a technique to generate a collection of images that depicts variations in the shape of a specific object. A particular challenge when generating object variations is accurately localizing the manipulation applied over the object's shape. To localize the image-space operation, we present two techniques that use the self-attention layers in conjunction with the cross-attention layers.
  arXiv  Detail & Related papers  (2023-03-20T17:45:08Z)
• DiffEdit: Diffusion-based semantic image editing with mask guidance [64.555930158319]
  DiffEdit is a method to take advantage of text-conditioned diffusion models for the task of semantic image editing. Our main contribution is able to automatically generate a mask highlighting regions of the input image that need to be edited.
  arXiv  Detail & Related papers  (2022-10-20T17:16:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.