LOCATEdit: Graph Laplacian Optimized Cross Attention for Localized Text-Guided Image Editing

• URL: http://arxiv.org/abs/2503.21541v2
• Date: Fri, 28 Mar 2025 12:17:07 GMT
• Title: LOCATEdit: Graph Laplacian Optimized Cross Attention for Localized Text-Guided Image Editing
• Authors: Achint Soni, Meet Soni, Sirisha Rambhatla,
• Abstract summary: Text-guided image editing aims to modify specific regions of an image according to natural language instructions.<n>Since cross-attention mechanisms focus on semantic relevance, they struggle to maintain the image integrity.<n>We introduce LOCATEdit, which enhances cross-attention maps through a graph-based approach.
• Score: 6.057289837472806
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-guided image editing aims to modify specific regions of an image according to natural language instructions while maintaining the general structure and the background fidelity. Existing methods utilize masks derived from cross-attention maps generated from diffusion models to identify the target regions for modification. However, since cross-attention mechanisms focus on semantic relevance, they struggle to maintain the image integrity. As a result, these methods often lack spatial consistency, leading to editing artifacts and distortions. In this work, we address these limitations and introduce LOCATEdit, which enhances cross-attention maps through a graph-based approach utilizing self-attention-derived patch relationships to maintain smooth, coherent attention across image regions, ensuring that alterations are limited to the designated items while retaining the surrounding structure. LOCATEdit consistently and substantially outperforms existing baselines on PIE-Bench, demonstrating its state-of-the-art performance and effectiveness on various editing tasks. Code can be found on https://github.com/LOCATEdit/LOCATEdit/

Related papers

• EditScout: Locating Forged Regions from Diffusion-based Edited Images with Multimodal LLM [50.054404519821745]
  We present a novel framework that integrates a multimodal Large Language Model for enhanced reasoning capabilities.<n>Our framework achieves promising results on MagicBrush, AutoSplice, and PerfBrush datasets.<n> Notably, our method excels on the PerfBrush dataset, a self-constructed test set featuring previously unseen types of edits.
  arXiv  Detail & Related papers  (2024-12-05T02:05:33Z)
• Uniform Attention Maps: Boosting Image Fidelity in Reconstruction and Editing [66.48853049746123]
  We analyze reconstruction from a structural perspective and propose a novel approach that replaces traditional cross-attention with uniform attention maps. Our method effectively minimizes distortions caused by varying text conditions during noise prediction. Experimental results demonstrate that our approach not only excels in achieving high-fidelity image reconstruction but also performs robustly in real image composition and editing scenarios.
  arXiv  Detail & Related papers  (2024-11-29T12:11:28Z)
• DiffUHaul: A Training-Free Method for Object Dragging in Images [78.93531472479202]
  We propose a training-free method, dubbed DiffUHaul, for the object dragging task. We first apply attention masking in each denoising step to make the generation more disentangled across different objects. In the early denoising steps, we interpolate the attention features between source and target images to smoothly fuse new layouts with the original appearance.
  arXiv  Detail & Related papers  (2024-06-03T17:59:53Z)
• Enhancing Text-to-Image Editing via Hybrid Mask-Informed Fusion [61.42732844499658]
  This paper systematically improves the text-guided image editing techniques based on diffusion models. We incorporate human annotation as an external knowledge to confine editing within a Mask-informed'' region.
  arXiv  Detail & Related papers  (2024-05-24T07:53:59Z)
• LocInv: Localization-aware Inversion for Text-Guided Image Editing [17.611103794346857]
  Text-guided image editing research aims to empower users to manipulate generated images by altering the text prompts. Existing image editing techniques are prone to editing over unintentional regions that are beyond the intended target area. We propose localization-aware Inversion (LocInv), which exploits segmentation maps or bounding boxes as extra localization priors to refine the cross-attention maps.
  arXiv  Detail & Related papers  (2024-05-02T17:27:04Z)
• Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing [47.71851180196975]
  tuning-free Text-guided Image Editing (TIE) is of greater importance for application developers. We conduct an in-depth probing analysis and demonstrate that cross-attention maps in Stable Diffusion often contain object attribution information. In contrast, self-attention maps play a crucial role in preserving the geometric and shape details of the source image.
  arXiv  Detail & Related papers  (2024-03-06T03:32:56Z)
• LIME: Localized Image Editing via Attention Regularization in Diffusion Models [69.33072075580483]
  This paper introduces LIME for localized image editing in diffusion models.<n>LIME does not require user-specified regions of interest (RoI) or additional text input, but rather employs features from pre-trained methods and a straightforward clustering method to obtain precise editing mask.<n>We propose a novel cross-attention regularization technique that penalizes unrelated cross-attention scores in the RoI during the denoising steps, ensuring localized edits.
  arXiv  Detail & Related papers  (2023-12-14T18:59:59Z)
• Accelerating Text-to-Image Editing via Cache-Enabled Sparse Diffusion Inference [36.73121523987844]
  We introduce Fast Image Semantically Edit (FISEdit), a cached-enabled sparse diffusion model inference engine for efficient text-to-image editing. FISEdit uses semantic mapping between the minor modifications on the input text and the affected regions on the output image. For each text editing step, FISEdit can automatically identify the affected image regions and utilize the cached unchanged regions' feature map to accelerate the inference process.
  arXiv  Detail & Related papers  (2023-05-27T09:14:03Z)
• DiffEdit: Diffusion-based semantic image editing with mask guidance [64.555930158319]
  DiffEdit is a method to take advantage of text-conditioned diffusion models for the task of semantic image editing. Our main contribution is able to automatically generate a mask highlighting regions of the input image that need to be edited.
  arXiv  Detail & Related papers  (2022-10-20T17:16:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.