Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free Real Image Editing

• URL: http://arxiv.org/abs/2409.01322v3
• Date: Wed, 25 Sep 2024 14:44:21 GMT
• Title: Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free Real Image Editing
• Authors: Vadim Titov, Madina Khalmatova, Alexandra Ivanova, Dmitry Vetrov, Aibek Alanov,
• Abstract summary: We propose a novel approach that is built upon a modified diffusion sampling process via the guidance mechanism. In this work, we explore the self-guidance technique to preserve the overall structure of the input image. We show through human evaluation and quantitative analysis that the proposed method allows to produce desired editing.
• Score: 42.73883397041092
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite recent advances in large-scale text-to-image generative models, manipulating real images with these models remains a challenging problem. The main limitations of existing editing methods are that they either fail to perform with consistent quality on a wide range of image edits or require time-consuming hyperparameter tuning or fine-tuning of the diffusion model to preserve the image-specific appearance of the input image. We propose a novel approach that is built upon a modified diffusion sampling process via the guidance mechanism. In this work, we explore the self-guidance technique to preserve the overall structure of the input image and its local regions appearance that should not be edited. In particular, we explicitly introduce layout-preserving energy functions that are aimed to save local and global structures of the source image. Additionally, we propose a noise rescaling mechanism that allows to preserve noise distribution by balancing the norms of classifier-free guidance and our proposed guiders during generation. Such a guiding approach does not require fine-tuning the diffusion model and exact inversion process. As a result, the proposed method provides a fast and high-quality editing mechanism. In our experiments, we show through human evaluation and quantitative analysis that the proposed method allows to produce desired editing which is more preferable by humans and also achieves a better trade-off between editing quality and preservation of the original image. Our code is available at https://github.com/MACderRu/Guide-and-Rescale.

Related papers

• TurboEdit: Text-Based Image Editing Using Few-Step Diffusion Models [53.757752110493215]
  We focus on a popular line of text-based editing frameworks - the edit-friendly'' DDPM-noise inversion approach. We analyze its application to fast sampling methods and categorize its failures into two classes: the appearance of visual artifacts, and insufficient editing strength. We propose a pseudo-guidance approach that efficiently increases the magnitude of edits without introducing new artifacts.
  arXiv  Detail & Related papers  (2024-08-01T17:27:28Z)
• DiffUHaul: A Training-Free Method for Object Dragging in Images [78.93531472479202]
  We propose a training-free method, dubbed DiffUHaul, for the object dragging task. We first apply attention masking in each denoising step to make the generation more disentangled across different objects. In the early denoising steps, we interpolate the attention features between source and target images to smoothly fuse new layouts with the original appearance.
  arXiv  Detail & Related papers  (2024-06-03T17:59:53Z)
• FreeDiff: Progressive Frequency Truncation for Image Editing with Diffusion Models [44.26371926512843]
  We introduce a novel free approach that employs progressive $textbfFre$qu$textbfe$ncy truncation to refine the guidance of $textbfDiff$usion models for universal editing tasks. Our method achieves comparable results with state-of-the-art methods across a variety of editing tasks and on a diverse set of images.
  arXiv  Detail & Related papers  (2024-04-18T04:47:28Z)
• ReNoise: Real Image Inversion Through Iterative Noising [62.96073631599749]
  We introduce an inversion method with a high quality-to-operation ratio, enhancing reconstruction accuracy without increasing the number of operations. We evaluate the performance of our ReNoise technique using various sampling algorithms and models, including recent accelerated diffusion models.
  arXiv  Detail & Related papers  (2024-03-21T17:52:08Z)
• Eta Inversion: Designing an Optimal Eta Function for Diffusion-based Real Image Editing [2.5602836891933074]
  A commonly adopted strategy for editing real images involves inverting the diffusion process to obtain a noisy representation of the original image. Current methods for diffusion inversion often struggle to produce edits that are both faithful to the specified text prompt and closely resemble the source image. We introduce a novel and adaptable diffusion inversion technique for real image editing, which is grounded in a theoretical analysis of the role of $eta$ in the DDIM sampling equation for enhanced editability.
  arXiv  Detail & Related papers  (2024-03-14T15:07:36Z)
• Diffusion Model-Based Image Editing: A Survey [46.244266782108234]
  Denoising diffusion models have emerged as a powerful tool for various image generation and editing tasks. We provide an exhaustive overview of existing methods using diffusion models for image editing. To further evaluate the performance of text-guided image editing algorithms, we propose a systematic benchmark, EditEval.
  arXiv  Detail & Related papers  (2024-02-27T14:07:09Z)
• High-Fidelity Diffusion-based Image Editing [19.85446433564999]
  The editing performance of diffusion models tends to be no more satisfactory even with increasing denoising steps. We propose an innovative framework where a Markov module is incorporated to modulate diffusion model weights with residual features. We introduce a novel learning paradigm aimed at minimizing error propagation during the editing process, which trains the editing procedure in a manner similar to denoising score-matching.
  arXiv  Detail & Related papers  (2023-12-25T12:12:36Z)
• LIME: Localized Image Editing via Attention Regularization in Diffusion Models [74.3811832586391]
  This paper introduces LIME for localized image editing in diffusion models that do not require user-specified regions of interest (RoI) or additional text input. Our method employs features from pre-trained methods and a simple clustering technique to obtain precise semantic segmentation maps. We propose a novel cross-attention regularization technique that penalizes unrelated cross-attention scores in the RoI during the denoising steps, ensuring localized edits.
  arXiv  Detail & Related papers  (2023-12-14T18:59:59Z)
• ReGeneration Learning of Diffusion Models with Rich Prompts for Zero-Shot Image Translation [8.803251014279502]
  Large-scale text-to-image models have demonstrated amazing ability to synthesize diverse and high-fidelity images. Current models can impose significant changes to the original image content during the editing process. We propose ReGeneration learning in an image-to-image Diffusion model (ReDiffuser)
  arXiv  Detail & Related papers  (2023-05-08T12:08:12Z)
• Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models [60.63556257324894]
  A key desired property of image generative models is the ability to disentangle different attributes. We propose a simple, light-weight image editing algorithm where the mixing weights of the two text embeddings are optimized for style matching and content preservation. Experiments show that the proposed method can modify a wide range of attributes, with the performance outperforming diffusion-model-based image-editing algorithms.
  arXiv  Detail & Related papers  (2022-12-16T19:58:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.