Source Prompt Disentangled Inversion for Boosting Image Editability with Diffusion Models

• URL: http://arxiv.org/abs/2403.11105v2
• Date: Fri, 5 Jul 2024 11:16:54 GMT
• Title: Source Prompt Disentangled Inversion for Boosting Image Editability with Diffusion Models
• Authors: Ruibin Li, Ruihuang Li, Song Guo, Lei Zhang,
• Abstract summary: One crucial step in text-driven image editing is to invert the original image into a latent noise code conditioned on the source prompt. We propose a novel method called Source Prompt Disentangled Inversion (SPDInv), which aims at reducing the impact of source prompt. The experimental results show that our proposed SPDInv method can effectively mitigate the conflicts between the target editing prompt and the source prompt.
• Score: 18.75409092764653
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-driven diffusion models have significantly advanced the image editing performance by using text prompts as inputs. One crucial step in text-driven image editing is to invert the original image into a latent noise code conditioned on the source prompt. While previous methods have achieved promising results by refactoring the image synthesizing process, the inverted latent noise code is tightly coupled with the source prompt, limiting the image editability by target text prompts. To address this issue, we propose a novel method called Source Prompt Disentangled Inversion (SPDInv), which aims at reducing the impact of source prompt, thereby enhancing the text-driven image editing performance by employing diffusion models. To make the inverted noise code be independent of the given source prompt as much as possible, we indicate that the iterative inversion process should satisfy a fixed-point constraint. Consequently, we transform the inversion problem into a searching problem to find the fixed-point solution, and utilize the pre-trained diffusion models to facilitate the searching process. The experimental results show that our proposed SPDInv method can effectively mitigate the conflicts between the target editing prompt and the source prompt, leading to a significant decrease in editing artifacts. In addition to text-driven image editing, with SPDInv we can easily adapt customized image generation models to localized editing tasks and produce promising performance. The source code are available at https://github.com/leeruibin/SPDInv.

Related papers

• TurboEdit: Instant text-based image editing [32.06820085957286]
  We address the challenges of precise image inversion and disentangled image editing in the context of few-step diffusion models. We introduce an encoder based iterative inversion technique. The inversion network is conditioned on the input image and the reconstructed image from the previous step, allowing for correction of the next reconstruction towards the input image. Our approach facilitates realistic text-guided image edits in real-time, requiring only 8 number of functional evaluations (NFEs) in inversion and 4 NFEs per edit.
  arXiv  Detail & Related papers  (2024-08-14T18:02:24Z)
• TurboEdit: Text-Based Image Editing Using Few-Step Diffusion Models [53.757752110493215]
  We focus on a popular line of text-based editing frameworks - the edit-friendly'' DDPM-noise inversion approach. We analyze its application to fast sampling methods and categorize its failures into two classes: the appearance of visual artifacts, and insufficient editing strength. We propose a pseudo-guidance approach that efficiently increases the magnitude of edits without introducing new artifacts.
  arXiv  Detail & Related papers  (2024-08-01T17:27:28Z)
• Enhancing Text-to-Image Editing via Hybrid Mask-Informed Fusion [61.42732844499658]
  This paper systematically improves the text-guided image editing techniques based on diffusion models. We incorporate human annotation as an external knowledge to confine editing within a Mask-informed'' region.
  arXiv  Detail & Related papers  (2024-05-24T07:53:59Z)
• Editable Image Elements for Controllable Synthesis [79.58148778509769]
  We propose an image representation that promotes spatial editing of input images using a diffusion model. We show the effectiveness of our representation on various image editing tasks, such as object resizing, rearrangement, dragging, de-occlusion, removal, variation, and image composition.
  arXiv  Detail & Related papers  (2024-04-24T17:59:11Z)
• DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editing [66.43179841884098]
  Large-scale Text-to-Image (T2I) diffusion models have revolutionized image generation over the last few years. We propose DiffEditor to rectify two weaknesses in existing diffusion-based image editing. Our method can efficiently achieve state-of-the-art performance on various fine-grained image editing tasks.
  arXiv  Detail & Related papers  (2024-02-04T18:50:29Z)
• InFusion: Inject and Attention Fusion for Multi Concept Zero-Shot Text-based Video Editing [27.661609140918916]
  InFusion is a framework for zero-shot text-based video editing. It supports editing of multiple concepts with pixel-level control over diverse concepts mentioned in the editing prompt. Our framework is a low-cost alternative to one-shot tuned models for editing since it does not require training.
  arXiv  Detail & Related papers  (2023-07-22T17:05:47Z)
• iEdit: Localised Text-guided Image Editing with Weak Supervision [53.082196061014734]
  We propose a novel learning method for text-guided image editing. It generates images conditioned on a source image and a textual edit prompt. It shows favourable results against its counterparts in terms of image fidelity, CLIP alignment score and qualitatively for editing both generated and real images.
  arXiv  Detail & Related papers  (2023-05-10T07:39:14Z)
• Eliminating Contextual Prior Bias for Semantic Image Editing via Dual-Cycle Diffusion [35.95513392917737]
  A novel approach called Dual-Cycle Diffusion generates an unbiased mask to guide image editing. Our experiments demonstrate the effectiveness of the proposed method, as it significantly improves the D-CLIP score from 0.272 to 0.283.
  arXiv  Detail & Related papers  (2023-02-05T14:30:22Z)
• Null-text Inversion for Editing Real Images using Guided Diffusion Models [44.27570654402436]
  We introduce an accurate inversion technique and thus facilitate an intuitive text-based modification of the image. Our Null-text inversion, based on the publicly available Stable Diffusion model, is extensively evaluated on a variety of images and prompt editing.
  arXiv  Detail & Related papers  (2022-11-17T18:58:14Z)
• DiffEdit: Diffusion-based semantic image editing with mask guidance [64.555930158319]
  DiffEdit is a method to take advantage of text-conditioned diffusion models for the task of semantic image editing. Our main contribution is able to automatically generate a mask highlighting regions of the input image that need to be edited.
  arXiv  Detail & Related papers  (2022-10-20T17:16:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.