TurboEdit: Instant text-based image editing

• URL: http://arxiv.org/abs/2408.08332v1
• Date: Wed, 14 Aug 2024 18:02:24 GMT
• Title: TurboEdit: Instant text-based image editing
• Authors: Zongze Wu, Nicholas Kolkin, Jonathan Brandt, Richard Zhang, Eli Shechtman,
• Abstract summary: We address the challenges of precise image inversion and disentangled image editing in the context of few-step diffusion models. We introduce an encoder based iterative inversion technique. The inversion network is conditioned on the input image and the reconstructed image from the previous step, allowing for correction of the next reconstruction towards the input image. Our approach facilitates realistic text-guided image edits in real-time, requiring only 8 number of functional evaluations (NFEs) in inversion and 4 NFEs per edit.
• Score: 32.06820085957286
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We address the challenges of precise image inversion and disentangled image editing in the context of few-step diffusion models. We introduce an encoder based iterative inversion technique. The inversion network is conditioned on the input image and the reconstructed image from the previous step, allowing for correction of the next reconstruction towards the input image. We demonstrate that disentangled controls can be easily achieved in the few-step diffusion model by conditioning on an (automatically generated) detailed text prompt. To manipulate the inverted image, we freeze the noise maps and modify one attribute in the text prompt (either manually or via instruction based editing driven by an LLM), resulting in the generation of a new image similar to the input image with only one attribute changed. It can further control the editing strength and accept instructive text prompt. Our approach facilitates realistic text-guided image edits in real-time, requiring only 8 number of functional evaluations (NFEs) in inversion (one-time cost) and 4 NFEs per edit. Our method is not only fast, but also significantly outperforms state-of-the-art multi-step diffusion editing techniques.

Related papers

• TurboEdit: Text-Based Image Editing Using Few-Step Diffusion Models [53.757752110493215]
  We focus on a popular line of text-based editing frameworks - the edit-friendly'' DDPM-noise inversion approach. We analyze its application to fast sampling methods and categorize its failures into two classes: the appearance of visual artifacts, and insufficient editing strength. We propose a pseudo-guidance approach that efficiently increases the magnitude of edits without introducing new artifacts.
  arXiv  Detail & Related papers  (2024-08-01T17:27:28Z)
• Enhancing Text-to-Image Editing via Hybrid Mask-Informed Fusion [61.42732844499658]
  This paper systematically improves the text-guided image editing techniques based on diffusion models. We incorporate human annotation as an external knowledge to confine editing within a Mask-informed'' region.
  arXiv  Detail & Related papers  (2024-05-24T07:53:59Z)
• Editable Image Elements for Controllable Synthesis [79.58148778509769]
  We propose an image representation that promotes spatial editing of input images using a diffusion model. We show the effectiveness of our representation on various image editing tasks, such as object resizing, rearrangement, dragging, de-occlusion, removal, variation, and image composition.
  arXiv  Detail & Related papers  (2024-04-24T17:59:11Z)
• Source Prompt Disentangled Inversion for Boosting Image Editability with Diffusion Models [18.75409092764653]
  One crucial step in text-driven image editing is to invert the original image into a latent noise code conditioned on the source prompt. We propose a novel method called Source Prompt Disentangled Inversion (SPDInv), which aims at reducing the impact of source prompt. The experimental results show that our proposed SPDInv method can effectively mitigate the conflicts between the target editing prompt and the source prompt.
  arXiv  Detail & Related papers  (2024-03-17T06:19:30Z)
• Forgedit: Text Guided Image Editing via Learning and Forgetting [17.26772361532044]
  We design a novel text-guided image editing method, named as Forgedit. First, we propose a vision-language joint optimization framework capable of reconstructing the original image in 30 seconds. Then, we propose a novel vector projection mechanism in text embedding space of Diffusion Models.
  arXiv  Detail & Related papers  (2023-09-19T12:05:26Z)
• Prompt Tuning Inversion for Text-Driven Image Editing Using Diffusion Models [6.34777393532937]
  We propose an accurate and quick inversion technique, Prompt Tuning Inversion, for text-driven image editing. Our proposed editing method consists of a reconstruction stage and an editing stage. Experiments on ImageNet demonstrate the superior editing performance of our method compared to the state-of-the-art baselines.
  arXiv  Detail & Related papers  (2023-05-08T03:34:33Z)
• FateZero: Fusing Attentions for Zero-shot Text-based Video Editing [104.27329655124299]
  We propose FateZero, a zero-shot text-based editing method on real-world videos without per-prompt training or use-specific mask. Our method is the first one to show the ability of zero-shot text-driven video style and local attribute editing from the trained text-to-image model.
  arXiv  Detail & Related papers  (2023-03-16T17:51:13Z)
• Zero-shot Image-to-Image Translation [57.46189236379433]
  We propose pix2pix-zero, an image-to-image translation method that can preserve the original image without manual prompting. We propose cross-attention guidance, which aims to retain the cross-attention maps of the input image throughout the diffusion process. Our method does not need additional training for these edits and can directly use the existing text-to-image diffusion model.
  arXiv  Detail & Related papers  (2023-02-06T18:59:51Z)
• Null-text Inversion for Editing Real Images using Guided Diffusion Models [44.27570654402436]
  We introduce an accurate inversion technique and thus facilitate an intuitive text-based modification of the image. Our Null-text inversion, based on the publicly available Stable Diffusion model, is extensively evaluated on a variety of images and prompt editing.
  arXiv  Detail & Related papers  (2022-11-17T18:58:14Z)
• DiffEdit: Diffusion-based semantic image editing with mask guidance [64.555930158319]
  DiffEdit is a method to take advantage of text-conditioned diffusion models for the task of semantic image editing. Our main contribution is able to automatically generate a mask highlighting regions of the input image that need to be edited.
  arXiv  Detail & Related papers  (2022-10-20T17:16:37Z)
• UniTune: Text-Driven Image Editing by Fine Tuning a Diffusion Model on a Single Image [2.999198565272416]
  We make the observation that image-generation models can be converted to image-editing models simply by fine-tuning them on a single image. We propose UniTune, a novel image editing method. UniTune gets as input an arbitrary image and a textual edit description, and carries out the edit while maintaining high fidelity to the input image. We demonstrate that it is broadly applicable and can perform a surprisingly wide range of expressive editing operations, including those requiring significant visual changes that were previously impossible.
  arXiv  Detail & Related papers  (2022-10-17T23:46:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.