Direct Inversion: Optimization-Free Text-Driven Real Image Editing with Diffusion Models

• URL: http://arxiv.org/abs/2211.07825v1
• Date: Tue, 15 Nov 2022 01:07:38 GMT
• Title: Direct Inversion: Optimization-Free Text-Driven Real Image Editing with Diffusion Models
• Authors: Adham Elarabawy, Harish Kamath, Samuel Denton
• Abstract summary: We propose an optimization-free and zero fine-tuning framework that applies complex and non-rigid edits to a single real image via a text prompt. We prove our method's efficacy in producing high-quality, diverse, semantically coherent, and faithful real image edits.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With the rise of large, publicly-available text-to-image diffusion models, text-guided real image editing has garnered much research attention recently. Existing methods tend to either rely on some form of per-instance or per-task fine-tuning and optimization, require multiple novel views, or they inherently entangle preservation of real image identity, semantic coherence, and faithfulness to text guidance. In this paper, we propose an optimization-free and zero fine-tuning framework that applies complex and non-rigid edits to a single real image via a text prompt, avoiding all the pitfalls described above. Using widely-available generic pre-trained text-to-image diffusion models, we demonstrate the ability to modulate pose, scene, background, style, color, and even racial identity in an extremely flexible manner through a single target text detailing the desired edit. Furthermore, our method, which we name $\textit{Direct Inversion}$, proposes multiple intuitively configurable hyperparameters to allow for a wide range of types and extents of real image edits. We prove our method's efficacy in producing high-quality, diverse, semantically coherent, and faithful real image edits through applying it on a variety of inputs for a multitude of tasks. We also formalize our method in well-established theory, detail future experiments for further improvement, and compare against state-of-the-art attempts.

Related papers

• PartEdit: Fine-Grained Image Editing using Pre-Trained Diffusion Models [80.98455219375862]
  We present the first text-based image editing approach for object parts based on pre-trained diffusion models. Our approach is preferred by users 77-90% of the time in conducted user studies.
  arXiv  Detail & Related papers  (2025-02-06T13:08:43Z)
• Textualize Visual Prompt for Image Editing via Diffusion Bridge [15.696208035498753]
  Current visual prompt methods rely on a pretrained text-guided image-to-image generative model. We present a framework based on any single text-to-image model without reliance on the explicit image-to-image model.
  arXiv  Detail & Related papers  (2025-01-07T03:33:22Z)
• Seek for Incantations: Towards Accurate Text-to-Image Diffusion Synthesis through Prompt Engineering [118.53208190209517]
  We propose a framework to learn the proper textual descriptions for diffusion models through prompt learning. Our method can effectively learn the prompts to improve the matches between the input text and the generated images.
  arXiv  Detail & Related papers  (2024-01-12T03:46:29Z)
• Text-Driven Image Editing via Learnable Regions [74.45313434129005]
  We introduce a method for region-based image editing driven by textual prompts, without the need for user-provided masks or sketches. We show that this simple approach enables flexible editing that is compatible with current image generation models. Experiments demonstrate the competitive performance of our method in manipulating images with high fidelity and realism that correspond to the provided language descriptions.
  arXiv  Detail & Related papers  (2023-11-28T02:27:31Z)
• Textual and Visual Prompt Fusion for Image Editing via Step-Wise Alignment [10.82748329166797]
  We propose a framework that integrates a fusion of generated visual references and text guidance into the semantic latent space of a textitfrozen pre-trained diffusion model. Using only a tiny neural network, our framework provides control over diverse content and attributes, driven intuitively by the text prompt.
  arXiv  Detail & Related papers  (2023-08-30T08:40:15Z)
• Prompt Tuning Inversion for Text-Driven Image Editing Using Diffusion Models [6.34777393532937]
  We propose an accurate and quick inversion technique, Prompt Tuning Inversion, for text-driven image editing. Our proposed editing method consists of a reconstruction stage and an editing stage. Experiments on ImageNet demonstrate the superior editing performance of our method compared to the state-of-the-art baselines.
  arXiv  Detail & Related papers  (2023-05-08T03:34:33Z)
• Towards Real-time Text-driven Image Manipulation with Unconditional Diffusion Models [33.993466872389085]
  We develop a novel algorithm that learns image manipulations 4.5-10 times faster and applies them 8 times faster. Our approach can adapt the pretrained model to the user-specified image and text description on the fly just for 4 seconds.
  arXiv  Detail & Related papers  (2023-04-10T01:21:56Z)
• Zero-shot Image-to-Image Translation [57.46189236379433]
  We propose pix2pix-zero, an image-to-image translation method that can preserve the original image without manual prompting. We propose cross-attention guidance, which aims to retain the cross-attention maps of the input image throughout the diffusion process. Our method does not need additional training for these edits and can directly use the existing text-to-image diffusion model.
  arXiv  Detail & Related papers  (2023-02-06T18:59:51Z)
• HumanDiffusion: a Coarse-to-Fine Alignment Diffusion Framework for Controllable Text-Driven Person Image Generation [73.3790833537313]
  Controllable person image generation promotes a wide range of applications such as digital human interaction and virtual try-on. We propose HumanDiffusion, a coarse-to-fine alignment diffusion framework, for text-driven person image generation.
  arXiv  Detail & Related papers  (2022-11-11T14:30:34Z)
• DiffEdit: Diffusion-based semantic image editing with mask guidance [64.555930158319]
  DiffEdit is a method to take advantage of text-conditioned diffusion models for the task of semantic image editing. Our main contribution is able to automatically generate a mask highlighting regions of the input image that need to be edited.
  arXiv  Detail & Related papers  (2022-10-20T17:16:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.