Paint by Example: Exemplar-based Image Editing with Diffusion Models

• URL: http://arxiv.org/abs/2211.13227v1
• Date: Wed, 23 Nov 2022 18:59:52 GMT
• Title: Paint by Example: Exemplar-based Image Editing with Diffusion Models
• Authors: Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen and Fang Wen
• Abstract summary: In this paper, we investigate exemplar-guided image editing for more precise control. We achieve this goal by leveraging self-supervised training to disentangle and re-organize the source image and the exemplar. We demonstrate that our method achieves an impressive performance and enables controllable editing on in-the-wild images with high fidelity.
• Score: 35.84464684227222
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language-guided image editing has achieved great success recently. In this paper, for the first time, we investigate exemplar-guided image editing for more precise control. We achieve this goal by leveraging self-supervised training to disentangle and re-organize the source image and the exemplar. However, the naive approach will cause obvious fusing artifacts. We carefully analyze it and propose an information bottleneck and strong augmentations to avoid the trivial solution of directly copying and pasting the exemplar image. Meanwhile, to ensure the controllability of the editing process, we design an arbitrary shape mask for the exemplar image and leverage the classifier-free guidance to increase the similarity to the exemplar image. The whole framework involves a single forward of the diffusion model without any iterative optimization. We demonstrate that our method achieves an impressive performance and enables controllable editing on in-the-wild images with high fidelity.

Related papers

• PartEdit: Fine-Grained Image Editing using Pre-Trained Diffusion Models [80.98455219375862]
  We present the first text-based image editing approach for object parts based on pre-trained diffusion models. Our approach is preferred by users 77-90% of the time in conducted user studies.
  arXiv  Detail & Related papers  (2025-02-06T13:08:43Z)
• Edicho: Consistent Image Editing in the Wild [90.42395533938915]
  Edicho steps in with a training-free solution based on diffusion models. It features a fundamental design principle of using explicit image correspondence to direct editing.
  arXiv  Detail & Related papers  (2024-12-30T16:56:44Z)
• Diffusion-Based Conditional Image Editing through Optimized Inference with Guidance [46.922018440110826]
  We present a training-free approach for text-driven image-to-image translation based on a pretrained text-to-image diffusion model. Our method achieves outstanding image-to-image translation performance on various tasks when combined with the pretrained Stable Diffusion model.
  arXiv  Detail & Related papers  (2024-12-20T11:15:31Z)
• Uniform Attention Maps: Boosting Image Fidelity in Reconstruction and Editing [66.48853049746123]
  We analyze reconstruction from a structural perspective and propose a novel approach that replaces traditional cross-attention with uniform attention maps. Our method effectively minimizes distortions caused by varying text conditions during noise prediction. Experimental results demonstrate that our approach not only excels in achieving high-fidelity image reconstruction but also performs robustly in real image composition and editing scenarios.
  arXiv  Detail & Related papers  (2024-11-29T12:11:28Z)
• Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free Real Image Editing [42.73883397041092]
  We propose a novel approach that is built upon a modified diffusion sampling process via the guidance mechanism. In this work, we explore the self-guidance technique to preserve the overall structure of the input image. We show through human evaluation and quantitative analysis that the proposed method allows to produce desired editing.
  arXiv  Detail & Related papers  (2024-09-02T15:21:46Z)
• TurboEdit: Text-Based Image Editing Using Few-Step Diffusion Models [53.757752110493215]
  We focus on a popular line of text-based editing frameworks - the edit-friendly'' DDPM-noise inversion approach. We analyze its application to fast sampling methods and categorize its failures into two classes: the appearance of visual artifacts, and insufficient editing strength. We propose a pseudo-guidance approach that efficiently increases the magnitude of edits without introducing new artifacts.
  arXiv  Detail & Related papers  (2024-08-01T17:27:28Z)
• The Curious Case of End Token: A Zero-Shot Disentangled Image Editing using CLIP [4.710921988115686]
  We show that CLIP is capable of performing disentangled editing in a zero-shot manner. This insight may open opportunities for applying this method to various tasks, including image and video editing.
  arXiv  Detail & Related papers  (2024-06-01T14:46:57Z)
• Eta Inversion: Designing an Optimal Eta Function for Diffusion-based Real Image Editing [2.5602836891933074]
  A commonly adopted strategy for editing real images involves inverting the diffusion process to obtain a noisy representation of the original image. Current methods for diffusion inversion often struggle to produce edits that are both faithful to the specified text prompt and closely resemble the source image. We introduce a novel and adaptable diffusion inversion technique for real image editing, which is grounded in a theoretical analysis of the role of $eta$ in the DDIM sampling equation for enhanced editability.
  arXiv  Detail & Related papers  (2024-03-14T15:07:36Z)
• DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing [94.24479528298252]
  DragGAN is an interactive point-based image editing framework that achieves impressive editing results with pixel-level precision. By harnessing large-scale pretrained diffusion models, we greatly enhance the applicability of interactive point-based editing on both real and diffusion-generated images. We present a challenging benchmark dataset called DragBench to evaluate the performance of interactive point-based image editing methods.
  arXiv  Detail & Related papers  (2023-06-26T06:04:09Z)
• ReGeneration Learning of Diffusion Models with Rich Prompts for Zero-Shot Image Translation [8.803251014279502]
  Large-scale text-to-image models have demonstrated amazing ability to synthesize diverse and high-fidelity images. Current models can impose significant changes to the original image content during the editing process. We propose ReGeneration learning in an image-to-image Diffusion model (ReDiffuser)
  arXiv  Detail & Related papers  (2023-05-08T12:08:12Z)
• End-to-End Visual Editing with a Generatively Pre-Trained Artist [78.5922562526874]
  We consider the targeted image editing problem: blending a region in a source image with a driver image that specifies the desired change. We propose a self-supervised approach that simulates edits by augmenting off-the-shelf images in a target domain. We show that different blending effects can be learned by an intuitive control of the augmentation process, with no other changes required to the model architecture.
  arXiv  Detail & Related papers  (2022-05-03T17:59:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.