The Curious Case of End Token: A Zero-Shot Disentangled Image Editing using CLIP

• URL: http://arxiv.org/abs/2406.00457v1
• Date: Sat, 1 Jun 2024 14:46:57 GMT
• Title: The Curious Case of End Token: A Zero-Shot Disentangled Image Editing using CLIP
• Authors: Hidir Yesiltepe, Yusuf Dalva, Pinar Yanardag,
• Abstract summary: We show that CLIP is capable of performing disentangled editing in a zero-shot manner. This insight may open opportunities for applying this method to various tasks, including image and video editing.
• Score: 4.710921988115686
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion models have become prominent in creating high-quality images. However, unlike GAN models celebrated for their ability to edit images in a disentangled manner, diffusion-based text-to-image models struggle to achieve the same level of precise attribute manipulation without compromising image coherence. In this paper, CLIP which is often used in popular text-to-image diffusion models such as Stable Diffusion is capable of performing disentangled editing in a zero-shot manner. Through both qualitative and quantitative comparisons with state-of-the-art editing methods, we show that our approach yields competitive results. This insight may open opportunities for applying this method to various tasks, including image and video editing, providing a lightweight and efficient approach for disentangled editing.

Related papers

• JIGMARK: A Black-Box Approach for Enhancing Image Watermarks against Diffusion Model Edits [76.25962336540226]
  JIGMARK is a first-of-its-kind watermarking technique that enhances robustness through contrastive learning. Our evaluation reveals that JIGMARK significantly surpasses existing watermarking solutions in resilience to diffusion-model edits.
  arXiv  Detail & Related papers  (2024-06-06T03:31:41Z)
• Unified Editing of Panorama, 3D Scenes, and Videos Through Disentangled Self-Attention Injection [60.47731445033151]
  We propose a novel unified editing framework that combines the strengths of both approaches by utilizing only a basic 2D image text-to-image (T2I) diffusion model. Experimental results confirm that our method enables editing across diverse modalities including 3D scenes, videos, and panorama images.
  arXiv  Detail & Related papers  (2024-05-27T04:44:36Z)
• Eta Inversion: Designing an Optimal Eta Function for Diffusion-based Real Image Editing [2.5602836891933074]
  A commonly adopted strategy for editing real images involves inverting the diffusion process to obtain a noisy representation of the original image. Current methods for diffusion inversion often struggle to produce edits that are both faithful to the specified text prompt and closely resemble the source image. We introduce a novel and adaptable diffusion inversion technique for real image editing, which is grounded in a theoretical analysis of the role of $eta$ in the DDIM sampling equation for enhanced editability.
  arXiv  Detail & Related papers  (2024-03-14T15:07:36Z)
• Diffusion Model-Based Image Editing: A Survey [46.244266782108234]
  Denoising diffusion models have emerged as a powerful tool for various image generation and editing tasks. We provide an exhaustive overview of existing methods using diffusion models for image editing. To further evaluate the performance of text-guided image editing algorithms, we propose a systematic benchmark, EditEval.
  arXiv  Detail & Related papers  (2024-02-27T14:07:09Z)
• DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editing [66.43179841884098]
  Large-scale Text-to-Image (T2I) diffusion models have revolutionized image generation over the last few years. We propose DiffEditor to rectify two weaknesses in existing diffusion-based image editing. Our method can efficiently achieve state-of-the-art performance on various fine-grained image editing tasks.
  arXiv  Detail & Related papers  (2024-02-04T18:50:29Z)
• DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models [66.43179841884098]
  We propose a novel image editing method, DragonDiffusion, enabling Drag-style manipulation on Diffusion models. Our method achieves various editing modes for the generated or real images, such as object moving, object resizing, object appearance replacement, and content dragging.
  arXiv  Detail & Related papers  (2023-07-05T16:43:56Z)
• DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing [94.24479528298252]
  DragGAN is an interactive point-based image editing framework that achieves impressive editing results with pixel-level precision. By harnessing large-scale pretrained diffusion models, we greatly enhance the applicability of interactive point-based editing on both real and diffusion-generated images. We present a challenging benchmark dataset called DragBench to evaluate the performance of interactive point-based image editing methods.
  arXiv  Detail & Related papers  (2023-06-26T06:04:09Z)
• Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models [60.63556257324894]
  A key desired property of image generative models is the ability to disentangle different attributes. We propose a simple, light-weight image editing algorithm where the mixing weights of the two text embeddings are optimized for style matching and content preservation. Experiments show that the proposed method can modify a wide range of attributes, with the performance outperforming diffusion-model-based image-editing algorithms.
  arXiv  Detail & Related papers  (2022-12-16T19:58:52Z)
• SINE: SINgle Image Editing with Text-to-Image Diffusion Models [10.67527134198167]
  This work aims to address the problem of single-image editing. We propose a novel model-based guidance built upon the classifier-free guidance. We show promising editing capabilities, including changing style, content addition, and object manipulation.
  arXiv  Detail & Related papers  (2022-12-08T18:57:13Z)
• Paint by Example: Exemplar-based Image Editing with Diffusion Models [35.84464684227222]
  In this paper, we investigate exemplar-guided image editing for more precise control. We achieve this goal by leveraging self-supervised training to disentangle and re-organize the source image and the exemplar. We demonstrate that our method achieves an impressive performance and enables controllable editing on in-the-wild images with high fidelity.
  arXiv  Detail & Related papers  (2022-11-23T18:59:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.