DragText: Rethinking Text Embedding in Point-based Image Editing

• URL: http://arxiv.org/abs/2407.17843v1
• Date: Thu, 25 Jul 2024 07:57:55 GMT
• Title: DragText: Rethinking Text Embedding in Point-based Image Editing
• Authors: Gayoon Choi, Taejin Jeong, Sujung Hong, Jaehoon Joo, Seong Jae Hwang,
• Abstract summary: We show that during the progressive editing of an input image in a diffusion model, the text embedding remains constant. We propose DragText, which optimize text embedding in conjunction with the dragging process to pair with the modified image embedding.
• Score: 3.1923251959845214
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Point-based image editing enables accurate and flexible control through content dragging. However, the role of text embedding in the editing process has not been thoroughly investigated. A significant aspect that remains unexplored is the interaction between text and image embeddings. In this study, we show that during the progressive editing of an input image in a diffusion model, the text embedding remains constant. As the image embedding increasingly diverges from its initial state, the discrepancy between the image and text embeddings presents a significant challenge. Moreover, we found that the text prompt significantly influences the dragging process, particularly in maintaining content integrity and achieving the desired manipulation. To utilize these insights, we propose DragText, which optimizes text embedding in conjunction with the dragging process to pair with the modified image embedding. Simultaneously, we regularize the text optimization process to preserve the integrity of the original text prompt. Our approach can be seamlessly integrated with existing diffusion-based drag methods with only a few lines of code.

Related papers

• Prompt-Softbox-Prompt: A free-text Embedding Control for Image Editing [10.12329842607126]
  The entanglement and opacity of text embeddings present significant challenges to achieving precise image editing. We propose a novel approach for controllable image editing using a free-text embedding control method called PSP (Prompt-Softbox-Prompt) PSP enables precise image editing by inserting or adding text embeddings within the cross-attention layers and using Softbox to define and control the specific area for semantic injection.
  arXiv  Detail & Related papers  (2024-08-24T16:33:26Z)
• E4C: Enhance Editability for Text-Based Image Editing by Harnessing Efficient CLIP Guidance [13.535394339438428]
  Diffusion-based image editing is a composite process of preserving the source image content and generating new content or applying modifications. We propose a zero-shot image editing method, named textbfEnhance textbfEditability for text-based image textbfEditing via textbfCLIP guidance.
  arXiv  Detail & Related papers  (2024-03-15T09:26:48Z)
• Latent Inversion with Timestep-aware Sampling for Training-free Non-rigid Editing [56.536695050042546]
  We propose a training-free approach for non-rigid editing with Stable Diffusion. Our approach comprises three stages: text optimization, latent inversion, and timestep-aware text injection sampling. We demonstrate the effectiveness of our method in terms of identity preservation, editability, and aesthetic quality.
  arXiv  Detail & Related papers  (2024-02-13T17:08:35Z)
• Unified Diffusion-Based Rigid and Non-Rigid Editing with Text and Image Guidance [15.130419159003816]
  We present a versatile image editing framework capable of executing both rigid and non-rigid edits. We leverage a dual-path injection scheme to handle diverse editing scenarios. We introduce an integrated self-attention mechanism for fusion of appearance and structural information.
  arXiv  Detail & Related papers  (2024-01-04T08:21:30Z)
• Enhancing Scene Text Detectors with Realistic Text Image Synthesis Using Diffusion Models [63.99110667987318]
  We present DiffText, a pipeline that seamlessly blends foreground text with the background's intrinsic features. With fewer text instances, our produced text images consistently surpass other synthetic data in aiding text detectors.
  arXiv  Detail & Related papers  (2023-11-28T06:51:28Z)
• Zero-shot Image-to-Image Translation [57.46189236379433]
  We propose pix2pix-zero, an image-to-image translation method that can preserve the original image without manual prompting. We propose cross-attention guidance, which aims to retain the cross-attention maps of the input image throughout the diffusion process. Our method does not need additional training for these edits and can directly use the existing text-to-image diffusion model.
  arXiv  Detail & Related papers  (2023-02-06T18:59:51Z)
• Towards Arbitrary Text-driven Image Manipulation via Space Alignment [49.3370305074319]
  We propose a new Text-driven image Manipulation framework via Space Alignment (TMSA) TMSA aims to align the same semantic regions in CLIP and StyleGAN spaces. The framework can support arbitrary image editing mode without additional cost.
  arXiv  Detail & Related papers  (2023-01-25T16:20:01Z)
• Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting [53.708523312636096]
  We present Imagen Editor, a cascaded diffusion model built, by fine-tuning on text-guided image inpainting. edits are faithful to the text prompts, which is accomplished by using object detectors to propose inpainting masks during training. To improve qualitative and quantitative evaluation, we introduce EditBench, a systematic benchmark for text-guided image inpainting.
  arXiv  Detail & Related papers  (2022-12-13T21:25:11Z)
• Interactive Image Manipulation with Complex Text Instructions [14.329411711887115]
  We propose a novel image manipulation method that interactively edits an image using complex text instructions. It allows users to not only improve the accuracy of image manipulation but also achieve complex tasks such as enlarging, dwindling, or removing objects. Extensive experiments on the Caltech-UCSD Birds-200-2011 (CUB) dataset and Microsoft Common Objects in Context (MS COCO) datasets demonstrate our proposed method can enable interactive, flexible, and accurate image manipulation in real-time.
  arXiv  Detail & Related papers  (2022-11-25T08:05:52Z)
• DiffEdit: Diffusion-based semantic image editing with mask guidance [64.555930158319]
  DiffEdit is a method to take advantage of text-conditioned diffusion models for the task of semantic image editing. Our main contribution is able to automatically generate a mask highlighting regions of the input image that need to be edited.
  arXiv  Detail & Related papers  (2022-10-20T17:16:37Z)
• Prompt-to-Prompt Image Editing with Cross Attention Control [41.26939787978142]
  We present an intuitive prompt-to-prompt editing framework, where the edits are controlled by text only. We show our results over diverse images and prompts, demonstrating high-quality synthesis and fidelity to the edited prompts.
  arXiv  Detail & Related papers  (2022-08-02T17:55:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.