E4C: Enhance Editability for Text-Based Image Editing by Harnessing Efficient CLIP Guidance

• URL: http://arxiv.org/abs/2403.10133v1
• Date: Fri, 15 Mar 2024 09:26:48 GMT
• Title: E4C: Enhance Editability for Text-Based Image Editing by Harnessing Efficient CLIP Guidance
• Authors: Tianrui Huang, Pu Cao, Lu Yang, Chun Liu, Mengjie Hu, Zhiwei Liu, Qing Song,
• Abstract summary: Diffusion-based image editing is a composite process of preserving the source image content and generating new content or applying modifications. We propose a zero-shot image editing method, named textbfEnhance textbfEditability for text-based image textbfEditing via textbfCLIP guidance.
• Score: 13.535394339438428
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion-based image editing is a composite process of preserving the source image content and generating new content or applying modifications. While current editing approaches have made improvements under text guidance, most of them have only focused on preserving the information of the input image, disregarding the importance of editability and alignment to the target prompt. In this paper, we prioritize the editability by proposing a zero-shot image editing method, named \textbf{E}nhance \textbf{E}ditability for text-based image \textbf{E}diting via \textbf{E}fficient \textbf{C}LIP guidance (\textbf{E4C}), which only requires inference-stage optimization to explicitly enhance the edibility and text alignment. Specifically, we develop a unified dual-branch feature-sharing pipeline that enables the preservation of the structure or texture of the source image while allowing the other to be adapted based on the editing task. We further integrate CLIP guidance into our pipeline by utilizing our novel random-gateway optimization mechanism to efficiently enhance the semantic alignment with the target prompt. Comprehensive quantitative and qualitative experiments demonstrate that our method effectively resolves the text alignment issues prevalent in existing methods while maintaining the fidelity to the source image, and performs well across a wide range of editing tasks.

Related papers

• Task-Oriented Diffusion Inversion for High-Fidelity Text-based Editing [60.730661748555214]
  We introduce textbfTask-textbfOriented textbfDiffusion textbfInversion (textbfTODInv), a novel framework that inverts and edits real images tailored to specific editing tasks. ToDInv seamlessly integrates inversion and editing through reciprocal optimization, ensuring both high fidelity and precise editability.
  arXiv  Detail & Related papers  (2024-08-23T22:16:34Z)
• DragText: Rethinking Text Embedding in Point-based Image Editing [3.1923251959845214]
  We show that during the progressive editing of an input image in a diffusion model, the text embedding remains constant. We propose DragText, which optimize text embedding in conjunction with the dragging process to pair with the modified image embedding.
  arXiv  Detail & Related papers  (2024-07-25T07:57:55Z)
• Unified Diffusion-Based Rigid and Non-Rigid Editing with Text and Image Guidance [15.130419159003816]
  We present a versatile image editing framework capable of executing both rigid and non-rigid edits. We leverage a dual-path injection scheme to handle diverse editing scenarios. We introduce an integrated self-attention mechanism for fusion of appearance and structural information.
  arXiv  Detail & Related papers  (2024-01-04T08:21:30Z)
• BARET : Balanced Attention based Real image Editing driven by Target-text Inversion [36.59406959595952]
  We propose a novel editing technique that only requires an input image and target text for various editing types including non-rigid edits without fine-tuning diffusion model. Our method contains three novelties: (I) Targettext Inversion Schedule (TTIS) is designed to fine-tune the input target text embedding to achieve fast image reconstruction without image caption and acceleration of convergence; (II) Progressive Transition Scheme applies progressive linear approaches between target text embedding and its fine-tuned version to generate transition embedding for maintaining non-rigid editing capability; (III) Balanced Attention Module (BAM) balances the tradeoff between textual description and image semantics
  arXiv  Detail & Related papers  (2023-12-09T07:18:23Z)
• Customize your NeRF: Adaptive Source Driven 3D Scene Editing via Local-Global Iterative Training [61.984277261016146]
  We propose a CustomNeRF model that unifies a text description or a reference image as the editing prompt. To tackle the first challenge, we propose a Local-Global Iterative Editing (LGIE) training scheme that alternates between foreground region editing and full-image editing. For the second challenge, we also design a class-guided regularization that exploits class priors within the generation model to alleviate the inconsistency problem.
  arXiv  Detail & Related papers  (2023-12-04T06:25:06Z)
• Optimisation-Based Multi-Modal Semantic Image Editing [58.496064583110694]
  We propose an inference-time editing optimisation to accommodate multiple editing instruction types. By allowing to adjust the influence of each loss function, we build a flexible editing solution that can be adjusted to user preferences. We evaluate our method using text, pose and scribble edit conditions, and highlight our ability to achieve complex edits.
  arXiv  Detail & Related papers  (2023-11-28T15:31:11Z)
• CLIP-Guided StyleGAN Inversion for Text-Driven Real Image Editing [22.40686064568406]
  We present CLIPInverter, a new text-driven image editing approach that is able to efficiently and reliably perform multi-attribute changes. Our method outperforms competing approaches in terms of manipulation accuracy and photo-realism on various domains including human faces, cats, and birds.
  arXiv  Detail & Related papers  (2023-07-17T11:29:48Z)
• iEdit: Localised Text-guided Image Editing with Weak Supervision [53.082196061014734]
  We propose a novel learning method for text-guided image editing. It generates images conditioned on a source image and a textual edit prompt. It shows favourable results against its counterparts in terms of image fidelity, CLIP alignment score and qualitatively for editing both generated and real images.
  arXiv  Detail & Related papers  (2023-05-10T07:39:14Z)
• Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting [53.708523312636096]
  We present Imagen Editor, a cascaded diffusion model built, by fine-tuning on text-guided image inpainting. edits are faithful to the text prompts, which is accomplished by using object detectors to propose inpainting masks during training. To improve qualitative and quantitative evaluation, we introduce EditBench, a systematic benchmark for text-guided image inpainting.
  arXiv  Detail & Related papers  (2022-12-13T21:25:11Z)
• Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors [58.71128866226768]
  Recent text-to-image generation methods have incrementally improved the generated image fidelity and text relevancy. We propose a novel text-to-image method that addresses these gaps by (i) enabling a simple control mechanism complementary to text in the form of a scene. Our model achieves state-of-the-art FID and human evaluation results, unlocking the ability to generate high fidelity images in a resolution of 512x512 pixels.
  arXiv  Detail & Related papers  (2022-03-24T15:44:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.