Drag Your Noise: Interactive Point-based Editing via Diffusion Semantic Propagation

• URL: http://arxiv.org/abs/2404.01050v1
• Date: Mon, 1 Apr 2024 11:09:40 GMT
• Title: Drag Your Noise: Interactive Point-based Editing via Diffusion Semantic Propagation
• Authors: Haofeng Liu, Chenshu Xu, Yifei Yang, Lihua Zeng, Shengfeng He,
• Abstract summary: DragNoise offers robust and accelerated editing without retracing the latent map. The bottleneck features of U-Net inherently possess semantically rich features ideal for interactive editing. DragNoise achieves superior control and semantic retention, reducing the optimization time by over 50% compared to DragDiffusion.
• Score: 30.737586652869457
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Point-based interactive editing serves as an essential tool to complement the controllability of existing generative models. A concurrent work, DragDiffusion, updates the diffusion latent map in response to user inputs, causing global latent map alterations. This results in imprecise preservation of the original content and unsuccessful editing due to gradient vanishing. In contrast, we present DragNoise, offering robust and accelerated editing without retracing the latent map. The core rationale of DragNoise lies in utilizing the predicted noise output of each U-Net as a semantic editor. This approach is grounded in two critical observations: firstly, the bottleneck features of U-Net inherently possess semantically rich features ideal for interactive editing; secondly, high-level semantics, established early in the denoising process, show minimal variation in subsequent stages. Leveraging these insights, DragNoise edits diffusion semantics in a single denoising step and efficiently propagates these changes, ensuring stability and efficiency in diffusion editing. Comparative experiments reveal that DragNoise achieves superior control and semantic retention, reducing the optimization time by over 50% compared to DragDiffusion. Our codes are available at https://github.com/haofengl/DragNoise.

Related papers

• DIFFVSGG: Diffusion-Driven Online Video Scene Graph Generation [61.59996525424585]
  DIFFVSGG is an online VSGG solution that frames this task as an iterative scene graph update problem.<n>We unify the decoding of object classification, bounding box regression, and graph generation three tasks using one shared feature embedding.<n>DIFFVSGG further facilitates continuous temporal reasoning, where predictions for subsequent frames leverage results of past frames as the conditional inputs of LDMs.
  arXiv  Detail & Related papers  (2025-03-18T06:49:51Z)
• OmniPaint: Mastering Object-Oriented Editing via Disentangled Insertion-Removal Inpainting [54.525583840585305]
  We introduce OmniPaint, a unified framework that re-conceptualizes object removal and insertion as interdependent processes. Our novel CFD metric offers a robust, reference-free evaluation of context consistency and object hallucination.
  arXiv  Detail & Related papers  (2025-03-11T17:55:27Z)
• Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis [9.11767497956649]
  This paper proposes leveraging the language comprehension capabilities of large vision-language models to guide the optimization of the initial noisy latent. We introduce the Noise Diffusion process, which updates the noisy latent to generate semantically faithful images while preserving distribution consistency. Experimental results demonstrate the effectiveness and adaptability of our framework, consistently enhancing semantic alignment across various diffusion models.
  arXiv  Detail & Related papers  (2024-11-25T15:40:47Z)
• Task-Oriented Diffusion Inversion for High-Fidelity Text-based Editing [60.730661748555214]
  We introduce textbfTask-textbfOriented textbfDiffusion textbfInversion (textbfTODInv), a novel framework that inverts and edits real images tailored to specific editing tasks. ToDInv seamlessly integrates inversion and editing through reciprocal optimization, ensuring both high fidelity and precise editability.
  arXiv  Detail & Related papers  (2024-08-23T22:16:34Z)
• TurboEdit: Text-Based Image Editing Using Few-Step Diffusion Models [53.757752110493215]
  We focus on a popular line of text-based editing frameworks - the edit-friendly'' DDPM-noise inversion approach. We analyze its application to fast sampling methods and categorize its failures into two classes: the appearance of visual artifacts, and insufficient editing strength. We propose a pseudo-guidance approach that efficiently increases the magnitude of edits without introducing new artifacts.
  arXiv  Detail & Related papers  (2024-08-01T17:27:28Z)
• COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing [57.76170824395532]
  Video editing is an emerging task, in which most current methods adopt the pre-trained text-to-image (T2I) diffusion model to edit the source video. We propose COrrespondence-guided Video Editing (COVE) to achieve high-quality and consistent video editing. COVE can be seamlessly integrated into the pre-trained T2I diffusion model without the need for extra training or optimization.
  arXiv  Detail & Related papers  (2024-06-13T06:27:13Z)
• FastDrag: Manipulate Anything in One Step [20.494157877241665]
  We introduce a novel one-step drag-based image editing method, i.e., FastDrag, to accelerate the editing process. This innovation achieves one-step latent semantic optimization and hence significantly promotes editing speeds. Our FastDrag is validated on the DragBench dataset, demonstrating substantial improvements in processing time over existing methods.
  arXiv  Detail & Related papers  (2024-05-24T17:59:26Z)
• GoodDrag: Towards Good Practices for Drag Editing with Diffusion Models [31.708968272342315]
  We introduce GoodDrag, a novel approach to improve the stability and image quality of drag editing. GoodDrag introduces an AlDD framework that alternates between drag and denoising operations within the diffusion process. We also propose an information-preserving motion supervision operation that maintains the original features of the starting point for precise manipulation and artifact reduction.
  arXiv  Detail & Related papers  (2024-04-10T17:59:59Z)
• Object-Centric Diffusion for Efficient Video Editing [64.71639719352636]
  Diffusion-based video editing has reached impressive quality. Such solutions typically incur heavy memory and computational costs to generate temporally-coherent frames. We propose modifications that allow significant speed-ups whilst maintaining quality.
  arXiv  Detail & Related papers  (2024-01-11T08:36:15Z)
• Inversion-Free Image Editing with Natural Language [18.373145158518135]
  We present inversion-free editing (InfEdit), which allows for consistent and faithful editing for both rigid and non-rigid semantic changes. InfEdit shows strong performance in various editing tasks and also maintains a seamless workflow (less than 3 seconds on one single A40), demonstrating the potential for real-time applications.
  arXiv  Detail & Related papers  (2023-12-07T18:58:27Z)
• Speech Synthesis By Unrolling Diffusion Process using Neural Network Layers [3.2634122554914002]
  UDPNet is a novel architecture designed to accelerate the reverse diffusion process in speech synthesis.<n>We show that UDPNet consistently outperforms state-of-the-art methods in both quality and efficiency.<n>These results position UDPNet as a robust solution for real-time speech synthesis applications.
  arXiv  Detail & Related papers  (2023-09-18T10:35:27Z)
• DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models [66.43179841884098]
  We propose a novel image editing method, DragonDiffusion, enabling Drag-style manipulation on Diffusion models. Our method achieves various editing modes for the generated or real images, such as object moving, object resizing, object appearance replacement, and content dragging.
  arXiv  Detail & Related papers  (2023-07-05T16:43:56Z)
• DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing [94.24479528298252]
  DragGAN is an interactive point-based image editing framework that achieves impressive editing results with pixel-level precision. By harnessing large-scale pretrained diffusion models, we greatly enhance the applicability of interactive point-based editing on both real and diffusion-generated images. We present a challenging benchmark dataset called DragBench to evaluate the performance of interactive point-based image editing methods.
  arXiv  Detail & Related papers  (2023-06-26T06:04:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.