Streamlining Image Editing with Layered Diffusion Brushes

• URL: http://arxiv.org/abs/2405.00313v1
• Date: Wed, 1 May 2024 04:30:03 GMT
• Title: Streamlining Image Editing with Layered Diffusion Brushes
• Authors: Peyman Gholami, Robert Xiao,
• Abstract summary: Our system renders a single edit on a 512x512 image within 140 ms using a high-end consumer GPU. Our approach demonstrates efficacy across a range of tasks, including object attribute adjustments, error correction, and sequential prompt-based object placement and manipulation.
• Score: 8.738398948669609
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Denoising diffusion models have recently gained prominence as powerful tools for a variety of image generation and manipulation tasks. Building on this, we propose a novel tool for real-time editing of images that provides users with fine-grained region-targeted supervision in addition to existing prompt-based controls. Our novel editing technique, termed Layered Diffusion Brushes, leverages prompt-guided and region-targeted alteration of intermediate denoising steps, enabling precise modifications while maintaining the integrity and context of the input image. We provide an editor based on Layered Diffusion Brushes modifications, which incorporates well-known image editing concepts such as layer masks, visibility toggles, and independent manipulation of layers; regardless of their order. Our system renders a single edit on a 512x512 image within 140 ms using a high-end consumer GPU, enabling real-time feedback and rapid exploration of candidate edits. We validated our method and editing system through a user study involving both natural images (using inversion) and generated images, showcasing its usability and effectiveness compared to existing techniques such as InstructPix2Pix and Stable Diffusion Inpainting for refining images. Our approach demonstrates efficacy across a range of tasks, including object attribute adjustments, error correction, and sequential prompt-based object placement and manipulation, demonstrating its versatility and potential for enhancing creative workflows.

Related papers

• Stable Flow: Vital Layers for Training-Free Image Editing [74.52248787189302]
  Diffusion models have revolutionized the field of content synthesis and editing. Recent models have replaced the traditional UNet architecture with the Diffusion Transformer (DiT) We propose an automatic method to identify "vital layers" within DiT, crucial for image formation. Next, to enable real-image editing, we introduce an improved image inversion method for flow models.
  arXiv  Detail & Related papers  (2024-11-21T18:59:51Z)
• ControlEdit: A MultiModal Local Clothing Image Editing Method [3.6604114810930946]
  Multimodal clothing image editing refers to the precise adjustment and modification of clothing images using data such as textual descriptions and visual images as control conditions. We propose a new image editing method ControlEdit, which transfers clothing image editing to multimodal-guided local inpainting of clothing images.
  arXiv  Detail & Related papers  (2024-09-23T05:34:59Z)
• TurboEdit: Text-Based Image Editing Using Few-Step Diffusion Models [53.757752110493215]
  We focus on a popular line of text-based editing frameworks - the edit-friendly'' DDPM-noise inversion approach. We analyze its application to fast sampling methods and categorize its failures into two classes: the appearance of visual artifacts, and insufficient editing strength. We propose a pseudo-guidance approach that efficiently increases the magnitude of edits without introducing new artifacts.
  arXiv  Detail & Related papers  (2024-08-01T17:27:28Z)
• FreeDiff: Progressive Frequency Truncation for Image Editing with Diffusion Models [44.26371926512843]
  We introduce a novel free approach that employs progressive $textbfFre$qu$textbfe$ncy truncation to refine the guidance of $textbfDiff$usion models for universal editing tasks. Our method achieves comparable results with state-of-the-art methods across a variety of editing tasks and on a diverse set of images.
  arXiv  Detail & Related papers  (2024-04-18T04:47:28Z)
• DesignEdit: Multi-Layered Latent Decomposition and Fusion for Unified & Accurate Image Editing [22.855660721387167]
  We transform the spatial-aware image editing task into a combination of two sub-tasks: multi-layered latent decomposition and multi-layered latent fusion. We show that our approach consistently surpasses the latest spatial editing methods, including Self-Guidance and DiffEditor.
  arXiv  Detail & Related papers  (2024-03-21T15:35:42Z)
• DiffEditor: Boosting Accuracy and Flexibility on Diffusion-based Image Editing [66.43179841884098]
  Large-scale Text-to-Image (T2I) diffusion models have revolutionized image generation over the last few years. We propose DiffEditor to rectify two weaknesses in existing diffusion-based image editing. Our method can efficiently achieve state-of-the-art performance on various fine-grained image editing tasks.
  arXiv  Detail & Related papers  (2024-02-04T18:50:29Z)
• Optimisation-Based Multi-Modal Semantic Image Editing [58.496064583110694]
  We propose an inference-time editing optimisation to accommodate multiple editing instruction types. By allowing to adjust the influence of each loss function, we build a flexible editing solution that can be adjusted to user preferences. We evaluate our method using text, pose and scribble edit conditions, and highlight our ability to achieve complex edits.
  arXiv  Detail & Related papers  (2023-11-28T15:31:11Z)
• Emu Edit: Precise Image Editing via Recognition and Generation Tasks [62.95717180730946]
  We present Emu Edit, a multi-task image editing model which sets state-of-the-art results in instruction-based image editing. We train it to multi-task across an unprecedented range of tasks, such as region-based editing, free-form editing, and Computer Vision tasks. We show that Emu Edit can generalize to new tasks, such as image inpainting, super-resolution, and compositions of editing tasks, with just a few labeled examples.
  arXiv  Detail & Related papers  (2023-11-16T18:55:58Z)
• PFB-Diff: Progressive Feature Blending Diffusion for Text-driven Image Editing [8.19063619210761]
  PFB-Diff is a Progressive Feature Blending method for Diffusion-based image editing. Our method demonstrates its superior performance in terms of image fidelity, editing accuracy, efficiency, and faithfulness to the original image.
  arXiv  Detail & Related papers  (2023-06-28T11:10:20Z)
• LayerDiffusion: Layered Controlled Image Editing with Diffusion Models [5.58892860792971]
  LayerDiffusion is a semantic-based layered controlled image editing method. We leverage a large-scale text-to-image model and employ a layered controlled optimization strategy. Experimental results demonstrate the effectiveness of our method in generating highly coherent images.
  arXiv  Detail & Related papers  (2023-05-30T01:26:41Z)
• StyleDiffusion: Prompt-Embedding Inversion for Text-Based Editing [86.92711729969488]
  We exploit the amazing capacities of pretrained diffusion models for the editing of images. They either finetune the model, or invert the image in the latent space of the pretrained model. They suffer from two problems: Unsatisfying results for selected regions, and unexpected changes in nonselected regions.
  arXiv  Detail & Related papers  (2023-03-28T00:16:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.