DesignEdit: Multi-Layered Latent Decomposition and Fusion for Unified & Accurate Image Editing

• URL: http://arxiv.org/abs/2403.14487v1
• Date: Thu, 21 Mar 2024 15:35:42 GMT
• Title: DesignEdit: Multi-Layered Latent Decomposition and Fusion for Unified & Accurate Image Editing
• Authors: Yueru Jia, Yuhui Yuan, Aosong Cheng, Chuke Wang, Ji Li, Huizhu Jia, Shanghang Zhang,
• Abstract summary: We transform the spatial-aware image editing task into a combination of two sub-tasks: multi-layered latent decomposition and multi-layered latent fusion. We show that our approach consistently surpasses the latest spatial editing methods, including Self-Guidance and DiffEditor.
• Score: 22.855660721387167
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, how to achieve precise image editing has attracted increasing attention, especially given the remarkable success of text-to-image generation models. To unify various spatial-aware image editing abilities into one framework, we adopt the concept of layers from the design domain to manipulate objects flexibly with various operations. The key insight is to transform the spatial-aware image editing task into a combination of two sub-tasks: multi-layered latent decomposition and multi-layered latent fusion. First, we segment the latent representations of the source images into multiple layers, which include several object layers and one incomplete background layer that necessitates reliable inpainting. To avoid extra tuning, we further explore the inner inpainting ability within the self-attention mechanism. We introduce a key-masking self-attention scheme that can propagate the surrounding context information into the masked region while mitigating its impact on the regions outside the mask. Second, we propose an instruction-guided latent fusion that pastes the multi-layered latent representations onto a canvas latent. We also introduce an artifact suppression scheme in the latent space to enhance the inpainting quality. Due to the inherent modular advantages of such multi-layered representations, we can achieve accurate image editing, and we demonstrate that our approach consistently surpasses the latest spatial editing methods, including Self-Guidance and DiffEditor. Last, we show that our approach is a unified framework that supports various accurate image editing tasks on more than six different editing tasks.

Related papers

• FACEMUG: A Multimodal Generative and Fusion Framework for Local Facial Editing [10.123066253648307]
  We present a novel framework for globally-consistent local facial editing (FACEMUG) It can handle a wide range of input modalities and enable fine-grained and semantic manipulation while remaining unedited parts unchanged. We introduce a novel self-supervised latent warping algorithm to rectify misaligned facial features.
  arXiv  Detail & Related papers  (2024-12-26T00:53:54Z)
• BrushEdit: All-In-One Image Inpainting and Editing [79.55816192146762]
  BrushEdit is a novel inpainting-based instruction-guided image editing paradigm. We devise a system enabling free-form instruction editing by integrating MLLMs and a dual-branch image inpainting model. Our framework effectively combines MLLMs and inpainting models, achieving superior performance across seven metrics.
  arXiv  Detail & Related papers  (2024-12-13T17:58:06Z)
• Generative Image Layer Decomposition with Visual Effects [49.75021036203426]
  LayerDecomp is a generative framework for image layer decomposition. It produces clean backgrounds and high-quality transparent foregrounds with faithfully preserved visual effects. Our method achieves superior quality in layer decomposition, outperforming existing approaches in object removal and spatial editing tasks.
  arXiv  Detail & Related papers  (2024-11-26T20:26:49Z)
• Streamlining Image Editing with Layered Diffusion Brushes [8.738398948669609]
  Our system renders a single edit on a 512x512 image within 140 ms using a high-end consumer GPU. Our approach demonstrates efficacy across a range of tasks, including object attribute adjustments, error correction, and sequential prompt-based object placement and manipulation.
  arXiv  Detail & Related papers  (2024-05-01T04:30:03Z)
• LoMOE: Localized Multi-Object Editing via Multi-Diffusion [8.90467024388923]
  We introduce a novel framework for zero-shot localized multi-object editing through a multi-diffusion process. Our approach leverages foreground masks and corresponding simple text prompts that exert localized influences on the target regions. A combination of cross-attention and background losses within the latent space ensures that the characteristics of the object being edited are preserved.
  arXiv  Detail & Related papers  (2024-03-01T10:46:47Z)
• Consolidating Attention Features for Multi-view Image Editing [126.19731971010475]
  We focus on spatial control-based geometric manipulations and introduce a method to consolidate the editing process across various views. We introduce QNeRF, a neural radiance field trained on the internal query features of the edited images. We refine the process through a progressive, iterative method that better consolidates queries across the diffusion timesteps.
  arXiv  Detail & Related papers  (2024-02-22T18:50:18Z)
• LayerDiffusion: Layered Controlled Image Editing with Diffusion Models [5.58892860792971]
  LayerDiffusion is a semantic-based layered controlled image editing method. We leverage a large-scale text-to-image model and employ a layered controlled optimization strategy. Experimental results demonstrate the effectiveness of our method in generating highly coherent images.
  arXiv  Detail & Related papers  (2023-05-30T01:26:41Z)
• DiffEdit: Diffusion-based semantic image editing with mask guidance [64.555930158319]
  DiffEdit is a method to take advantage of text-conditioned diffusion models for the task of semantic image editing. Our main contribution is able to automatically generate a mask highlighting regions of the input image that need to be edited.
  arXiv  Detail & Related papers  (2022-10-20T17:16:37Z)
• SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Editing [94.31103255204933]
  We propose a unified model for open-domain image editing focusing on color and tone adjustment of open-domain images. Our model learns a unified editing space that is more semantic, intuitive, and easy to manipulate. We show that by inverting image pairs into latent codes of the learned editing space, our model can be leveraged for various downstream editing tasks.
  arXiv  Detail & Related papers  (2021-11-30T23:53:32Z)
• Look here! A parametric learning based approach to redirect visual attention [49.609412873346386]
  We introduce an automatic method to make an image region more attention-capturing via subtle image edits. Our model predicts a distinct set of global parametric transformations to be applied to the foreground and background image regions. Our edits enable inference at interactive rates on any image size, and easily generalize to videos.
  arXiv  Detail & Related papers  (2020-08-12T16:08:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.