Continuous Layout Editing of Single Images with Diffusion Models

• URL: http://arxiv.org/abs/2306.13078v1
• Date: Thu, 22 Jun 2023 17:51:05 GMT
• Title: Continuous Layout Editing of Single Images with Diffusion Models
• Authors: Zhiyuan Zhang, Zhitong Huang, Jing Liao
• Abstract summary: We propose the first framework for layout editing of a single image while preserving its visual properties. Our approach is achieved through two key modules. Our code will be freely available for public use upon acceptance.
• Score: 24.581184791106562
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advancements in large-scale text-to-image diffusion models have enabled many applications in image editing. However, none of these methods have been able to edit the layout of single existing images. To address this gap, we propose the first framework for layout editing of a single image while preserving its visual properties, thus allowing for continuous editing on a single image. Our approach is achieved through two key modules. First, to preserve the characteristics of multiple objects within an image, we disentangle the concepts of different objects and embed them into separate textual tokens using a novel method called masked textual inversion. Next, we propose a training-free optimization method to perform layout control for a pre-trained diffusion model, which allows us to regenerate images with learned concepts and align them with user-specified layouts. As the first framework to edit the layout of existing images, we demonstrate that our method is effective and outperforms other baselines that were modified to support this task. Our code will be freely available for public use upon acceptance.

Related papers

• A Survey of Multimodal-Guided Image Editing with Text-to-Image Diffusion Models [117.77807994397784]
  Image editing aims to edit the given synthetic or real image to meet the specific requirements from users. Recent significant advancement in this field is based on the development of text-to-image (T2I) diffusion models. T2I-based image editing methods significantly enhance editing performance and offer a user-friendly interface for modifying content guided by multimodal inputs.
  arXiv  Detail & Related papers  (2024-06-20T17:58:52Z)
• Zero-shot Image Editing with Reference Imitation [50.75310094611476]
  We present a new form of editing, termed imitative editing, to help users exercise their creativity more conveniently. We propose a generative training framework, dubbed MimicBrush, which randomly selects two frames from a video clip, masks some regions of one frame, and learns to recover the masked regions using the information from the other frame. We experimentally show the effectiveness of our method under various test cases as well as its superiority over existing alternatives.
  arXiv  Detail & Related papers  (2024-06-11T17:59:51Z)
• Unified Editing of Panorama, 3D Scenes, and Videos Through Disentangled Self-Attention Injection [60.47731445033151]
  We propose a novel unified editing framework that combines the strengths of both approaches by utilizing only a basic 2D image text-to-image (T2I) diffusion model. Experimental results confirm that our method enables editing across diverse modalities including 3D scenes, videos, and panorama images.
  arXiv  Detail & Related papers  (2024-05-27T04:44:36Z)
• Tuning-Free Image Customization with Image and Text Guidance [65.9504243633169]
  We introduce a tuning-free framework for simultaneous text-image-guided image customization. Our approach preserves the semantic features of the reference image subject while allowing modification of detailed attributes based on text descriptions. Our approach outperforms previous methods in both human and quantitative evaluations.
  arXiv  Detail & Related papers  (2024-03-19T11:48:35Z)
• LayerDiffusion: Layered Controlled Image Editing with Diffusion Models [5.58892860792971]
  LayerDiffusion is a semantic-based layered controlled image editing method. We leverage a large-scale text-to-image model and employ a layered controlled optimization strategy. Experimental results demonstrate the effectiveness of our method in generating highly coherent images.
  arXiv  Detail & Related papers  (2023-05-30T01:26:41Z)
• Break-A-Scene: Extracting Multiple Concepts from a Single Image [80.47666266017207]
  We introduce the task of textual scene decomposition. We propose augmenting the input image with masks that indicate the presence of target concepts. We then present a novel two-phase customization process.
  arXiv  Detail & Related papers  (2023-05-25T17:59:04Z)
• Prompt Tuning Inversion for Text-Driven Image Editing Using Diffusion Models [6.34777393532937]
  We propose an accurate and quick inversion technique, Prompt Tuning Inversion, for text-driven image editing. Our proposed editing method consists of a reconstruction stage and an editing stage. Experiments on ImageNet demonstrate the superior editing performance of our method compared to the state-of-the-art baselines.
  arXiv  Detail & Related papers  (2023-05-08T03:34:33Z)
• Zero-shot Image-to-Image Translation [57.46189236379433]
  We propose pix2pix-zero, an image-to-image translation method that can preserve the original image without manual prompting. We propose cross-attention guidance, which aims to retain the cross-attention maps of the input image throughout the diffusion process. Our method does not need additional training for these edits and can directly use the existing text-to-image diffusion model.
  arXiv  Detail & Related papers  (2023-02-06T18:59:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.