MFTF: Mask-free Training-free Object Level Layout Control Diffusion Model

• URL: http://arxiv.org/abs/2412.01284v2
• Date: Wed, 18 Dec 2024 01:56:53 GMT
• Title: MFTF: Mask-free Training-free Object Level Layout Control Diffusion Model
• Authors: Shan Yang,
• Abstract summary: Mask-free Training-free Object-Level Layout Control Diffusion Model (MFTF) MFTF provides precise control over object positions without requiring additional masks or images.
• Score: 11.699591936909325
• License:
• Abstract: Text-to-image generation models have revolutionized content creation, but diffusion-based vision-language models still face challenges in precisely controlling the shape, appearance, and positional placement of objects in generated images using text guidance alone. Existing global image editing models rely on additional masks or images as guidance to achieve layout control, often requiring retraining of the model. While local object-editing models allow modifications to object shapes, they lack the capability to control object positions. To address these limitations, we propose the Mask-free Training-free Object-Level Layout Control Diffusion Model (MFTF), which provides precise control over object positions without requiring additional masks or images. The MFTF model supports both single-object and multi-object positional adjustments, such as translation and rotation, while enabling simultaneous layout control and object semantic editing. The MFTF model employs a parallel denoising process for both the source and target diffusion models. During this process, attention masks are dynamically generated from the cross-attention layers of the source diffusion model and applied to queries from the self-attention layers to isolate objects. These queries, generated in the source diffusion model, are then adjusted according to the layout control parameters and re-injected into the self-attention layers of the target diffusion model. This approach ensures accurate and precise positional control of objects. Project source code available at https://github.com/syang-genai/MFTF.

Related papers

• DICE: Discrete Inversion Enabling Controllable Editing for Multinomial Diffusion and Masked Generative Models [79.0135981840682]
  We introduce DICE (Discrete Inversion for Controllable Editing), the first approach to enable precise inversion for discrete diffusion models. By recording noise sequences and masking patterns during the reverse diffusion process, DICE enables accurate reconstruction and flexible editing of discrete data. Our results show that DICE preserves high data fidelity while enhancing editing capabilities, offering new opportunities for fine-grained content manipulation in discrete spaces.
  arXiv  Detail & Related papers  (2024-10-10T17:59:48Z)
• DiffUHaul: A Training-Free Method for Object Dragging in Images [78.93531472479202]
  We propose a training-free method, dubbed DiffUHaul, for the object dragging task. We first apply attention masking in each denoising step to make the generation more disentangled across different objects. In the early denoising steps, we interpolate the attention features between source and target images to smoothly fuse new layouts with the original appearance.
  arXiv  Detail & Related papers  (2024-06-03T17:59:53Z)
• DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models [66.43179841884098]
  We propose a novel image editing method, DragonDiffusion, enabling Drag-style manipulation on Diffusion models. Our method achieves various editing modes for the generated or real images, such as object moving, object resizing, object appearance replacement, and content dragging.
  arXiv  Detail & Related papers  (2023-07-05T16:43:56Z)
• Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis [59.10787643285506]
  Diffusion-based models have achieved state-of-the-art performance on text-to-image synthesis tasks. One critical limitation of these models is the low fidelity of generated images with respect to the text description. We propose a new text-to-image algorithm that adds explicit control over spatial-temporal cross-attention in diffusion models.
  arXiv  Detail & Related papers  (2023-04-07T23:49:34Z)
• LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation [46.567682868550285]
  We propose a diffusion model named LayoutDiffusion that can obtain higher generation quality and greater controllability than the previous works. In this paper, we propose to construct a structural image patch with region information and transform the patched image into a special layout to fuse with the normal layout in a unified form. Our experiments show that our LayoutDiffusion outperforms the previous SOTA methods on FID, CAS by relatively 46.35%, 26.70% on COCO-stuff and 44.29%, 41.82% on VG Code.
  arXiv  Detail & Related papers  (2023-03-30T06:56:12Z)
• MODIFY: Model-driven Face Stylization without Style Images [77.24793103549158]
  Existing face stylization methods always acquire the presence of the target (style) domain during the translation process. We propose a new method called MODel-drIven Face stYlization (MODIFY), which relies on the generative model to bypass the dependence of the target images. Experimental results on several different datasets validate the effectiveness of MODIFY for unsupervised face stylization.
  arXiv  Detail & Related papers  (2023-03-17T08:35:17Z)
• Collage Diffusion [17.660410448312717]
  Collage Diffusion harmonizes the input layers to make objects fit together. We preserve key visual attributes of input layers by learning specialized text representations per layer. Collage Diffusion generates globally harmonized images that maintain desired object characteristics better than prior approaches.
  arXiv  Detail & Related papers  (2023-03-01T06:35:42Z)
• Shape-Guided Diffusion with Inside-Outside Attention [60.557437251084465]
  We introduce precise object silhouette as a new form of user control in text-to-image diffusion models. Our training-free method uses an Inside-Outside Attention mechanism to apply a shape constraint to the cross- and self-attention maps.
  arXiv  Detail & Related papers  (2022-12-01T01:39:28Z)
• Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis [12.449076001538552]
  This paper focuses on a recent emerged task, layout-to-image, to learn generative models capable of synthesizing photo-realistic images from spatial layout. Style control at the image level is the same as in vanilla GANs, while style control at the object mask level is realized by a proposed novel feature normalization scheme. In experiments, the proposed method is tested in the COCO-Stuff dataset and the Visual Genome dataset with state-of-the-art performance obtained.
  arXiv  Detail & Related papers  (2020-03-25T18:16:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.