Generative Photomontage

• URL: http://arxiv.org/abs/2408.07116v2
• Date: Fri, 16 Aug 2024 17:59:51 GMT
• Title: Generative Photomontage
• Authors: Sean J. Liu, Nupur Kumari, Ariel Shamir, Jun-Yan Zhu,
• Abstract summary: We propose a framework for creating the desired image by compositing it from various parts of generated images. We let users select desired parts from the generated results using a brush stroke interface. We show compelling results for each application and demonstrate that our method outperforms existing image blending methods.
• Score: 40.49579203394384
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-to-image models are powerful tools for image creation. However, the generation process is akin to a dice roll and makes it difficult to achieve a single image that captures everything a user wants. In this paper, we propose a framework for creating the desired image by compositing it from various parts of generated images, in essence forming a Generative Photomontage. Given a stack of images generated by ControlNet using the same input condition and different seeds, we let users select desired parts from the generated results using a brush stroke interface. We introduce a novel technique that takes in the user's brush strokes, segments the generated images using a graph-based optimization in diffusion feature space, and then composites the segmented regions via a new feature-space blending method. Our method faithfully preserves the user-selected regions while compositing them harmoniously. We demonstrate that our flexible framework can be used for many applications, including generating new appearance combinations, fixing incorrect shapes and artifacts, and improving prompt alignment. We show compelling results for each application and demonstrate that our method outperforms existing image blending methods and various baselines.

Related papers

• Scene Graph Disentanglement and Composition for Generalizable Complex Image Generation [44.457347230146404]
  We leverage the scene graph, a powerful structured representation, for complex image generation. We employ the generative capabilities of variational autoencoders and diffusion models in a generalizable manner. Our method outperforms recent competitors based on text, layout, or scene graph.
  arXiv  Detail & Related papers  (2024-10-01T07:02:46Z)
• Zero-shot Image Editing with Reference Imitation [50.75310094611476]
  We present a new form of editing, termed imitative editing, to help users exercise their creativity more conveniently. We propose a generative training framework, dubbed MimicBrush, which randomly selects two frames from a video clip, masks some regions of one frame, and learns to recover the masked regions using the information from the other frame. We experimentally show the effectiveness of our method under various test cases as well as its superiority over existing alternatives.
  arXiv  Detail & Related papers  (2024-06-11T17:59:51Z)
• Image Collage on Arbitrary Shape via Shape-Aware Slicing and Optimization [6.233023267175408]
  We present a shape slicing algorithm and an optimization scheme that can create image collages of arbitrary shapes. Shape-Aware Slicing, which is designed specifically for irregular shapes, takes human perception and shape structure into account to generate visually pleasing partitions.
  arXiv  Detail & Related papers  (2023-11-17T09:41:30Z)
• Zero-shot spatial layout conditioning for text-to-image diffusion models [52.24744018240424]
  Large-scale text-to-image diffusion models have significantly improved the state of the art in generative image modelling. We consider image generation from text associated with segments on the image canvas, which combines an intuitive natural language interface with precise spatial control over the generated content. We propose ZestGuide, a zero-shot segmentation guidance approach that can be plugged into pre-trained text-to-image diffusion models.
  arXiv  Detail & Related papers  (2023-06-23T19:24:48Z)
• Diffusion Brush: A Latent Diffusion Model-based Editing Tool for AI-generated Images [10.323260768204461]
  Text-to-image generative models have made remarkable advancements in generating high-quality images. Existing techniques to fine-tune generated images are time-consuming (manual editing), produce poorly-integrated results (inpainting), or result in unexpected changes across the entire image. We present Diffusion Brush, a Latent Diffusion Model-based (LDM) tool to efficiently fine-tune desired regions within an AI-synthesized image.
  arXiv  Detail & Related papers  (2023-05-31T22:27:21Z)
• MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing [54.712205852602736]
  We develop MasaCtrl, a tuning-free method to achieve consistent image generation and complex non-rigid image editing simultaneously. Specifically, MasaCtrl converts existing self-attention in diffusion models into mutual self-attention, so that it can query correlated local contents and textures from source images for consistency. Extensive experiments show that the proposed MasaCtrl can produce impressive results in both consistent image generation and complex non-rigid real image editing.
  arXiv  Detail & Related papers  (2023-04-17T17:42:19Z)
• Deep Image Compositing [93.75358242750752]
  We propose a new method which can automatically generate high-quality image composites without any user input. Inspired by Laplacian pyramid blending, a dense-connected multi-stream fusion network is proposed to effectively fuse the information from the foreground and background images. Experiments show that the proposed method can automatically generate high-quality composites and outperforms existing methods both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2020-11-04T06:12:24Z)
• Swapping Autoencoder for Deep Image Manipulation [94.33114146172606]
  We propose the Swapping Autoencoder, a deep model designed specifically for image manipulation. The key idea is to encode an image with two independent components and enforce that any swapped combination maps to a realistic image. Experiments on multiple datasets show that our model produces better results and is substantially more efficient compared to recent generative models.
  arXiv  Detail & Related papers  (2020-07-01T17:59:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.