Transparent Image Layer Diffusion using Latent Transparency

• URL: http://arxiv.org/abs/2402.17113v4
• Date: Sun, 23 Jun 2024 03:47:27 GMT
• Title: Transparent Image Layer Diffusion using Latent Transparency
• Authors: Lvmin Zhang, Maneesh Agrawala,
• Abstract summary: We present LayerDiffuse, an approach enabling large-scale pretrained latent diffusion models to generate transparent images. The method learns a "latent transparency" that encodes alpha channel transparency into the latent manifold of a pretrained latent diffusion model. It preserves the production-ready quality of the large diffusion model by regulating the added transparency as a latent offset.
• Score: 30.77316047044662
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present LayerDiffuse, an approach enabling large-scale pretrained latent diffusion models to generate transparent images. The method allows generation of single transparent images or of multiple transparent layers. The method learns a "latent transparency" that encodes alpha channel transparency into the latent manifold of a pretrained latent diffusion model. It preserves the production-ready quality of the large diffusion model by regulating the added transparency as a latent offset with minimal changes to the original latent distribution of the pretrained model. In this way, any latent diffusion model can be converted into a transparent image generator by finetuning it with the adjusted latent space. We train the model with 1M transparent image layer pairs collected using a human-in-the-loop collection scheme. We show that latent transparency can be applied to different open source image generators, or be adapted to various conditional control systems to achieve applications like foreground/background-conditioned layer generation, joint layer generation, structural control of layer contents, etc. A user study finds that in most cases (97%) users prefer our natively generated transparent content over previous ad-hoc solutions such as generating and then matting. Users also report the quality of our generated transparent images is comparable to real commercial transparent assets like Adobe Stock.

Related papers

• Trans-Adapter: A Plug-and-Play Framework for Transparent Image Inpainting [60.062438188868306]
  existing image inpainting methods are designed exclusively for RGB images.<n>Trans-Adapter is a plug-and-play adapter that enables diffusion-based inpainting models to process transparent images directly.
  arXiv  Detail & Related papers  (2025-08-01T22:27:21Z)
• PrismLayers: Open Data for High-Quality Multi-Layer Transparent Image Generative Models [25.859278092788237]
  We release the first open, ultra-high-fidelity PrismLayers dataset of 200K (20K) multilayer transparent images with accurate alpha mattes.<n>We also deliver a strong, open-source multi-layer generation model, ART+, which matches the aesthetics of modern text-to-image generation models.
  arXiv  Detail & Related papers  (2025-05-28T16:09:33Z)
• ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation [108.69315278353932]
  We introduce the Anonymous Region Transformer (ART), which facilitates the direct generation of variable multi-layer transparent images. By enabling precise control and scalable layer generation, ART establishes a new paradigm for interactive content creation.
  arXiv  Detail & Related papers  (2025-02-25T16:57:04Z)
• ILDiff: Generate Transparent Animated Stickers by Implicit Layout Distillation [17.449524943353904]
  High-quality animated stickers usually contain transparent channels, which are often ignored by current video generation models. To generate fine-grained animated transparency channels, existing methods can be roughly divided into video matting algorithms and diffusion-based algorithms. In this paper, we propose an ILDiff method to generate animated transparent channels through implicit layout distillation.
  arXiv  Detail & Related papers  (2024-12-30T12:27:35Z)
• LayerFusion: Harmonized Multi-Layer Text-to-Image Generation with Generative Priors [38.47462111828742]
  Layered content generation is crucial for creative fields like graphic design, animation, and digital art. We propose a novel image generation pipeline based on Latent Diffusion Models (LDMs) that generates images with two layers. We show significant improvements in visual coherence, image quality, and layer consistency compared to baseline methods.
  arXiv  Detail & Related papers  (2024-12-05T18:59:18Z)
• Stable Flow: Vital Layers for Training-Free Image Editing [74.52248787189302]
  Diffusion models have revolutionized the field of content synthesis and editing. Recent models have replaced the traditional UNet architecture with the Diffusion Transformer (DiT) We propose an automatic method to identify "vital layers" within DiT, crucial for image formation. Next, to enable real-image editing, we introduce an improved image inversion method for flow models.
  arXiv  Detail & Related papers  (2024-11-21T18:59:51Z)
• Merging and Splitting Diffusion Paths for Semantically Coherent Panoramas [33.334956022229846]
  We propose the Merge-Attend-Diffuse operator, which can be plugged into different types of pretrained diffusion models used in a joint diffusion setting. Specifically, we merge the diffusion paths, reprogramming self- and cross-attention to operate on the aggregated latent space. Our method maintains compatibility with the input prompt and visual quality of the generated images while increasing their semantic coherence.
  arXiv  Detail & Related papers  (2024-08-28T09:22:32Z)
• Zippo: Zipping Color and Transparency Distributions into a Single Diffusion Model [56.84435912555532]
  We present Zippo, a unified framework for zipping the color and transparency distributions into a single diffusion model. Zippo is capable of generating RGB images from alpha mattes and predicting transparency from input images. Our experiments showcase Zippo's ability of efficient text-conditioned transparent image generation.
  arXiv  Detail & Related papers  (2024-03-17T04:02:39Z)
• ControlMat: A Controlled Generative Approach to Material Capture [7.641962898125423]
  Material reconstruction from a photograph is a key component of 3D content creation democratization. We present ControlMat, a method which, given a single photograph with uncontrolled illumination as input, conditions a diffusion model to generate plausible, tileable, high-resolution physically-based digital materials.
  arXiv  Detail & Related papers  (2023-09-04T16:18:49Z)
• DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing [94.24479528298252]
  DragGAN is an interactive point-based image editing framework that achieves impressive editing results with pixel-level precision. By harnessing large-scale pretrained diffusion models, we greatly enhance the applicability of interactive point-based editing on both real and diffusion-generated images. We present a challenging benchmark dataset called DragBench to evaluate the performance of interactive point-based image editing methods.
  arXiv  Detail & Related papers  (2023-06-26T06:04:09Z)
• Real-World Image Variation by Aligning Diffusion Inversion Chain [53.772004619296794]
  A domain gap exists between generated images and real-world images, which poses a challenge in generating high-quality variations of real-world images. We propose a novel inference pipeline called Real-world Image Variation by ALignment (RIVAL) Our pipeline enhances the generation quality of image variations by aligning the image generation process to the source image's inversion chain.
  arXiv  Detail & Related papers  (2023-05-30T04:09:47Z)
• Controlled and Conditional Text to Image Generation with Diffusion Prior [1.8690858882873838]
  DALLE-2's two step process comprises a Diffusion Prior that generates a CLIP image embedding from text and a Diffusion Decoder that generates an image from a CLIP image embedding. We show quantitatively and qualitatively that the proposed approaches perform better than prompt engineering for domain specific generation and existing baselines for color conditioned generation.
  arXiv  Detail & Related papers  (2023-02-23T00:10:40Z)
• MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation [34.61940502872307]
  MultiDiffusion is a unified framework that enables versatile and controllable image generation. We show that MultiDiffusion can be readily applied to generate high quality and diverse images.
  arXiv  Detail & Related papers  (2023-02-16T06:28:29Z)
• Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models [60.63556257324894]
  A key desired property of image generative models is the ability to disentangle different attributes. We propose a simple, light-weight image editing algorithm where the mixing weights of the two text embeddings are optimized for style matching and content preservation. Experiments show that the proposed method can modify a wide range of attributes, with the performance outperforming diffusion-model-based image-editing algorithms.
  arXiv  Detail & Related papers  (2022-12-16T19:58:52Z)
• SinDiffusion: Learning a Diffusion Model from a Single Natural Image [159.4285444680301]
  We present SinDiffusion, leveraging denoising diffusion models to capture internal distribution of patches from a single natural image. It is based on two core designs. First, SinDiffusion is trained with a single model at a single scale instead of multiple models with progressive growing of scales. Second, we identify that a patch-level receptive field of the diffusion network is crucial and effective for capturing the image's patch statistics.
  arXiv  Detail & Related papers  (2022-11-22T18:00:03Z)
• Semantic Image Synthesis via Diffusion Models [174.24523061460704]
  Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in various image generation tasks. Recent work on semantic image synthesis mainly follows the de facto GAN-based approaches. We propose a novel framework based on DDPM for semantic image synthesis.
  arXiv  Detail & Related papers  (2022-06-30T18:31:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.