Reduce, Reuse, Recycle: Compositional Generation with Energy-Based
Diffusion Models and MCMC
- URL: http://arxiv.org/abs/2302.11552v4
- Date: Sat, 18 Nov 2023 20:19:04 GMT
- Title: Reduce, Reuse, Recycle: Compositional Generation with Energy-Based
Diffusion Models and MCMC
- Authors: Yilun Du, Conor Durkan, Robin Strudel, Joshua B. Tenenbaum, Sander
Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, Will Grathwohl
- Abstract summary: diffusion models have quickly become the prevailing approach to generative modeling in many domains.
We propose an energy-based parameterization of diffusion models which enables the use of new compositional operators.
We find these samplers lead to notable improvements in compositional generation across a wide set of problems.
- Score: 106.06185677214353
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Since their introduction, diffusion models have quickly become the prevailing
approach to generative modeling in many domains. They can be interpreted as
learning the gradients of a time-varying sequence of log-probability density
functions. This interpretation has motivated classifier-based and
classifier-free guidance as methods for post-hoc control of diffusion models.
In this work, we build upon these ideas using the score-based interpretation of
diffusion models, and explore alternative ways to condition, modify, and reuse
diffusion models for tasks involving compositional generation and guidance. In
particular, we investigate why certain types of composition fail using current
techniques and present a number of solutions. We conclude that the sampler (not
the model) is responsible for this failure and propose new samplers, inspired
by MCMC, which enable successful compositional generation. Further, we propose
an energy-based parameterization of diffusion models which enables the use of
new compositional operators and more sophisticated, Metropolis-corrected
samplers. Intriguingly we find these samplers lead to notable improvements in
compositional generation across a wide set of problems such as
classifier-guided ImageNet modeling and compositional text-to-image generation.
Related papers
- TC-DiffRecon: Texture coordination MRI reconstruction method based on
diffusion model and modified MF-UNet method [2.626378252978696]
We propose a novel diffusion model-based MRI reconstruction method, named TC-DiffRecon, which does not rely on a specific acceleration factor for training.
We also suggest the incorporation of the MF-UNet module, designed to enhance the quality of MRI images generated by the model.
arXiv Detail & Related papers (2024-02-17T13:09:00Z) - Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional
Image Synthesis [62.07413805483241]
Steered Diffusion is a framework for zero-shot conditional image generation using a diffusion model trained for unconditional generation.
We present experiments using steered diffusion on several tasks including inpainting, colorization, text-guided semantic editing, and image super-resolution.
arXiv Detail & Related papers (2023-09-30T02:03:22Z) - Diff-Instruct: A Universal Approach for Transferring Knowledge From
Pre-trained Diffusion Models [77.83923746319498]
We propose a framework called Diff-Instruct to instruct the training of arbitrary generative models.
We show that Diff-Instruct results in state-of-the-art single-step diffusion-based models.
Experiments on refining GAN models show that the Diff-Instruct can consistently improve the pre-trained generators of GAN models.
arXiv Detail & Related papers (2023-05-29T04:22:57Z) - A Reparameterized Discrete Diffusion Model for Text Generation [42.620568198168826]
This work studies discrete diffusion probabilistic models with applications to natural language generation.
We derive an alternative yet equivalent formulation of the sampling from discrete diffusion processes.
We conduct extensive experiments to evaluate the text generation capability of our model, demonstrating significant improvements over existing diffusion models.
arXiv Detail & Related papers (2023-02-11T16:26:57Z) - Unifying Diffusion Models' Latent Space, with Applications to
CycleDiffusion and Guidance [95.12230117950232]
We show that a common latent space emerges from two diffusion models trained independently on related domains.
Applying CycleDiffusion to text-to-image diffusion models, we show that large-scale text-to-image diffusion models can be used as zero-shot image-to-image editors.
arXiv Detail & Related papers (2022-10-11T15:53:52Z) - A Survey on Generative Diffusion Model [75.93774014861978]
Diffusion models are an emerging class of deep generative models.
They have certain limitations, including a time-consuming iterative generation process and confinement to high-dimensional Euclidean space.
This survey presents a plethora of advanced techniques aimed at enhancing diffusion models.
arXiv Detail & Related papers (2022-09-06T16:56:21Z) - Improving the Reconstruction of Disentangled Representation Learners via Multi-Stage Modeling [55.28436972267793]
Current autoencoder-based disentangled representation learning methods achieve disentanglement by penalizing the ( aggregate) posterior to encourage statistical independence of the latent factors.
We present a novel multi-stage modeling approach where the disentangled factors are first learned using a penalty-based disentangled representation learning method.
Then, the low-quality reconstruction is improved with another deep generative model that is trained to model the missing correlated latent variables.
arXiv Detail & Related papers (2020-10-25T18:51:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.