Large-scale Reinforcement Learning for Diffusion Models
- URL: http://arxiv.org/abs/2401.12244v1
- Date: Sat, 20 Jan 2024 08:10:43 GMT
- Title: Large-scale Reinforcement Learning for Diffusion Models
- Authors: Yinan Zhang, Eric Tzeng, Yilun Du, Dmitry Kislyuk
- Abstract summary: Text-to-image diffusion models are susceptible to implicit biases that arise from web-scale text-image training pairs.
We present an effective scalable algorithm to improve diffusion models using Reinforcement Learning (RL)
We show how our approach substantially outperforms existing methods for aligning diffusion models with human preferences.
- Score: 30.164571425479824
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Text-to-image diffusion models are a class of deep generative models that
have demonstrated an impressive capacity for high-quality image generation.
However, these models are susceptible to implicit biases that arise from
web-scale text-image training pairs and may inaccurately model aspects of
images we care about. This can result in suboptimal samples, model bias, and
images that do not align with human ethics and preferences. In this paper, we
present an effective scalable algorithm to improve diffusion models using
Reinforcement Learning (RL) across a diverse set of reward functions, such as
human preference, compositionality, and fairness over millions of images. We
illustrate how our approach substantially outperforms existing methods for
aligning diffusion models with human preferences. We further illustrate how
this substantially improves pretrained Stable Diffusion (SD) models, generating
samples that are preferred by humans 80.3% of the time over those from the base
SD model while simultaneously improving both the composition and diversity of
generated samples.
Related papers
- Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Models [20.70550870149442]
We introduce Annealed Importance Guidance (AIG), an inference-time regularization inspired by Annealed Importance Sampling.
Our experiments demonstrate the benefits of AIG for Stable Diffusion models, striking the optimal balance between reward optimization and image diversity.
arXiv Detail & Related papers (2024-09-09T16:27:26Z) - YaART: Yet Another ART Rendering Technology [119.09155882164573]
This study introduces YaART, a novel production-grade text-to-image cascaded diffusion model aligned to human preferences.
We analyze how these choices affect both the efficiency of the training process and the quality of the generated images.
We demonstrate that models trained on smaller datasets of higher-quality images can successfully compete with those trained on larger datasets.
arXiv Detail & Related papers (2024-04-08T16:51:19Z) - Training Class-Imbalanced Diffusion Model Via Overlap Optimization [55.96820607533968]
Diffusion models trained on real-world datasets often yield inferior fidelity for tail classes.
Deep generative models, including diffusion models, are biased towards classes with abundant training images.
We propose a method based on contrastive learning to minimize the overlap between distributions of synthetic images for different classes.
arXiv Detail & Related papers (2024-02-16T16:47:21Z) - Conditional Image Generation with Pretrained Generative Model [1.4685355149711303]
diffusion models have gained popularity for their ability to generate higher-quality images in comparison to GAN models.
These models require a huge amount of data, computational resources, and meticulous tuning for successful training.
We propose methods to leverage pre-trained unconditional diffusion models with additional guidance for the purpose of conditional image generative.
arXiv Detail & Related papers (2023-12-20T18:27:53Z) - Enhancing Diffusion Models with 3D Perspective Geometry Constraints [10.21800236402905]
We introduce a novel geometric constraint in the training process of generative models to enforce perspective accuracy.
We show that outputs of models trained with this constraint both appear more realistic and improve performance of downstream models trained on generated images.
arXiv Detail & Related papers (2023-12-01T21:56:43Z) - Training Diffusion Models with Reinforcement Learning [82.29328477109826]
Diffusion models are trained with an approximation to the log-likelihood objective.
In this paper, we investigate reinforcement learning methods for directly optimizing diffusion models for downstream objectives.
We describe how posing denoising as a multi-step decision-making problem enables a class of policy gradient algorithms.
arXiv Detail & Related papers (2023-05-22T17:57:41Z) - Your Diffusion Model is Secretly a Zero-Shot Classifier [90.40799216880342]
We show that density estimates from large-scale text-to-image diffusion models can be leveraged to perform zero-shot classification.
Our generative approach to classification attains strong results on a variety of benchmarks.
Our results are a step toward using generative over discriminative models for downstream tasks.
arXiv Detail & Related papers (2023-03-28T17:59:56Z) - Extracting Training Data from Diffusion Models [77.11719063152027]
We show that diffusion models memorize individual images from their training data and emit them at generation time.
With a generate-and-filter pipeline, we extract over a thousand training examples from state-of-the-art models.
We train hundreds of diffusion models in various settings to analyze how different modeling and data decisions affect privacy.
arXiv Detail & Related papers (2023-01-30T18:53:09Z) - On Distillation of Guided Diffusion Models [94.95228078141626]
We propose an approach to distilling classifier-free guided diffusion models into models that are fast to sample from.
For standard diffusion models trained on the pixelspace, our approach is able to generate images visually comparable to that of the original model.
For diffusion models trained on the latent-space (e.g., Stable Diffusion), our approach is able to generate high-fidelity images using as few as 1 to 4 denoising steps.
arXiv Detail & Related papers (2022-10-06T18:03:56Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.