Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality

• URL: http://arxiv.org/abs/2202.05830v1
• Date: Fri, 11 Feb 2022 18:53:18 GMT
• Title: Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality
• Authors: Daniel Watson, William Chan, Jonathan Ho, Mohammad Norouzi
• Abstract summary: We introduce Differentiable Diffusion Sampler Search (DDSS), a method that optimize fast samplers for any pre-trained diffusion model. We also present Generalized Gaussian Diffusion Models (GGDM), a family of flexible non-Markovian samplers for diffusion models. Our method is compatible with any pre-trained diffusion model without fine-tuning or re-training required.
• Score: 44.37533757879762
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion models have emerged as an expressive family of generative models rivaling GANs in sample quality and autoregressive models in likelihood scores. Standard diffusion models typically require hundreds of forward passes through the model to generate a single high-fidelity sample. We introduce Differentiable Diffusion Sampler Search (DDSS): a method that optimizes fast samplers for any pre-trained diffusion model by differentiating through sample quality scores. We also present Generalized Gaussian Diffusion Models (GGDM), a family of flexible non-Markovian samplers for diffusion models. We show that optimizing the degrees of freedom of GGDM samplers by maximizing sample quality scores via gradient descent leads to improved sample quality. Our optimization procedure backpropagates through the sampling process using the reparametrization trick and gradient rematerialization. DDSS achieves strong results on unconditional image generation across various datasets (e.g., FID scores on LSUN church 128x128 of 11.6 with only 10 inference steps, and 4.82 with 20 steps, compared to 51.1 and 14.9 with strongest DDPM/DDIM baselines). Our method is compatible with any pre-trained diffusion model without fine-tuning or re-training required.

Related papers

• Provable Statistical Rates for Consistency Diffusion Models [87.28777947976573]
  Despite the state-of-the-art performance, diffusion models are known for their slow sample generation due to the extensive number of steps involved. This paper contributes towards the first statistical theory for consistency models, formulating their training as a distribution discrepancy minimization problem.
  arXiv  Detail & Related papers  (2024-06-23T20:34:18Z)
• Diffusion Rejection Sampling [13.945372555871414]
  Diffusion Rejection Sampling (DiffRS) is a rejection sampling scheme that aligns the sampling transition kernels with the true ones at each timestep. The proposed method can be viewed as a mechanism that evaluates the quality of samples at each intermediate timestep and refines them with varying effort depending on the sample. Empirical results demonstrate the state-of-the-art performance of DiffRS on the benchmark datasets and the effectiveness of DiffRS for fast diffusion samplers and large-scale text-to-image diffusion models.
  arXiv  Detail & Related papers  (2024-05-28T07:00:28Z)
• Boosting Diffusion Models with Moving Average Sampling in Frequency Domain [101.43824674873508]
  Diffusion models rely on the current sample to denoise the next one, possibly resulting in denoising instability. In this paper, we reinterpret the iterative denoising process as model optimization and leverage a moving average mechanism to ensemble all the prior samples. We name the complete approach "Moving Average Sampling in Frequency domain (MASF)"
  arXiv  Detail & Related papers  (2024-03-26T16:57:55Z)
• Solving Diffusion ODEs with Optimal Boundary Conditions for Better Image Super-Resolution [82.50210340928173]
  randomness of diffusion models results in ineffectiveness and instability, making it challenging for users to guarantee the quality of SR results. We propose a plug-and-play sampling method that owns the potential to benefit a series of diffusion-based SR methods. The quality of SR results sampled by the proposed method with fewer steps outperforms the quality of results sampled by current methods with randomness from the same pre-trained diffusion-based SR model.
  arXiv  Detail & Related papers  (2023-05-24T17:09:54Z)
• Fast Inference in Denoising Diffusion Models via MMD Finetuning [23.779985842891705]
  We present MMD-DDM, a novel method for fast sampling of diffusion models. Our approach is based on the idea of using the Maximum Mean Discrepancy (MMD) to finetune the learned distribution with a given budget of timesteps. Our findings show that the proposed method is able to produce high-quality samples in a fraction of the time required by widely-used diffusion models.
  arXiv  Detail & Related papers  (2023-01-19T09:48:07Z)
• On Distillation of Guided Diffusion Models [94.95228078141626]
  We propose an approach to distilling classifier-free guided diffusion models into models that are fast to sample from. For standard diffusion models trained on the pixelspace, our approach is able to generate images visually comparable to that of the original model. For diffusion models trained on the latent-space (e.g., Stable Diffusion), our approach is able to generate high-fidelity images using as few as 1 to 4 denoising steps.
  arXiv  Detail & Related papers  (2022-10-06T18:03:56Z)
• Improved Denoising Diffusion Probabilistic Models [4.919647298882951]
  We show that DDPMs can achieve competitive log-likelihoods while maintaining high sample quality. We also find that learning variances of the reverse diffusion process allows sampling with an order of magnitude fewer forward passes. We show that the sample quality and likelihood of these models scale smoothly with model capacity and training compute, making them easily scalable.
  arXiv  Detail & Related papers  (2021-02-18T23:44:17Z)
• Denoising Diffusion Implicit Models [117.03720513930335]
  We present denoising diffusion implicit models (DDIMs) for iterative implicit probabilistic models with the same training procedure as DDPMs. DDIMs can produce high quality samples $10 times$ to $50 times$ faster in terms of wall-clock time compared to DDPMs.
  arXiv  Detail & Related papers  (2020-10-06T06:15:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.