A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models

• URL: http://arxiv.org/abs/2408.05927v1
• Date: Mon, 12 Aug 2024 05:33:45 GMT
• Title: A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models
• Authors: Taehong Moon, Moonseok Choi, EungGu Yun, Jongmin Yoon, Gayoung Lee, Jaewoong Cho, Juho Lee,
• Abstract summary: A practical bottleneck of diffusion models is their sampling speed. We propose a novel framework capable of adaptively allocating compute required for the score estimation. We show that our method could significantly improve the sampling throughput of the diffusion models without compromising image quality.
• Score: 14.859580045688487
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion models have shown remarkable performance in generation problems over various domains including images, videos, text, and audio. A practical bottleneck of diffusion models is their sampling speed, due to the repeated evaluation of score estimation networks during the inference. In this work, we propose a novel framework capable of adaptively allocating compute required for the score estimation, thereby reducing the overall sampling time of diffusion models. We observe that the amount of computation required for the score estimation may vary along the time step for which the score is estimated. Based on this observation, we propose an early-exiting scheme, where we skip the subset of parameters in the score estimation network during the inference, based on a time-dependent exit schedule. Using the diffusion models for image synthesis, we show that our method could significantly improve the sampling throughput of the diffusion models without compromising image quality. Furthermore, we also demonstrate that our method seamlessly integrates with various types of solvers for faster sampling, capitalizing on their compatibility to enhance overall efficiency. The source code and our experiments are available at \url{https://github.com/taehong-moon/ee-diffusion}

Related papers

• Efficient Denoising using Score Embedding in Score-based Diffusion Models [0.24578723416255752]
  We propose to increase the efficiency in training score-based diffusion models. We accomplish this by solving the log-density Fokker-Planck (FP) Equation numerically. The pre-computed score is embedded into the image to encourage faster training under slice Wasserstein distance.
  arXiv  Detail & Related papers  (2024-04-10T00:05:55Z)
• Diffusion-TS: Interpretable Diffusion for General Time Series Generation [6.639630994040322]
  Diffusion-TS is a novel diffusion-based framework that generates time series samples of high quality. We train the model to directly reconstruct the sample instead of the noise in each diffusion step, combining a Fourier-based loss term. Results show that Diffusion-TS achieves the state-of-the-art results on various realistic analyses of time series.
  arXiv  Detail & Related papers  (2024-03-04T05:39:23Z)
• Improved off-policy training of diffusion samplers [93.66433483772055]
  We study the problem of training diffusion models to sample from a distribution with an unnormalized density or energy function. We benchmark several diffusion-structured inference methods, including simulation-based variational approaches and off-policy methods. Our results shed light on the relative advantages of existing algorithms while bringing into question some claims from past work.
  arXiv  Detail & Related papers  (2024-02-07T18:51:49Z)
• Reflected Diffusion Models [93.26107023470979]
  We present Reflected Diffusion Models, which reverse a reflected differential equation evolving on the support of the data. Our approach learns the score function through a generalized score matching loss and extends key components of standard diffusion models.
  arXiv  Detail & Related papers  (2023-04-10T17:54:38Z)
• Fast Inference in Denoising Diffusion Models via MMD Finetuning [23.779985842891705]
  We present MMD-DDM, a novel method for fast sampling of diffusion models. Our approach is based on the idea of using the Maximum Mean Discrepancy (MMD) to finetune the learned distribution with a given budget of timesteps. Our findings show that the proposed method is able to produce high-quality samples in a fraction of the time required by widely-used diffusion models.
  arXiv  Detail & Related papers  (2023-01-19T09:48:07Z)
• Score-based Continuous-time Discrete Diffusion Models [102.65769839899315]
  We extend diffusion models to discrete variables by introducing a Markov jump process where the reverse process denoises via a continuous-time Markov chain. We show that an unbiased estimator can be obtained via simple matching the conditional marginal distributions. We demonstrate the effectiveness of the proposed method on a set of synthetic and real-world music and image benchmarks.
  arXiv  Detail & Related papers  (2022-11-30T05:33:29Z)
• Fast Sampling of Diffusion Models via Operator Learning [74.37531458470086]
  We use neural operators, an efficient method to solve the probability flow differential equations, to accelerate the sampling process of diffusion models. Compared to other fast sampling methods that have a sequential nature, we are the first to propose a parallel decoding method. We show our method achieves state-of-the-art FID of 3.78 for CIFAR-10 and 7.83 for ImageNet-64 in the one-model-evaluation setting.
  arXiv  Detail & Related papers  (2022-11-24T07:30:27Z)
• Markup-to-Image Diffusion Models with Scheduled Sampling [111.30188533324954]
  Building on recent advances in image generation, we present a data-driven approach to rendering markup into images. The approach is based on diffusion models, which parameterize the distribution of data using a sequence of denoising operations. We conduct experiments on four markup datasets: mathematical formulas (La), table layouts (HTML), sheet music (LilyPond), and molecular images (SMILES)
  arXiv  Detail & Related papers  (2022-10-11T04:56:12Z)
• How Much is Enough? A Study on Diffusion Times in Score-based Generative Models [76.76860707897413]
  Current best practice advocates for a large T to ensure that the forward dynamics brings the diffusion sufficiently close to a known and simple noise distribution. We show how an auxiliary model can be used to bridge the gap between the ideal and the simulated forward dynamics, followed by a standard reverse diffusion process.
  arXiv  Detail & Related papers  (2022-06-10T15:09:46Z)
• Score-based diffusion models for accelerated MRI [35.3148116010546]
  We introduce a way to sample data from a conditional distribution given the measurements, such that the model can be readily used for solving inverse problems in imaging. Our model requires magnitude images only for training, and yet is able to reconstruct complex-valued data, and even extends to parallel imaging.
  arXiv  Detail & Related papers  (2021-10-08T08:42:03Z)
• Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting [4.1573460459258245]
  We use diffusion probabilistic models, a class of latent variable models closely connected to score matching and energy-based methods. Our model learns gradients by optimizing a variational bound on the data likelihood and at inference time converts white noise into a sample of the distribution of interest.
  arXiv  Detail & Related papers  (2021-01-28T15:46:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.