Simple and Fast Distillation of Diffusion Models

• URL: http://arxiv.org/abs/2409.19681v1
• Date: Sun, 29 Sep 2024 12:13:06 GMT
• Title: Simple and Fast Distillation of Diffusion Models
• Authors: Zhenyu Zhou, Defang Chen, Can Wang, Chun Chen, Siwei Lyu,
• Abstract summary: We propose Simple and Fast Distillation (SFD) of diffusion models, which simplifies the paradigm used in existing methods. SFD achieves 4.53 FID (NFE=2) on CIFAR-10 with only 0.64 hours of fine-tuning on a single NVIDIA A100 GPU.
• Score: 39.79747569096888
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion-based generative models have demonstrated their powerful performance across various tasks, but this comes at a cost of the slow sampling speed. To achieve both efficient and high-quality synthesis, various distillation-based accelerated sampling methods have been developed recently. However, they generally require time-consuming fine tuning with elaborate designs to achieve satisfactory performance in a specific number of function evaluation (NFE), making them difficult to employ in practice. To address this issue, we propose Simple and Fast Distillation (SFD) of diffusion models, which simplifies the paradigm used in existing methods and largely shortens their fine-tuning time up to 1000$\times$. We begin with a vanilla distillation-based sampling method and boost its performance to state of the art by identifying and addressing several small yet vital factors affecting the synthesis efficiency and quality. Our method can also achieve sampling with variable NFEs using a single distilled model. Extensive experiments demonstrate that SFD strikes a good balance between the sample quality and fine-tuning costs in few-step image generation task. For example, SFD achieves 4.53 FID (NFE=2) on CIFAR-10 with only 0.64 hours of fine-tuning on a single NVIDIA A100 GPU. Our code is available at https://github.com/zju-pi/diff-sampler.

Related papers

• One-Step Diffusion Distillation via Deep Equilibrium Models [64.11782639697883]
  We introduce a simple yet effective means of distilling diffusion models directly from initial noise to the resulting image. Our method enables fully offline training with just noise/image pairs from the diffusion model. We demonstrate that the DEQ architecture is crucial to this capability, as GET matches a $5times$ larger ViT in terms of FID scores.
  arXiv  Detail & Related papers  (2023-12-12T07:28:40Z)
• Sampler Scheduler for Diffusion Models [0.0]
  Diffusion modeling (DM) has high-quality generative performance. Currently, there is a contradiction in samplers for diffusion-based generative models. We propose the feasibility of using different samplers (ODE/SDE) on different sampling steps of the same sampling process.
  arXiv  Detail & Related papers  (2023-11-12T13:35:25Z)
• Parallel Sampling of Diffusion Models [76.3124029406809]
  Diffusion models are powerful generative models but suffer from slow sampling. We present ParaDiGMS, a novel method to accelerate the sampling of pretrained diffusion models by denoising multiple steps in parallel.
  arXiv  Detail & Related papers  (2023-05-25T17:59:42Z)
• Accelerating Diffusion Sampling with Classifier-based Feature Distillation [20.704675568555082]
  Progressive distillation is proposed for fast sampling by progressively aligning output images of $N$-step teacher sampler with $N/2$-step student sampler. We distill teacher's sharpened feature distribution into the student with a dataset-independent classifier to improve performance. Experiments on CIFAR-10 show the superiority of our method in achieving high quality and fast sampling.
  arXiv  Detail & Related papers  (2022-11-22T06:21:31Z)
• On Distillation of Guided Diffusion Models [94.95228078141626]
  We propose an approach to distilling classifier-free guided diffusion models into models that are fast to sample from. For standard diffusion models trained on the pixelspace, our approach is able to generate images visually comparable to that of the original model. For diffusion models trained on the latent-space (e.g., Stable Diffusion), our approach is able to generate high-fidelity images using as few as 1 to 4 denoising steps.
  arXiv  Detail & Related papers  (2022-10-06T18:03:56Z)
• ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech [63.780196620966905]
  We propose ProDiff, on progressive fast diffusion model for high-quality text-to-speech. ProDiff parameterizes the denoising model by directly predicting clean data to avoid distinct quality degradation in accelerating sampling. Our evaluation demonstrates that ProDiff needs only 2 iterations to synthesize high-fidelity mel-spectrograms. ProDiff enables a sampling speed of 24x faster than real-time on a single NVIDIA 2080Ti GPU.
  arXiv  Detail & Related papers  (2022-07-13T17:45:43Z)
• Fast Sampling of Diffusion Models with Exponential Integrator [9.467521554542273]
  We propose a fast sampling method for DMs with much less number of steps while retaining high sample quality. The proposed method can be applied to any DMs and can generate high-fidelity samples in as few as 10 steps.
  arXiv  Detail & Related papers  (2022-04-29T06:32:38Z)
• Pseudo Numerical Methods for Diffusion Models on Manifolds [77.40343577960712]
  Denoising Diffusion Probabilistic Models (DDPMs) can generate high-quality samples such as image and audio samples. DDPMs require hundreds to thousands of iterations to produce final samples. We propose pseudo numerical methods for diffusion models (PNDMs) PNDMs can generate higher quality synthetic images with only 50 steps compared with 1000-step DDIMs (20x speedup)
  arXiv  Detail & Related papers  (2022-02-20T10:37:52Z)
• Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality [44.37533757879762]
  We introduce Differentiable Diffusion Sampler Search (DDSS), a method that optimize fast samplers for any pre-trained diffusion model. We also present Generalized Gaussian Diffusion Models (GGDM), a family of flexible non-Markovian samplers for diffusion models. Our method is compatible with any pre-trained diffusion model without fine-tuning or re-training required.
  arXiv  Detail & Related papers  (2022-02-11T18:53:18Z)
• Progressive Distillation for Fast Sampling of Diffusion Models [17.355749359987648]
  We present a method to distill a trained deterministic diffusion sampler, using many steps, into a new diffusion model that takes half as many sampling steps. On standard image generation benchmarks like CIFAR-10, ImageNet, and LSUN, we start out with state-of-the-art samplers taking as many as 8192 steps, and are able to distill down to models taking as few as 4 steps without losing much perceptual quality.
  arXiv  Detail & Related papers  (2022-02-01T16:07:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.