A Survey on Pre-Trained Diffusion Model Distillations

• URL: http://arxiv.org/abs/2502.08364v1
• Date: Wed, 12 Feb 2025 12:50:24 GMT
• Title: A Survey on Pre-Trained Diffusion Model Distillations
• Authors: Xuhui Fan, Zhangkai Wu, Hongyu Wu,
• Abstract summary: Diffusion Models (DMs) have emerged as the dominant approach in Generative Artificial Intelligence (GenAI)<n>DMs are typically trained on massive datasets and usually require large storage.<n>Distillation methods on pre-trained DM have become widely adopted practices to develop smaller, more efficient models.
• Score: 8.633764273043488
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion Models~(DMs) have emerged as the dominant approach in Generative Artificial Intelligence (GenAI), owing to their remarkable performance in tasks such as text-to-image synthesis. However, practical DMs, such as stable diffusion, are typically trained on massive datasets and thus usually require large storage. At the same time, many steps may be required, i.e., recursively evaluating the trained neural network, to generate a high-quality image, which results in significant computational costs during sample generation. As a result, distillation methods on pre-trained DM have become widely adopted practices to develop smaller, more efficient models capable of rapid, few-step generation in low-resource environment. When these distillation methods are developed from different perspectives, there is an urgent need for a systematic survey, particularly from a methodological perspective. In this survey, we review distillation methods through three aspects: output loss distillation, trajectory distillation and adversarial distillation. We also discuss current challenges and outline future research directions in the conclusion.

Related papers

• Honey, I Shrunk the Language Model: Impact of Knowledge Distillation Methods on Performance and Explainability [3.224880576815583]
  High computational and storage demands of Large Language Models limit their deployment in resource-constrained environments. Previous research has introduced several distillation methods for both generating training data and for training the student model. Despite their relevance, the effects of state-of-the-art distillation methods on model performance and explainability have not been thoroughly investigated.
  arXiv  Detail & Related papers  (2025-04-22T17:32:48Z)
• Denoising Score Distillation: From Noisy Diffusion Pretraining to One-Step High-Quality Generation [82.39763984380625]
  We introduce denoising score distillation (DSD), a surprisingly effective and novel approach for training high-quality generative models from low-quality data. DSD pretrains a diffusion model exclusively on noisy, corrupted samples and then distills it into a one-step generator capable of producing refined, clean outputs.
  arXiv  Detail & Related papers  (2025-03-10T17:44:46Z)
• Relational Diffusion Distillation for Efficient Image Generation [27.127061578093674]
  Diffusion model's high delay hinders its wide application in edge devices with scarce computing resources.<n>We propose Diffusion Distillation (RDD), a novel distillation method tailored specifically for distilling diffusion models.<n>Our proposed RDD leads to 1.47 FID decrease under 1 sampling step compared to state-of-the-art diffusion distillation methods and achieving 256x speed-up.
  arXiv  Detail & Related papers  (2024-10-10T07:40:51Z)
• Optimizing Resource Consumption in Diffusion Models through Hallucination Early Detection [87.22082662250999]
  We introduce HEaD (Hallucination Early Detection), a new paradigm designed to swiftly detect incorrect generations at the beginning of the diffusion process. We demonstrate that using HEaD saves computational resources and accelerates the generation process to get a complete image. Our findings reveal that HEaD can save up to 12% of the generation time on a two objects scenario.
  arXiv  Detail & Related papers  (2024-09-16T18:00:00Z)
• DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture [69.58440626023541]
  Diffusion models (DMs) have demonstrated exceptional generative capabilities across various domains. DMs now consume increasingly large amounts of data. We propose a novel scenario: using existing DMs as data sources to train new DMs with any architecture.
  arXiv  Detail & Related papers  (2024-09-05T14:12:22Z)
• EM Distillation for One-step Diffusion Models [65.57766773137068]
  We propose a maximum likelihood-based approach that distills a diffusion model to a one-step generator model with minimal loss of quality.<n>We develop a reparametrized sampling scheme and a noise cancellation technique that together stabilizes the distillation process.
  arXiv  Detail & Related papers  (2024-05-27T05:55:22Z)
• One-Step Diffusion Distillation via Deep Equilibrium Models [64.11782639697883]
  We introduce a simple yet effective means of distilling diffusion models directly from initial noise to the resulting image. Our method enables fully offline training with just noise/image pairs from the diffusion model. We demonstrate that the DEQ architecture is crucial to this capability, as GET matches a $5times$ larger ViT in terms of FID scores.
  arXiv  Detail & Related papers  (2023-12-12T07:28:40Z)
• BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping [64.54271680071373]
  Diffusion models have demonstrated excellent potential for generating diverse images. Knowledge distillation has been recently proposed as a remedy that can reduce the number of inference steps to one or a few. We present a novel technique called BOOT, that overcomes limitations with an efficient data-free distillation algorithm.
  arXiv  Detail & Related papers  (2023-06-08T20:30:55Z)
• A Comprehensive Survey on Knowledge Distillation of Diffusion Models [0.0]
  Diffusion Models (DMs) utilize neural networks to specify score functions. Our tutorial is intended for individuals with a basic understanding of generative models who wish to apply DM's distillation or embark on a research project in this field.
  arXiv  Detail & Related papers  (2023-04-09T15:49:28Z)
• HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers [49.79405257763856]
  This paper focuses on task-agnostic distillation. It produces a compact pre-trained model that can be easily fine-tuned on various tasks with small computational costs and memory footprints. We propose Homotopic Distillation (HomoDistil), a novel task-agnostic distillation approach equipped with iterative pruning.
  arXiv  Detail & Related papers  (2023-02-19T17:37:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.