Enhanced Distribution Alignment for Post-Training Quantization of
Diffusion Models
- URL: http://arxiv.org/abs/2401.04585v1
- Date: Tue, 9 Jan 2024 14:42:49 GMT
- Title: Enhanced Distribution Alignment for Post-Training Quantization of
Diffusion Models
- Authors: Xuewen Liu, Zhikai Li, Junrui Xiao, Qingyi Gu
- Abstract summary: Quantization can effectively reduce model complexity, and post-training quantization is highly promising in accelerating the denoising process.
Existing PTQ methods for diffusion models suffer from distribution mismatch issues at both calibration sample level and reconstruction output level.
We propose Enhanced Distribution Alignment for Post-Training Quantization of Diffusion Models (EDA-DM) to address the above issues.
EDA-DM outperforms the existing post-training quantization frameworks in both unconditional and conditional generation scenarios.
- Score: 4.601488148143309
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Diffusion models have achieved great success in image generation tasks
through iterative noise estimation. However, the heavy denoising process and
complex neural networks hinder their low-latency applications in real-world
scenarios. Quantization can effectively reduce model complexity, and
post-training quantization (PTQ), which does not require fine-tuning, is highly
promising in accelerating the denoising process. Unfortunately, we find that
due to the highly dynamic distribution of activations in different denoising
steps, existing PTQ methods for diffusion models suffer from distribution
mismatch issues at both calibration sample level and reconstruction output
level, which makes the performance far from satisfactory, especially in low-bit
cases. In this paper, we propose Enhanced Distribution Alignment for
Post-Training Quantization of Diffusion Models (EDA-DM) to address the above
issues. Specifically, at the calibration sample level, we select calibration
samples based on the density and diversity in the latent space, thus
facilitating the alignment of their distribution with the overall samples; and
at the reconstruction output level, we propose Fine-grained Block
Reconstruction, which can align the outputs of the quantized model and the
full-precision model at different network granularity. Extensive experiments
demonstrate that EDA-DM outperforms the existing post-training quantization
frameworks in both unconditional and conditional generation scenarios. At
low-bit precision, the quantized models with our method even outperform the
full-precision models on most datasets.
Related papers
- Timestep-Aware Correction for Quantized Diffusion Models [28.265582848911574]
We propose a timestep-aware correction method for quantized diffusion model, which dynamically corrects the quantization error.
By leveraging the proposed method in low-precision diffusion models, substantial enhancement of output quality could be achieved with only negligible overhead.
arXiv Detail & Related papers (2024-07-04T13:22:31Z) - Provable Statistical Rates for Consistency Diffusion Models [87.28777947976573]
Despite the state-of-the-art performance, diffusion models are known for their slow sample generation due to the extensive number of steps involved.
This paper contributes towards the first statistical theory for consistency models, formulating their training as a distribution discrepancy minimization problem.
arXiv Detail & Related papers (2024-06-23T20:34:18Z) - Physics-Informed Diffusion Models [0.0]
We present a framework to inform denoising diffusion models of underlying constraints on generated samples during model training.
Our approach improves the alignment of the generated samples with the imposed constraints and significantly outperforms existing methods.
arXiv Detail & Related papers (2024-03-21T13:52:55Z) - Post-training Quantization for Text-to-Image Diffusion Models with Progressive Calibration and Activation Relaxing [49.800746112114375]
We propose a novel post-training quantization method (Progressive and Relaxing) for text-to-image diffusion models.
We are the first to achieve quantization for Stable Diffusion XL while maintaining the performance.
arXiv Detail & Related papers (2023-11-10T09:10:09Z) - Semi-Implicit Denoising Diffusion Models (SIDDMs) [50.30163684539586]
Existing models such as Denoising Diffusion Probabilistic Models (DDPM) deliver high-quality, diverse samples but are slowed by an inherently high number of iterative steps.
We introduce a novel approach that tackles the problem by matching implicit and explicit factors.
We demonstrate that our proposed method obtains comparable generative performance to diffusion-based models and vastly superior results to models with a small number of sampling steps.
arXiv Detail & Related papers (2023-06-21T18:49:22Z) - Hierarchical Integration Diffusion Model for Realistic Image Deblurring [71.76410266003917]
Diffusion models (DMs) have been introduced in image deblurring and exhibited promising performance.
We propose the Hierarchical Integration Diffusion Model (HI-Diff), for realistic image deblurring.
Experiments on synthetic and real-world blur datasets demonstrate that our HI-Diff outperforms state-of-the-art methods.
arXiv Detail & Related papers (2023-05-22T12:18:20Z) - PTQD: Accurate Post-Training Quantization for Diffusion Models [22.567863065523902]
Post-training quantization of diffusion models can significantly reduce the model size and accelerate the sampling process without re-training.
Applying existing PTQ methods directly to low-bit diffusion models can significantly impair the quality of generated samples.
We propose a unified formulation for the quantization noise and diffusion perturbed noise in the quantized denoising process.
arXiv Detail & Related papers (2023-05-18T02:28:42Z) - Reflected Diffusion Models [93.26107023470979]
We present Reflected Diffusion Models, which reverse a reflected differential equation evolving on the support of the data.
Our approach learns the score function through a generalized score matching loss and extends key components of standard diffusion models.
arXiv Detail & Related papers (2023-04-10T17:54:38Z) - Q-Diffusion: Quantizing Diffusion Models [52.978047249670276]
Post-training quantization (PTQ) is considered a go-to compression method for other tasks.
We propose a novel PTQ method specifically tailored towards the unique multi-timestep pipeline and model architecture.
We show that our proposed method is able to quantize full-precision unconditional diffusion models into 4-bit while maintaining comparable performance.
arXiv Detail & Related papers (2023-02-08T19:38:59Z) - Fast Inference in Denoising Diffusion Models via MMD Finetuning [23.779985842891705]
We present MMD-DDM, a novel method for fast sampling of diffusion models.
Our approach is based on the idea of using the Maximum Mean Discrepancy (MMD) to finetune the learned distribution with a given budget of timesteps.
Our findings show that the proposed method is able to produce high-quality samples in a fraction of the time required by widely-used diffusion models.
arXiv Detail & Related papers (2023-01-19T09:48:07Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.