MPQ-DM: Mixed Precision Quantization for Extremely Low Bit Diffusion Models

• URL: http://arxiv.org/abs/2412.11549v1
• Date: Mon, 16 Dec 2024 08:31:55 GMT
• Title: MPQ-DM: Mixed Precision Quantization for Extremely Low Bit Diffusion Models
• Authors: Weilun Feng, Haotong Qin, Chuanguang Yang, Zhulin An, Libo Huang, Boyu Diao, Fei Wang, Renshuai Tao, Yongjun Xu, Michele Magno,
• Abstract summary: This paper presents MPQ-DM, a Mixed-Precision Quantization method for Diffusion Models. To mitigate the quantization error caused by outlier severe weight channels, we propose an Outlier-Driven Mixed Quantization technique. To robustly learn representations crossing time steps, we construct a Time-Smoothed Relation Distillation scheme.
• Score: 37.061975191553
• License:
• Abstract: Diffusion models have received wide attention in generation tasks. However, the expensive computation cost prevents the application of diffusion models in resource-constrained scenarios. Quantization emerges as a practical solution that significantly saves storage and computation by reducing the bit-width of parameters. However, the existing quantization methods for diffusion models still cause severe degradation in performance, especially under extremely low bit-widths (2-4 bit). The primary decrease in performance comes from the significant discretization of activation values at low bit quantization. Too few activation candidates are unfriendly for outlier significant weight channel quantization, and the discretized features prevent stable learning over different time steps of the diffusion model. This paper presents MPQ-DM, a Mixed-Precision Quantization method for Diffusion Models. The proposed MPQ-DM mainly relies on two techniques:(1) To mitigate the quantization error caused by outlier severe weight channels, we propose an Outlier-Driven Mixed Quantization (OMQ) technique that uses $Kurtosis$ to quantify outlier salient channels and apply optimized intra-layer mixed-precision bit-width allocation to recover accuracy performance within target efficiency.(2) To robustly learn representations crossing time steps, we construct a Time-Smoothed Relation Distillation (TRD) scheme between the quantized diffusion model and its full-precision counterpart, transferring discrete and continuous latent to a unified relation space to reduce the representation inconsistency. Comprehensive experiments demonstrate that MPQ-DM achieves significant accuracy gains under extremely low bit-widths compared with SOTA quantization methods. MPQ-DM achieves a 58\% FID decrease under W2A4 setting compared with baseline, while all other methods even collapse.

Related papers

• TCAQ-DM: Timestep-Channel Adaptive Quantization for Diffusion Models [49.65286242048452]
  We propose a novel method dubbed Timestep-Channel Adaptive Quantization for Diffusion Models (TCAQ-DM) The proposed method substantially outperforms the state-of-the-art approaches in most cases.
  arXiv  Detail & Related papers  (2024-12-21T16:57:54Z)
• Timestep-Aware Correction for Quantized Diffusion Models [28.265582848911574]
  We propose a timestep-aware correction method for quantized diffusion model, which dynamically corrects the quantization error. By leveraging the proposed method in low-precision diffusion models, substantial enhancement of output quality could be achieved with only negligible overhead.
  arXiv  Detail & Related papers  (2024-07-04T13:22:31Z)
• 2DQuant: Low-bit Post-Training Quantization for Image Super-Resolution [83.09117439860607]
  Low-bit quantization has become widespread for compressing image super-resolution (SR) models for edge deployment. It is notorious that low-bit quantization degrades the accuracy of SR models compared to their full-precision (FP) counterparts. We present a dual-stage low-bit post-training quantization (PTQ) method for image super-resolution, namely 2DQuant, which achieves efficient and accurate SR under low-bit quantization.
  arXiv  Detail & Related papers  (2024-06-10T06:06:11Z)
• DB-LLM: Accurate Dual-Binarization for Efficient LLMs [83.70686728471547]
  Large language models (LLMs) have significantly advanced the field of natural language processing. Existing ultra-low-bit quantization always causes severe accuracy drops. We propose a novel Dual-Binarization method for LLMs, namely DB-LLM.
  arXiv  Detail & Related papers  (2024-02-19T09:04:30Z)
• QuEST: Low-bit Diffusion Model Quantization via Efficient Selective Finetuning [52.157939524815866]
  In this paper, we empirically unravel three properties in quantized diffusion models that compromise the efficacy of current methods. We identify two critical types of quantized layers: those holding vital temporal information and those sensitive to reduced bit-width. Our method is evaluated over three high-resolution image generation tasks and achieves state-of-the-art performance under various bit-width settings.
  arXiv  Detail & Related papers  (2024-02-06T03:39:44Z)
• Post-training Quantization for Text-to-Image Diffusion Models with Progressive Calibration and Activation Relaxing [49.800746112114375]
  We propose a novel post-training quantization method (Progressive and Relaxing) for text-to-image diffusion models. We are the first to achieve quantization for Stable Diffusion XL while maintaining the performance.
  arXiv  Detail & Related papers  (2023-11-10T09:10:09Z)
• PTQD: Accurate Post-Training Quantization for Diffusion Models [22.567863065523902]
  Post-training quantization of diffusion models can significantly reduce the model size and accelerate the sampling process without re-training. Applying existing PTQ methods directly to low-bit diffusion models can significantly impair the quality of generated samples. We propose a unified formulation for the quantization noise and diffusion perturbed noise in the quantized denoising process.
  arXiv  Detail & Related papers  (2023-05-18T02:28:42Z)
• Q-Diffusion: Quantizing Diffusion Models [52.978047249670276]
  Post-training quantization (PTQ) is considered a go-to compression method for other tasks. We propose a novel PTQ method specifically tailored towards the unique multi-timestep pipeline and model architecture. We show that our proposed method is able to quantize full-precision unconditional diffusion models into 4-bit while maintaining comparable performance.
  arXiv  Detail & Related papers  (2023-02-08T19:38:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.