Pioneering 4-Bit FP Quantization for Diffusion Models: Mixup-Sign Quantization and Timestep-Aware Fine-Tuning

• URL: http://arxiv.org/abs/2505.21591v1
• Date: Tue, 27 May 2025 13:40:47 GMT
• Title: Pioneering 4-Bit FP Quantization for Diffusion Models: Mixup-Sign Quantization and Timestep-Aware Fine-Tuning
• Abstract summary: Model quantization reduces the bit-width of weights and activations, improving memory efficiency and inference speed.<n>Existing methods, primarily based on integer quantization and post-training quantization fine-tuning, struggle with inconsistent performance.<n>We propose a mixup-sign floating-point quantization framework, first introducing unsigned FP quantization in model quantization, along with timestep-aware LoRA and denoising-factor loss alignment.
• Score: 14.145862114439831
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Model quantization reduces the bit-width of weights and activations, improving memory efficiency and inference speed in diffusion models. However, achieving 4-bit quantization remains challenging. Existing methods, primarily based on integer quantization and post-training quantization fine-tuning, struggle with inconsistent performance. Inspired by the success of floating-point (FP) quantization in large language models, we explore low-bit FP quantization for diffusion models and identify key challenges: the failure of signed FP quantization to handle asymmetric activation distributions, the insufficient consideration of temporal complexity in the denoising process during fine-tuning, and the misalignment between fine-tuning loss and quantization error. To address these challenges, we propose the mixup-sign floating-point quantization (MSFP) framework, first introducing unsigned FP quantization in model quantization, along with timestep-aware LoRA (TALoRA) and denoising-factor loss alignment (DFA), which ensure precise and stable fine-tuning. Extensive experiments show that we are the first to achieve superior performance in 4-bit FP quantization for diffusion models, outperforming existing PTQ fine-tuning methods in 4-bit INT quantization.

Related papers

• Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMs [14.385983354036142]
  Low-bit quantization offers a promising avenue for reducing the computational and memory demands of Multimodal Large Language Models.<n>Recent hardware support for low-precision formats, such as MXFP8 and ultra-low-bit formats such as MXFP4 and HiF4, has accelerated research into efficient MLLM training and deployment.<n>We present a systematic study of these quantization schemes in representative MLLMs that span both video generation and reasoning tasks.
  arXiv  Detail & Related papers  (2026-08-27T03:47:58Z)
• Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs [78.01752802974855]
  We propose Mix-Quant, a simple and effective phase-aware quantization framework for fast agentic inference.<n>We show that Mix-Quant preserves task performance while delivering significant efficiency improvements, achieving up to a 3x speedup during prefilling.
  arXiv  Detail & Related papers  (2026-05-19T17:50:17Z)
• SOAR: Scale Optimization for Accurate Reconstruction in NVFP4 Quantization [19.022444007775896]
  NVFP4 has recently emerged as an efficient 4-bit microscaling format for large language models.<n>Existing methods often yield suboptimal performance due to inflexible scale selection and the coupled treatment of quantization and dequantization scales.<n>We propose Scale Optimization for Accurate Reconstruction (SOAR), a novel post-training quantization framework that improves the accuracy of NVFP4 quantization.
  arXiv  Detail & Related papers  (2026-05-12T15:13:18Z)
• MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models with Temporal Distillation [74.34220141721231]
  We present MPQ-DMv2, an improved textbfMixed textbfPrecision textbfQuantization framework for extremely low-bit textbfDiffusion textbfModels.
  arXiv  Detail & Related papers  (2025-07-06T08:16:50Z)
• MergeQuant: Accurate 4-bit Static Quantization of Large Language Models by Channel-wise Calibration [23.752021919501207]
  We propose MergeQuant, an accurate and efficient per-channel static quantization framework.<n> MergeQuant integrates the per-channel quantization steps with the corresponding scalings and linear mappings through a Quantization Step Migration (QSM) method.<n>On Llama-2-7B model, MergeQuant achieves up to 1.77x speedup in decoding, and up to 2.06x speedup in end-to-end compared to FP16 baseline.
  arXiv  Detail & Related papers  (2025-03-07T04:52:28Z)
• Optimizing Large Language Model Training Using FP4 Quantization [73.55459961002371]
  Quantized training presents a promising solution by enabling low-bit arithmetic operations to reduce costs.<n>This work introduces the first FP4 training framework for large language models (LLMs)
  arXiv  Detail & Related papers  (2025-01-28T18:04:50Z)
• TCAQ-DM: Timestep-Channel Adaptive Quantization for Diffusion Models [49.65286242048452]
  We propose a novel method dubbed Timestep-Channel Adaptive Quantization for Diffusion Models (TCAQ-DM)<n>The proposed method substantially outperforms the state-of-the-art approaches in most cases.
  arXiv  Detail & Related papers  (2024-12-21T16:57:54Z)
• 2DQuant: Low-bit Post-Training Quantization for Image Super-Resolution [83.09117439860607]
  Low-bit quantization has become widespread for compressing image super-resolution (SR) models for edge deployment. It is notorious that low-bit quantization degrades the accuracy of SR models compared to their full-precision (FP) counterparts. We present a dual-stage low-bit post-training quantization (PTQ) method for image super-resolution, namely 2DQuant, which achieves efficient and accurate SR under low-bit quantization.
  arXiv  Detail & Related papers  (2024-06-10T06:06:11Z)
• QNCD: Quantization Noise Correction for Diffusion Models [15.189069680672239]
  Diffusion models have revolutionized image synthesis, setting new benchmarks in quality and creativity. Post-training quantization presents a solution to accelerate sampling, aibeit at the expense of sample quality. We introduce a unified Quantization Noise Correction Scheme (QNCD) aimed at minishing quantization noise throughout the sampling process.
  arXiv  Detail & Related papers  (2024-03-28T04:24:56Z)
• QuEST: Low-bit Diffusion Model Quantization via Efficient Selective Finetuning [52.157939524815866]
  In this paper, we identify imbalanced activation distributions as a primary source of quantization difficulty.<n>We propose to adjust these distributions through weight finetuning to be more quantization-friendly.<n>Our method demonstrates its efficacy across three high-resolution image generation tasks.
  arXiv  Detail & Related papers  (2024-02-06T03:39:44Z)
• LLM-FP4: 4-Bit Floating-Point Quantized Transformers [38.23587031169402]
  We propose LLM-FP4 for quantizing both weights and activations in large language models (LLMs) down to 4-bit floating-point values. Compared to integer quantization, floating-point (FP) quantization is more flexible and can better handle long-tail or bell-shaped distributions. Our method, for the first time, can quantize both weights and activations in the LLaMA-13B to only 4-bit and achieves an average score of 63.1.
  arXiv  Detail & Related papers  (2023-10-25T17:59:32Z)
• Q-Diffusion: Quantizing Diffusion Models [52.978047249670276]
  Post-training quantization (PTQ) is considered a go-to compression method for other tasks. We propose a novel PTQ method specifically tailored towards the unique multi-timestep pipeline and model architecture. We show that our proposed method is able to quantize full-precision unconditional diffusion models into 4-bit while maintaining comparable performance.
  arXiv  Detail & Related papers  (2023-02-08T19:38:59Z)
• QFT: Post-training quantization via fast joint finetuning of all degrees of freedom [1.1744028458220428]
  We rethink quantized network parameterization in HW-aware fashion, towards a unified analysis of all quantization DoF. Our single-step simple and extendable method, dubbed quantization-aware finetuning (QFT), achieves 4-bit weight quantization results on-par with SoTA.
  arXiv  Detail & Related papers  (2022-12-05T22:38:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.