FourTune: Towards Fully 4-Bit Efficient Post-Training for Diffusion Models

• URL: http://arxiv.org/abs/2607.05711v1
• Date: Tue, 07 Jul 2026 00:34:30 GMT
• Title: FourTune: Towards Fully 4-Bit Efficient Post-Training for Diffusion Models
• Abstract summary: FourTune is an efficient post-training framework for diffusion models based on an end-to-end W4A4G4 paradigm.<n>FourTune reduces memory overhead by 2.25$times$ and increases end-to-end training throughput by 2.27$times$ compared to BF16 LoRA.
• Score: 50.66675248468456
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion models have become a dominant paradigm for high-quality generative modeling, while post-training is essential for adapting them to diverse downstream applications. However, post-training of large diffusion models is still challenging due to the prohibitive memory footprints and slow training speed, which existing parameter-efficient fine-tuning methods only partially address. To overcome these limitations, we propose FourTune, an efficient post-training framework for diffusion models based on an end-to-end W4A4G4 paradigm. FourTune introduces a triple-branch hybrid pipeline that augments the standard LoRA architecture with a frozen numerical stabilizer to isolate quantization-sensitive outliers, enabling stable training under native 4-bit computation. In addition, FourTune employs hardware-efficient block-wise quantization and customized fused kernels to support efficient quantized backpropagation and reduce memory bandwidth overhead. Across customization, reinforcement learning, and distillation tasks, FourTune matches the quality of full-precision fine-tuning. On FLUX.1-dev (12B), FourTune reduces memory overhead by 2.25$\times$ and increases end-to-end training throughput by 2.27$\times$ compared to BF16 LoRA.

Related papers

• Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention [34.07319488267708]
  Recent NVFP4 pretraining methods mainly target transformer linear layers, leaving computation states, arithmetic and attention underexplored in 4-bit pipelines.<n>This critical gap blocks stable full-stack 4-bit pretraining, as the three core modules exhibit unique numerical failure patterns.<n>We propose Full-Stack FP4, the first complete NVFP4 pretraining framework resolving all three stability bottlenecks.
  arXiv  Detail & Related papers  (2026-07-05T17:31:36Z)
• FP4 Explore, BF16 Train: Diffusion Reinforcement Learning via Efficient Rollout Scaling [38.64059734487925]
  Reinforcement-Learning-based post-training has emerged as a promising paradigm for aligning text-to-image diffusion models with human preferences.<n> scaling rollouts on large-scale foundational diffusion models (e.g., FLUX.1-12B) imposes a heavy computational burden.<n>We propose Sol-RL, a novel FP4-empowered Two-stage Reinforcement Learning framework.
  arXiv  Detail & Related papers  (2026-04-08T10:14:47Z)
• BWTA: Accurate and Efficient Binarized Transformer by Algorithm-Hardware Co-design [71.97035034203275]
  We analyze zero-point distortion in binarization and propose a Binary Weights & Ternary Activations (BWTA) quantization scheme.<n>For training, we propose Smooth Multi-Stage Quantization, combining a Levelwise Degradation Strategy and a MagnitudeAlignment Projection Factor.<n>Experiments show that BWTA approaches full-precision performance for Transformer-based models, with an average 3.5% drop on GLUE and less than 2% drop on five tasks.
  arXiv  Detail & Related papers  (2026-04-05T04:25:07Z)
• HQ-DM: Single Hadamard Transformation-Based Quantization-Aware Training for Low-Bit Diffusion Models [5.320690460117234]
  HQ-DM is a novel Quantization-Aware Training framework that applies Single Hadamard Transformation to activation matrices.<n>This approach effectively reduces activation outliers while preserving model performance under quantization.<n>For conditional generation on the ImageNet 256x256 dataset using the LDM-4 model, our W4A4 and W4A3 quantization schemes improve the Inception Score by 12.8% and 467.73%, respectively.
  arXiv  Detail & Related papers  (2025-12-05T14:28:40Z)
• Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling [13.357423392911036]
  We introduce Four Over Six (4/6), a modification to the NVFP4 quantization algorithm that evaluates two potential scale factors for each block of values.<n>We find that for some blocks, scaling to smaller FP4 values makes the distribution of representable values more uniform.<n>We also find that 4/6 can be easily incorporated into many different post-training quantization methods and generally improves downstream accuracy.
  arXiv  Detail & Related papers  (2025-12-01T18:59:45Z)
• Pretraining Large Language Models with NVFP4 [53.235038214986865]
  We introduce a novel approach for stable and accurate training of large language models (LLMs) using the NVFP4 format.<n>Our method integrates two-dimensional quantization scheme for consistent representations across both the forward and backward passes.<n>Our results show that the model trained with our NVFP4-based pretraining technique achieves training loss and downstream task accuracies comparable to an FP8 baseline.
  arXiv  Detail & Related papers  (2025-09-29T17:53:17Z)
• Optimizing Large Language Model Training Using FP4 Quantization [73.55459961002371]
  Quantized training presents a promising solution by enabling low-bit arithmetic operations to reduce costs.<n>This work introduces the first FP4 training framework for large language models (LLMs)
  arXiv  Detail & Related papers  (2025-01-28T18:04:50Z)
• SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models [61.474101404805545]
  Diffusion models can generate high-quality images, but as they scale, rising memory demands and higher latency pose deployment challenges.<n>We propose SVDQuant, a new 4-bit quantization paradigm to overcome this limitation.<n>We reduce the memory usage for the 12B FLUX.1 models by 3.5$times$, achieving 3.0$times$ speedup over the 4-bit weight-only quantization (W4A16) baseline.
  arXiv  Detail & Related papers  (2024-11-07T18:59:58Z)
• QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models [57.04178959678024]
  We show that the majority of inference computations for large generative models can be performed with both weights and activations being cast to 4 bits. We achieve this via a hybrid quantization strategy called QUIK, which compresses most of the weights and activations to 4-bit. We provide GPU kernels matching the QUIK format with highly-efficient layer-wise runtimes, which lead to practical end-to-end throughput improvements of up to 3.4x.
  arXiv  Detail & Related papers  (2023-10-13T17:15:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.