A Closer Look at Time Steps is Worthy of Triple Speed-Up for Diffusion Model Training

• URL: http://arxiv.org/abs/2405.17403v3
• Date: Tue, 25 Mar 2025 08:38:28 GMT
• Title: A Closer Look at Time Steps is Worthy of Triple Speed-Up for Diffusion Model Training
• Authors: Kai Wang, Mingjia Shi, Yukun Zhou, Zekai Li, Zhihang Yuan, Yuzhang Shang, Xiaojiang Peng, Hanwang Zhang, Yang You,
• Abstract summary: We introduce a novel speed-up method for diffusion model training, called, which is based on a closer look at time steps.<n>As a plug-and-play and architecture-agnostic approach, SpeeD consistently achieves 3-times acceleration across various diffusion architectures, datasets, and tasks.
• Score: 53.93563224892207
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training diffusion models is always a computation-intensive task. In this paper, we introduce a novel speed-up method for diffusion model training, called, which is based on a closer look at time steps. Our key findings are: i) Time steps can be empirically divided into acceleration, deceleration, and convergence areas based on the process increment. ii) These time steps are imbalanced, with many concentrated in the convergence area. iii) The concentrated steps provide limited benefits for diffusion training. To address this, we design an asymmetric sampling strategy that reduces the frequency of steps from the convergence area while increasing the sampling probability for steps from other areas. Additionally, we propose a weighting strategy to emphasize the importance of time steps with rapid-change process increments. As a plug-and-play and architecture-agnostic approach, SpeeD consistently achieves 3-times acceleration across various diffusion architectures, datasets, and tasks. Notably, due to its simple design, our approach significantly reduces the cost of diffusion model training with minimal overhead. Our research enables more researchers to train diffusion models at a lower cost.

Related papers

• Efficient Diffusion Training through Parallelization with Truncated Karhunen-Loève Expansion [5.770347328961063]
  Diffusion denoising models suffer from slow convergence during training. We propose a novel forward-time process for training and sampling. Our method significantly outperforms baseline diffusion models.
  arXiv  Detail & Related papers  (2025-03-22T05:34:02Z)
• TPDiff: Temporal Pyramid Video Diffusion Model [16.48006100084994]
  We propose TPDiff, a unified framework to enhance training and inference efficiency. By dividing diffusion into several stages, our framework progressively increases frame rate along the diffusion process. By solving the partitioned probability flow ordinary differential equations (ODE) of diffusion under aligned data and noise, our training strategy is applicable to various diffusion forms.
  arXiv  Detail & Related papers  (2025-03-12T17:33:22Z)
• RayFlow: Instance-Aware Diffusion Acceleration via Adaptive Flow Trajectories [17.934379261227388]
  Existing acceleration methods compromise sample quality, controllability, or introduce training complexities. We propose RayFlow, a novel diffusion framework that addresses these limitations. Extensive experiments demonstrate RayFlow's superiority in generating high-quality images with improved speed, control, and training efficiency.
  arXiv  Detail & Related papers  (2025-03-10T17:20:52Z)
• Effortless Efficiency: Low-Cost Pruning of Diffusion Models [29.821803522137913]
  We propose a model-agnostic structural pruning framework for diffusion models that learns a differentiable mask to sparsify the model. To ensure effective pruning that preserves the quality of the final denoised latent, we design a novel end-to-end pruning objective that spans the entire diffusion process. Results on state-of-the-art U-Net diffusion models SDXL and diffusion transformers (FLUX) demonstrate that our method can effectively prune up to 20% parameters with minimal perceptible performance degradation.
  arXiv  Detail & Related papers  (2024-12-03T21:37:50Z)
• Adaptive Non-Uniform Timestep Sampling for Diffusion Model Training [4.760537994346813]
  As data distributions grow more complex, training diffusion models to convergence becomes increasingly intensive. We introduce a non-uniform timestep sampling method that prioritizes these more critical timesteps. Our method shows robust performance across various datasets, scheduling strategies, and diffusion architectures.
  arXiv  Detail & Related papers  (2024-11-15T07:12:18Z)
• Flexiffusion: Segment-wise Neural Architecture Search for Flexible Denoising Schedule [50.260693393896716]
  Diffusion models are cutting-edge generative models adept at producing diverse, high-quality images. Recent techniques have been employed to automatically search for faster generation processes. We introduce Flexiffusion, a novel training-free NAS paradigm designed to accelerate diffusion models.
  arXiv  Detail & Related papers  (2024-09-26T06:28:05Z)
• Adv-KD: Adversarial Knowledge Distillation for Faster Diffusion Sampling [2.91204440475204]
  Diffusion Probabilistic Models (DPMs) have emerged as a powerful class of deep generative models. They rely on sequential denoising steps during sample generation. We propose a novel method that integrates denoising phases directly into the model's architecture.
  arXiv  Detail & Related papers  (2024-05-31T08:19:44Z)
• Efficient Diffusion Model for Image Restoration by Residual Shifting [63.02725947015132]
  This study proposes a novel and efficient diffusion model for image restoration. Our method avoids the need for post-acceleration during inference, thereby avoiding the associated performance deterioration. Our method achieves superior or comparable performance to current state-of-the-art methods on three classical IR tasks.
  arXiv  Detail & Related papers  (2024-03-12T05:06:07Z)
• Towards More Accurate Diffusion Model Acceleration with A Timestep Aligner [84.97253871387028]
  A diffusion model, which is formulated to produce an image using thousands of denoising steps, usually suffers from a slow inference speed. We propose a timestep aligner that helps find a more accurate integral direction for a particular interval at the minimum cost. Experiments show that our plug-in design can be trained efficiently and boost the inference performance of various state-of-the-art acceleration methods.
  arXiv  Detail & Related papers  (2023-10-14T02:19:07Z)
• AutoDiffusion: Training-Free Optimization of Time Steps and Architectures for Automated Diffusion Model Acceleration [57.846038404893626]
  We propose to search the optimal time steps sequence and compressed model architecture in a unified framework to achieve effective image generation for diffusion models without any further training. Experimental results show that our method achieves excellent performance by using only a few time steps, e.g. 17.86 FID score on ImageNet 64 $times$ 64 with only four steps, compared to 138.66 with DDIM.
  arXiv  Detail & Related papers  (2023-09-19T08:57:24Z)
• Fast Diffusion Model [122.36693015093041]
  Diffusion models (DMs) have been adopted across diverse fields with their abilities in capturing intricate data distributions. In this paper, we propose a Fast Diffusion Model (FDM) to significantly speed up DMs from a DM optimization perspective.
  arXiv  Detail & Related papers  (2023-06-12T09:38:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.