Early-Bird Diffusion: Investigating and Leveraging Timestep-Aware Early-Bird Tickets in Diffusion Models for Efficient Training

• URL: http://arxiv.org/abs/2504.09606v1
• Date: Sun, 13 Apr 2025 14:53:18 GMT
• Title: Early-Bird Diffusion: Investigating and Leveraging Timestep-Aware Early-Bird Tickets in Diffusion Models for Efficient Training
• Authors: Lexington Whalen, Zhenbang Du, Haoran You, Chaojian Li, Sixu Li, Yingyan, Lin,
• Abstract summary: Training diffusion models (DMs) requires substantial computational resources.<n>We propose EB-Diff-Train, a new efficient DM training approach.<n>This approach can significantly reduce training time both spatially and temporally.
• Score: 9.680799617245194
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training diffusion models (DMs) requires substantial computational resources due to multiple forward and backward passes across numerous timesteps, motivating research into efficient training techniques. In this paper, we propose EB-Diff-Train, a new efficient DM training approach that is orthogonal to other methods of accelerating DM training, by investigating and leveraging Early-Bird (EB) tickets -- sparse subnetworks that manifest early in the training process and maintain high generation quality. We first investigate the existence of traditional EB tickets in DMs, enabling competitive generation quality without fully training a dense model. Then, we delve into the concept of diffusion-dedicated EB tickets, drawing on insights from varying importance of different timestep regions. These tickets adapt their sparsity levels according to the importance of corresponding timestep regions, allowing for aggressive sparsity during non-critical regions while conserving computational resources for crucial timestep regions. Building on this, we develop an efficient DM training technique that derives timestep-aware EB tickets, trains them in parallel, and combines them during inference for image generation. Extensive experiments validate the existence of both traditional and timestep-aware EB tickets, as well as the effectiveness of our proposed EB-Diff-Train method. This approach can significantly reduce training time both spatially and temporally -- achieving 2.9$\times$ to 5.8$\times$ speedups over training unpruned dense models, and up to 10.3$\times$ faster training compared to standard train-prune-finetune pipelines -- without compromising generative quality. Our code is available at https://github.com/GATECH-EIC/Early-Bird-Diffusion.

Related papers

• Exploring the Benefit of Activation Sparsity in Pre-training [117.25661020250658]
  We study how activation properties change during pre-training. We propose Switchable Sparse-Dense Learning (SSD) SSD achieves comparable performance with identical model size and reduces pre-training costs.
  arXiv  Detail & Related papers  (2024-10-04T13:53:33Z)
• Efficient Stagewise Pretraining via Progressive Subnetworks [53.00045381931778]
  The prevailing view suggests that stagewise dropping strategies, such as layer dropping, are ineffective when compared to stacking-based approaches. This paper challenges this notion by demonstrating that, with proper design, dropping strategies can be competitive, if not better, than stacking methods. We propose an instantiation of this framework - Random Part Training (RAPTR) - that selects and trains only a random subnetwork at each step, progressively increasing the size in stages.
  arXiv  Detail & Related papers  (2024-02-08T18:49:09Z)
• Always-Sparse Training by Growing Connections with Guided Stochastic Exploration [46.4179239171213]
  We propose an efficient always-sparse training algorithm with excellent scaling to larger and sparser models. We evaluate our method on CIFAR-10/100 and ImageNet using VGG, and ViT models, and compare it against a range of sparsification methods.
  arXiv  Detail & Related papers  (2024-01-12T21:32:04Z)
• Efficient Adversarial Training with Robust Early-Bird Tickets [57.72115485770303]
  We find that robust connectivity patterns emerge in the early training phase, far before parameters converge. Inspired by this finding, we dig out robust early-bird tickets to develop an efficient adversarial training method. Experiments show that the proposed efficient adversarial training method can achieve up to $7times sim 13 times$ training speedups.
  arXiv  Detail & Related papers  (2022-11-14T10:44:25Z)
• EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets [106.79387235014379]
  EarlyBERT is a general computationally-efficient training algorithm applicable to both pre-training and fine-tuning of large-scale language models. We are the first to identify structured winning tickets in the early stage of BERT training, and use them for efficient training. EarlyBERT easily achieves comparable performance to standard BERT with 3545% less training time.
  arXiv  Detail & Related papers  (2020-12-31T20:38:20Z)
• Fast is better than free: Revisiting adversarial training [86.11788847990783]
  We show that it is possible to train empirically robust models using a much weaker and cheaper adversary. We identify a failure mode referred to as "catastrophic overfitting" which may have caused previous attempts to use FGSM adversarial training to fail.
  arXiv  Detail & Related papers  (2020-01-12T20:30:22Z)
• Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks [82.52404247479359]
  Early-bird (EB) tickets can be identified at the very early training stage.<n>We propose a mask distance metric that can be used to identify EB tickets with low computational overhead.
  arXiv  Detail & Related papers  (2019-09-26T07:43:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.