ParaGAN: A Scalable Distributed Training Framework for Generative Adversarial Networks

• URL: http://arxiv.org/abs/2411.03999v1
• Date: Wed, 06 Nov 2024 15:40:46 GMT
• Title: ParaGAN: A Scalable Distributed Training Framework for Generative Adversarial Networks
• Authors: Ziji Shi, Jialin Li, Yang You,
• Abstract summary: ParaGAN is a scalable distributed GAN training framework that leverages asynchronous training and asymmetric optimization policy to accelerate GAN training. With ParaGAN, we reduce the training time of BigGAN from 15 days to 14 hours while achieving 91% scaling efficiency. ParaGAN enables unprecedented high-resolution image generation using BigGAN.
• Score: 8.736828068085263
• License:
• Abstract: Recent advances in Generative Artificial Intelligence have fueled numerous applications, particularly those involving Generative Adversarial Networks (GANs), which are essential for synthesizing realistic photos and videos. However, efficiently training GANs remains a critical challenge due to their computationally intensive and numerically unstable nature. Existing methods often require days or even weeks for training, posing significant resource and time constraints. In this work, we introduce ParaGAN, a scalable distributed GAN training framework that leverages asynchronous training and an asymmetric optimization policy to accelerate GAN training. ParaGAN employs a congestion-aware data pipeline and hardware-aware layout transformation to enhance accelerator utilization, resulting in over 30% improvements in throughput. With ParaGAN, we reduce the training time of BigGAN from 15 days to 14 hours while achieving 91% scaling efficiency. Additionally, ParaGAN enables unprecedented high-resolution image generation using BigGAN.

Related papers

• E$^{2}$GAN: Efficient Training of Efficient GANs for Image-to-Image Translation [69.72194342962615]
  We introduce and address a novel research direction: can the process of distilling GANs from diffusion models be made significantly more efficient? First, we construct a base GAN model with generalized features, adaptable to different concepts through fine-tuning, eliminating the need for training from scratch. Second, we identify crucial layers within the base GAN model and employ Low-Rank Adaptation (LoRA) with a simple yet effective rank search process, rather than fine-tuning the entire base model. Third, we investigate the minimal amount of data necessary for fine-tuning, further reducing the overall training time.
  arXiv  Detail & Related papers  (2024-01-11T18:59:14Z)
• LD-GAN: Low-Dimensional Generative Adversarial Network for Spectral Image Generation with Variance Regularization [72.4394510913927]
  Deep learning methods are state-of-the-art for spectral image (SI) computational tasks. GANs enable diverse augmentation by learning and sampling from the data distribution. GAN-based SI generation is challenging since the high-dimensionality nature of this kind of data hinders the convergence of the GAN training yielding to suboptimal generation. We propose a statistical regularization to control the low-dimensional representation variance for the autoencoder training and to achieve high diversity of samples generated with the GAN.
  arXiv  Detail & Related papers  (2023-04-29T00:25:02Z)
• Time Efficient Training of Progressive Generative Adversarial Network using Depthwise Separable Convolution and Super Resolution Generative Adversarial Network [0.0]
  We propose a novel pipeline that combines Progressive GAN with slight modifications and Super Resolution GAN. Super Resolution GAN up samples low-resolution images to high-resolution images which can prove to be a useful resource to reduce the training time exponentially.
  arXiv  Detail & Related papers  (2022-02-24T19:53:37Z)
• EAGAN: Efficient Two-stage Evolutionary Architecture Search for GANs [25.791031022393643]
  Generative Adversarial Networks (GANs) have been proven hugely successful in image generation tasks, but GAN training has the problem of instability. We propose an efficient two-stage evolutionary algorithm (EA) based NAS framework to discover GANs, dubbed textbfEAGAN.
  arXiv  Detail & Related papers  (2021-11-30T03:28:09Z)
• Ultra-Data-Efficient GAN Training: Drawing A Lottery Ticket First, Then Training It Toughly [114.81028176850404]
  Training generative adversarial networks (GANs) with limited data generally results in deteriorated performance and collapsed models. We decompose the data-hungry GAN training into two sequential sub-problems. Such a coordinated framework enables us to focus on lower-complexity and more data-efficient sub-problems.
  arXiv  Detail & Related papers  (2021-02-28T05:20:29Z)
• Learning Efficient GANs for Image Translation via Differentiable Masks and co-Attention Distillation [130.30465659190773]
  Generative Adversarial Networks (GANs) have been widely-used in image translation, but their high computation and storage costs impede the deployment on mobile devices. We introduce a novel GAN compression method, termed DMAD, by proposing a Differentiable Mask and a co-Attention Distillation. Experiments show DMAD can reduce the Multiply Accumulate Operations (MACs) of CycleGAN by 13x and that of Pix2Pix by 4x while retaining a comparable performance against the full model.
  arXiv  Detail & Related papers  (2020-11-17T02:39:19Z)
• Improving the Speed and Quality of GAN by Adversarial Training [87.70013107142142]
  We develop FastGAN to improve the speed and quality of GAN training based on the adversarial training technique. Our training algorithm brings ImageNet training to the broader public by requiring 2-4 GPUs.
  arXiv  Detail & Related papers  (2020-08-07T20:21:31Z)
• Improving GAN Training with Probability Ratio Clipping and Sample Reweighting [145.5106274085799]
  generative adversarial networks (GANs) often suffer from inferior performance due to unstable training. We propose a new variational GAN training framework which enjoys superior training stability. By plugging the training approach in diverse state-of-the-art GAN architectures, we obtain significantly improved performance over a range of tasks.
  arXiv  Detail & Related papers  (2020-06-12T01:39:48Z)
• FusedProp: Towards Efficient Training of Generative Adversarial Networks [0.0]
  We propose the fused propagation algorithm which can be used to efficiently train the discriminator and the generator of common GANs simultaneously. We show that FusedProp achieves 1.49 times the training speed compared to the conventional training of GANs.
  arXiv  Detail & Related papers  (2020-03-30T06:46:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.