Hybrid Dual-Batch and Cyclic Progressive Learning for Efficient Distributed Training

• URL: http://arxiv.org/abs/2509.26092v2
• Date: Fri, 31 Oct 2025 07:41:36 GMT
• Title: Hybrid Dual-Batch and Cyclic Progressive Learning for Efficient Distributed Training
• Authors: Kuan-Wei Lu, Ding-Yong Hong, Pangfeng Liu, Jan-Jan Wu,
• Abstract summary: Experimental results with ResNet-18 demonstrate that, compared to conventional training methods, our approach improves accuracy by 3.3%.<n>By combining cyclic progressive learning with dual-batch learning, our hybrid approach improves both model generalization and training efficiency.
• Score: 1.084959821967413
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Distributed machine learning is critical for training deep learning models on large datasets with numerous parameters. Current research primarily focuses on leveraging additional hardware resources and powerful computing units to accelerate the training process. As a result, larger batch sizes are often employed to speed up training. However, training with large batch sizes can lead to lower accuracy due to poor generalization. To address this issue, we propose the dual-batch learning scheme, a distributed training method built on the parameter server framework. This approach maximizes training efficiency by utilizing the largest batch size that the hardware can support while incorporating a smaller batch size to enhance model generalization. By using two different batch sizes simultaneously, this method improves accuracy with minimal additional training time. Additionally, to mitigate the time overhead caused by dual-batch learning, we propose the cyclic progressive learning scheme. This technique repeatedly and gradually increases image resolution from low to high during training, thereby reducing training time. By combining cyclic progressive learning with dual-batch learning, our hybrid approach improves both model generalization and training efficiency. Experimental results with ResNet-18 demonstrate that, compared to conventional training methods, our approach improves accuracy by 3.3% while reducing training time by 10.1% on CIFAR-100, and further achieves a 34.8% reduction in training time on ImageNet.

Related papers

• Always-Sparse Training by Growing Connections with Guided Stochastic Exploration [43.26615926465987]
  We propose an efficient always-sparse training algorithm with excellent scaling to larger and sparser models.<n>We evaluate our method on CIFAR-10/100 and ImageNet using VGG, and ViT models, and compare it against a range of sparsification methods.
  arXiv  Detail & Related papers  (2024-01-12T21:32:04Z)
• Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Size [58.762959061522736]
  We show that scaling mini-batch sizes with appropriate learning rate adjustments can speed up the training process by orders of magnitude. We show that scaling the mini-batch size and naively adjusting the learning rate allows for (1) a reduced size of the Q-ensemble, (2) stronger penalization of out-of-distribution actions, and (3) improved convergence time.
  arXiv  Detail & Related papers  (2022-11-20T21:48:25Z)
• Efficient and Effective Augmentation Strategy for Adversarial Training [48.735220353660324]
  Adversarial training of Deep Neural Networks is known to be significantly more data-hungry than standard training. We propose Diverse Augmentation-based Joint Adversarial Training (DAJAT) to use data augmentations effectively in adversarial training.
  arXiv  Detail & Related papers  (2022-10-27T10:59:55Z)
• Online Convolutional Re-parameterization [51.97831675242173]
  We present online convolutional re- parameterization (OREPA), a two-stage pipeline, aiming to reduce the huge training overhead by squeezing the complex training-time block into a single convolution. Compared with the state-of-the-art re-param models, OREPA is able to save the training-time memory cost by about 70% and accelerate the training speed by around 2x. We also conduct experiments on object detection and semantic segmentation and show consistent improvements on the downstream tasks.
  arXiv  Detail & Related papers  (2022-04-02T09:50:19Z)
• Curriculum Learning: A Regularization Method for Efficient and Stable Billion-Scale GPT Model Pre-Training [18.640076155697415]
  We present a study of a curriculum learning based approach, which helps improve the pre-training convergence speed of autoregressive models. Our evaluations demonstrate that curriculum learning enables training GPT-2 models with 8x larger batch size and 4x larger learning rate.
  arXiv  Detail & Related papers  (2021-08-13T06:32:53Z)
• Automated Learning Rate Scheduler for Large-batch Training [24.20872850681828]
  Large-batch training has been essential in leveraging large-scale datasets and models in deep learning. It often requires a specially designed learning rate (LR) schedule to achieve a comparable level of performance as in smaller batch training. We propose an automated LR scheduling algorithm which is effective for neural network training with a large batch size under the given epoch budget.
  arXiv  Detail & Related papers  (2021-07-13T05:23:13Z)
• Concurrent Adversarial Learning for Large-Batch Training [83.55868483681748]
  Adversarial learning is a natural choice for smoothing the decision surface and biasing towards a flat region. We propose a novel Concurrent Adversarial Learning (ConAdv) method that decouples the sequential gradient computations in adversarial learning by utilizing staled parameters. This is the first work successfully scales ResNet-50 training batch size to 96K.
  arXiv  Detail & Related papers  (2021-06-01T04:26:02Z)
• Jigsaw Clustering for Unsupervised Visual Representation Learning [68.09280490213399]
  We propose a new jigsaw clustering pretext task in this paper. Our method makes use of information from both intra- and inter-images. It is even comparable to the contrastive learning methods when only half of training batches are used.
  arXiv  Detail & Related papers  (2021-04-01T08:09:26Z)
• Improving the Speed and Quality of GAN by Adversarial Training [87.70013107142142]
  We develop FastGAN to improve the speed and quality of GAN training based on the adversarial training technique. Our training algorithm brings ImageNet training to the broader public by requiring 2-4 GPUs.
  arXiv  Detail & Related papers  (2020-08-07T20:21:31Z)
• Accelerated Large Batch Optimization of BERT Pretraining in 54 minutes [9.213729275749452]
  We propose an accelerated gradient method called LANS to improve the efficiency of using large mini-batches for training. It takes 54 minutes on 192 AWS EC2 P3dn.24xlarge instances to achieve a target F1 score of 90.5 or higher on SQuAD v1.1, achieving the fastest BERT training time in the cloud.
  arXiv  Detail & Related papers  (2020-06-24T05:00:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.