Tula: Optimizing Time, Cost, and Generalization in Distributed Large-Batch Training

• URL: http://arxiv.org/abs/2603.18112v1
• Date: Wed, 18 Mar 2026 13:56:32 GMT
• Title: Tula: Optimizing Time, Cost, and Generalization in Distributed Large-Batch Training
• Abstract summary: Tula is an online service that automatically optimize time, cost, and convergence quality for large-batch training of convolutional models.<n>Tula predicts training time and cost within 7.5-14% error across multiple models, and achieves up to 20x overall speedup.
• Score: 2.19670601855638
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Distributed training increases the number of batches processed per iteration either by scaling-out (adding more nodes) or scaling-up (increasing the batch-size). However, the largest configuration does not necessarily yield the best performance. Horizontal scaling introduces additional communication overhead, while vertical scaling is constrained by computation cost and device memory limits. Thus, simply increasing the batch-size leads to diminishing returns: training time and cost decrease initially but eventually plateaus, creating a knee-point in the time/cost versus batch-size pareto curve. The optimal batch-size therefore depends on the underlying model, data and available compute resources. Large batches also suffer from worse model quality due to the well-known generalization gap. In this paper, we present Tula, an online service that automatically optimizes time, cost, and convergence quality for large-batch training of convolutional models. It combines parallel-systems modeling with statistical performance prediction to identify the optimal batch-size. Tula predicts training time and cost within 7.5-14% error across multiple models, and achieves up to 20x overall speedup and improves test accuracy by 9% on average over standard large-batch training on various vision tasks, thus successfully mitigating the generalization gap and accelerating training at the same time.

Related papers

• When Do Larger Batches Help Scale LLM Reinforcement Learning? [13.656143976571007]
  We study tradeoffs in reinforcement learning for large language models.<n>We find that larger batches improve generation throughput by up to 2.29x on fixed hardware.
  arXiv  Detail & Related papers  (2026-08-29T14:32:09Z)
• Deep Progressive Training: scaling up depth capacity of zero/one-layer models [19.649807308477527]
  We study the depth expansion of large models through the lens of optimization theory.<n>We propose zero/one-layer progressive training for the optimal tradeoff between computation and loss.
  arXiv  Detail & Related papers  (2025-11-07T04:56:45Z)
• Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling [75.36692892951018]
  Increasing the batch size during training is a promising strategy to accelerate large language model pretraining.<n>This work develops a principled framework for batch-size scheduling.<n>It introduces Seesaw: whenever a standard scheduler would halve the learning rate, Seesaw instead multiplies it by $1/sqrt2$ and doubles the batch size.
  arXiv  Detail & Related papers  (2025-10-16T14:17:38Z)
• Hybrid Dual-Batch and Cyclic Progressive Learning for Efficient Distributed Training [1.084959821967413]
  Experimental results with ResNet-18 demonstrate that, compared to conventional training methods, our approach improves accuracy by 3.3%.<n>By combining cyclic progressive learning with dual-batch learning, our hybrid approach improves both model generalization and training efficiency.
  arXiv  Detail & Related papers  (2025-09-30T11:10:47Z)
• Compute-Optimal Scaling for Value-Based Deep RL [99.680827753493]
  We investigate compute scaling for online, value-based deep RL.<n>Our analysis reveals a nuanced interplay between model size, batch size, and UTD.<n>We provide a mental model for understanding this phenomenon and build guidelines for choosing batch size and UTD.
  arXiv  Detail & Related papers  (2025-08-20T17:54:21Z)
• CoMERA: Computing- and Memory-Efficient Training via Rank-Adaptive Tensor Optimization [9.826264204082095]
  Training large AI models such as LLMs and DLRMs costs massive GPU and computing time.<n>CoMERA achieves rank-adaptive tensor-compressed (pre)-training via a multi-objective optimization formulation.<n>CoMERA is $2times$ faster per training epoch and $9times$ more memory-efficient than GaLore.
  arXiv  Detail & Related papers  (2024-05-23T09:52:15Z)
• A Dynamical Model of Neural Scaling Laws [79.59705237659547]
  We analyze a random feature model trained with gradient descent as a solvable model of network training and generalization. Our theory shows how the gap between training and test loss can gradually build up over time due to repeated reuse of data.
  arXiv  Detail & Related papers  (2024-02-02T01:41:38Z)
• Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Model [89.8764435351222]
  We propose a new family of unbiased estimators called WTA-CRS, for matrix production with reduced variance. Our work provides both theoretical and experimental evidence that, in the context of tuning transformers, our proposed estimators exhibit lower variance compared to existing ones.
  arXiv  Detail & Related papers  (2023-05-24T15:52:08Z)
• Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Size [58.762959061522736]
  We show that scaling mini-batch sizes with appropriate learning rate adjustments can speed up the training process by orders of magnitude. We show that scaling the mini-batch size and naively adjusting the learning rate allows for (1) a reduced size of the Q-ensemble, (2) stronger penalization of out-of-distribution actions, and (3) improved convergence time.
  arXiv  Detail & Related papers  (2022-11-20T21:48:25Z)
• Automated Learning Rate Scheduler for Large-batch Training [24.20872850681828]
  Large-batch training has been essential in leveraging large-scale datasets and models in deep learning. It often requires a specially designed learning rate (LR) schedule to achieve a comparable level of performance as in smaller batch training. We propose an automated LR scheduling algorithm which is effective for neural network training with a large batch size under the given epoch budget.
  arXiv  Detail & Related papers  (2021-07-13T05:23:13Z)
• Concurrent Adversarial Learning for Large-Batch Training [83.55868483681748]
  Adversarial learning is a natural choice for smoothing the decision surface and biasing towards a flat region. We propose a novel Concurrent Adversarial Learning (ConAdv) method that decouples the sequential gradient computations in adversarial learning by utilizing staled parameters. This is the first work successfully scales ResNet-50 training batch size to 96K.
  arXiv  Detail & Related papers  (2021-06-01T04:26:02Z)
• AdaScale SGD: A User-Friendly Algorithm for Distributed Training [29.430153773234363]
  We propose AdaScale SGD, an algorithm that reliably adapts learning rates to large-batch training. By continually adapting to the gradient's variance, AdaScale achieves speed-ups for a wide range of batch sizes. This includes large-batch training with no model degradation for machine translation, image classification, object detection, and speech recognition tasks.
  arXiv  Detail & Related papers  (2020-07-09T23:26:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.