Related papers: How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size

How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size

URL: http://arxiv.org/abs/2607.01487v1
Date: Wed, 01 Jul 2026 21:32:14 GMT
Title: How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size
Abstract summary: We propose a scaling law that takes into account model size and training data while explicitly splitting the latter into training steps and batch size.<n> Fitting the proposed law on a large set of training runs, we find that it correctly recovers the scaling of the optimal batch size.<n>Because it makes use of training runs with suboptimal batch size, our proposed law can be robustly fit with a significantly smaller amount of training runs.
Score: 1.1603243575080533
License: http://creativecommons.org/licenses/by/4.0/
Abstract: We propose a scaling law that takes into account model size and training data while explicitly splitting the latter into training steps and batch size (called three-term law). Fitting the proposed law on a large set of training runs, we find that it correctly recovers the scaling of the optimal batch size. Moreover, because it makes use of training runs with suboptimal batch size, our proposed law can be robustly fit with a significantly smaller amount of training runs. We further show that the three-term law can be used to derive scaling laws for suboptimal batch sizes, and that it matches previous empirical findings related to the critical batch size.

Related papers

Amortizing Scaling Law Construction Costs [19.769099912679824]
We propose a framework for efficient scaling law construction that formulates data collection as a Bayesian optimization problem.<n>We find that progressively expanding the compute budget during acquisition, mirroring the compute-ordered evaluation of configurations in practice, substantially improves recovery efficiency.
arXiv Detail & Related papers (2026-09-04T11:25:41Z)
Predictive Scaling Laws for Efficient GRPO Training of Large Reasoning Models [0.41942958779358663]
We propose a predictive framework that models training dynamics and helps optimize resource usage.<n>We derive an empirical scaling law based on model size, initial performance, and training progress.<n>We find that training beyond certain number of an epoch offers little gain, suggesting earlier stopping can significantly reduce compute without sacrificing performance.
arXiv Detail & Related papers (2025-07-24T01:09:25Z)
Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful [69.57125049281993]
This work revisits small batch sizes all the way down to batch size one.<n>We find that small batch sizes train stably and achieve equal or better per-FLOP performance than larger batch sizes.
arXiv Detail & Related papers (2025-07-09T17:57:36Z)
Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training [54.860881625923184]
We show how a critical batch size (CBS) can be estimated based on the gradient noise scale during training.<n>Our findings about how the CBS changes over training motivate batch size warmup, suggesting CBS from small training runs can inform larger-scale training runs.
arXiv Detail & Related papers (2025-05-29T19:53:39Z)
The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws [51.608402959163925]
We present the first systematic exploration of optimal sparse pre-training configurations for large language models.<n>We find that initiating pruning at 25% of total training compute and concluding at 75% achieves near-optimal final evaluation loss.<n>We propose a new scaling law that modifies the Chinchilla scaling law to use the average parameter count over pre-training.
arXiv Detail & Related papers (2025-01-21T20:23:22Z)
Scaling Law for Language Models Training Considering Batch Size [17.09348741898811]
Large language models (LLMs) have made remarkable advances in recent years, with scaling laws playing a critical role in this rapid progress.<n>We empirically investigate how a critical hyper- parameter, i.e., the global batch size, influences the LLM training prdocess.<n>We establish a basic scaling law on model size and training data amount.<n>We then examine how varying batch sizes and learning rates affect the convergence and generalization of these models.
arXiv Detail & Related papers (2024-12-02T13:58:35Z)
Scaling Laws for Precision [73.24325358259753]
We devise "precision-aware" scaling laws for both training and inference.<n>For inference, we find that the degradation introduced by post-training quantization increases as models are trained on more data.<n>For training, our scaling laws allow us to predict the loss of a model with different parts in different precisions.
arXiv Detail & Related papers (2024-11-07T00:10:10Z)
Bayesian scaling laws for in-context learning [85.34114399339741]
In-context learning (ICL) is a powerful technique for getting language models to perform complex tasks with no training updates.<n>We show that ICL approximates a Bayesian learner, which gives rise to a novel Bayesian scaling law for ICL.<n>Our scaling law matches existing scaling laws in accuracy while also offering interpretable terms for task priors, learning efficiency, and per-example probabilities.
arXiv Detail & Related papers (2024-10-21T21:45:22Z)
Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations [62.132347451049455]
Scale has become a main ingredient in obtaining strong machine learning models. In this work, we argue that scale and training research has been needlessly complex due to reliance on the cosine schedule. We show that weight averaging yields improved performance along the training trajectory, without additional training costs, across different scales.
arXiv Detail & Related papers (2024-05-28T17:33:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.