On the Nonlinearity of Learning Rate Scaling for LLM Training

• URL: http://arxiv.org/abs/2606.29158v1
• Date: Sun, 28 Jun 2026 02:42:47 GMT
• Title: On the Nonlinearity of Learning Rate Scaling for LLM Training
• Abstract summary: Learning-rate transfer can reduce the cost of training large language models.<n>Existing approaches often assume that the optimal learning rate follows a log-linear scaling law in data scale and model size.
• Score: 18.557199144090017
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning-rate transfer can reduce the cost of training large language models: instead of sweeping learning rates at target scale, practitioners extrapolate from smaller runs. Existing approaches often assume that the optimal learning rate follows a log-linear scaling law in data scale and model size. We carefully examine and evaluate this scaling law. In our empirical study of GPT-2--style models from 22M to 707M parameters trained on 5B to 100B tokens, the optimal learning rate develops upward curvature at larger scales, leading to inaccurate extrapolation. We find that this curvature largely disappears when learning rates are replaced by effective learning rate (the step size in normalized weight space), and when data $D$ extrapolation is used instead of model size $N$ extrapolation. Next, we explain nonlinearity in scaling: weight-norm converges to equilibrium slower when optimal learning is small, requiring a larger step size to reduce the transient phase. Experiments with AdamH, which directly controls the effective learning rate, further support this explanation.

Related papers

• Sketched Linear Contrastive Learning: Approximation, Optimization, and Statistical Scaling [12.993805198900498]
  We study a sketched linear model for contrastive learning under a paired Gaussian latent-variable setup.<n>We derive a risk decomposition into irreducible risk, approximation error, GD bias, GD variance, and a cross term.<n>Compared with standard linear-regression scaling laws, the contrastive setting must learn interactions between two views.
  arXiv  Detail & Related papers  (2026-06-25T05:20:21Z)
• Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales [55.91454326946738]
  We study how the optimal learning rate and weight decay should scale with model width and depth for a wide range of languages.<n>We find that scaling the learning rate according to $$P improves transfer, but can still suffer from significant finite-width deviations.<n>For compute-optimal scaling, we find scaling independent weight decay as $1/mathrmwidth$ is nearly optimal across languages.
  arXiv  Detail & Related papers  (2025-12-05T11:03:41Z)
• Deep Progressive Training: scaling up depth capacity of zero/one-layer models [19.649807308477527]
  We study the depth expansion of large models through the lens of optimization theory.<n>We propose zero/one-layer progressive training for the optimal tradeoff between computation and loss.
  arXiv  Detail & Related papers  (2025-11-07T04:56:45Z)
• Scaling with Collapse: Efficient and Predictable Training of LLM Families [8.979516613284174]
  Collapse emerges as a signature of compute-efficient training.<n>We show that collapse emerges as a signature of compute-efficient training.<n>We demonstrate two applications at scale.
  arXiv  Detail & Related papers  (2025-09-29T17:26:11Z)
• Compute-Optimal Scaling for Value-Based Deep RL [99.680827753493]
  We investigate compute scaling for online, value-based deep RL.<n>Our analysis reveals a nuanced interplay between model size, batch size, and UTD.<n>We provide a mental model for understanding this phenomenon and build guidelines for choosing batch size and UTD.
  arXiv  Detail & Related papers  (2025-08-20T17:54:21Z)
• AutoScale: Scale-Aware Data Mixing for Pre-Training LLMs [59.12061830645018]
  We show that data mixtures that perform well at smaller scales may not retain their advantage at larger scales.<n>We propose AutoScale, a two-stage, scale-aware data composition framework.
  arXiv  Detail & Related papers  (2024-07-29T17:06:30Z)
• Occam Gradient Descent [0.0]
  We propose an algorithm that combines gradient descent with magnitude pruning into "Occam Gradient Descent"<n> neural networks trained with Occam Gradient Descent outperform traditional gradient descent, as well as Random Forests.
  arXiv  Detail & Related papers  (2024-05-30T15:58:22Z)
• Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling [27.058009599819012]
  We study the connection between optimal learning rates and batch sizes for Adam styles. We prove that the optimal learning rate first rises and then falls as the batch size increases.
  arXiv  Detail & Related papers  (2024-05-23T13:52:36Z)
• A Dynamical Model of Neural Scaling Laws [79.59705237659547]
  We analyze a random feature model trained with gradient descent as a solvable model of network training and generalization. Our theory shows how the gap between training and test loss can gradually build up over time due to repeated reuse of data.
  arXiv  Detail & Related papers  (2024-02-02T01:41:38Z)
• Direction Matters: On the Implicit Bias of Stochastic Gradient Descent with Moderate Learning Rate [105.62979485062756]
  This paper attempts to characterize the particular regularization effect of SGD in the moderate learning rate regime. We show that SGD converges along the large eigenvalue directions of the data matrix, while GD goes after the small eigenvalue directions.
  arXiv  Detail & Related papers  (2020-11-04T21:07:52Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.