Amortizing Scaling Law Construction Costs

• URL: http://arxiv.org/abs/2609.05016v1
• Date: Fri, 04 Sep 2026 11:25:41 GMT
• Title: Amortizing Scaling Law Construction Costs
• Abstract summary: We propose a framework for efficient scaling law construction that formulates data collection as a Bayesian optimization problem.<n>We find that progressively expanding the compute budget during acquisition, mirroring the compute-ordered evaluation of configurations in practice, substantially improves recovery efficiency.
• Score: 19.769099912679824
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scaling laws guide the design choices for training large foundation models, but deriving them involves training an exhaustive grid over hyperparameters, token budgets, and parameter counts, which is computationally expensive. Fitting a scaling law, however, only requires the best-loss frontier across compute scales, discarding most of the trained configurations. We propose a framework for efficient scaling law construction that formulates data collection as a Bayesian optimization problem, and introduce metrics for comparing scaling law fitting methods under constrained compute budgets. We find that progressively expanding the compute budget during acquisition, mirroring the compute-ordered evaluation of configurations in practice, substantially improves recovery efficiency. Augmenting the observed configurations with surrogate-fantasized evaluations then recovers the broader experimental grid, allowing accurate scaling law fitting without training every configuration. Together, these can closely match scaling law fits over a full dense grid at computational savings of up to $10\text{--}100\times$.

Related papers

• SGD in Multiclass Logistic Regression: Sequential Learning and Scaling Laws [55.81298869922287]
  We study the training dynamics of multiclass logistic regression on high-dimensional Gaussian mixture models with a large number of classes.<n>We establish precise scaling laws governing the cross-entropy risk under gradient-based optimization.
  arXiv  Detail & Related papers  (2026-09-07T18:30:09Z)
• Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search [51.33002870475756]
  We introduce Power-Law Entropy Search (PLES), a cost-aware acquisition function built on multi-fidelity Bayesian optimization.<n>PLES searches for candidates that reduce the overall uncertainty of a scaling law estimate, instead of optimizing a single objective function.
  arXiv  Detail & Related papers  (2026-09-01T15:41:59Z)
• Prescriptive Scaling Laws for Data Constrained Training [21.261009598515066]
  Training compute is increasingly outpacing the availability of high-quality data.<n>Chinchilla scaling law assumes every training token is unique.<n>Our scaling law yields qualitatively new compute-optimal allocation advice.
  arXiv  Detail & Related papers  (2026-05-02T23:14:20Z)
• Configuration-to-Performance Scaling Law with Neural Ansatz [19.686833161453464]
  We learn a textitConfiguration-to-Performance Scaling Law (CPL)<n>CPL accurately predicts how training configurations influence the final pretraining loss.<n>It achieves 20-40% lower prediction error than the configuration-agnostic Chinchilla law.
  arXiv  Detail & Related papers  (2026-02-10T21:16:59Z)
• Towards Robust Scaling Laws for Optimizers [89.21160945066737]
  Empirical scaling laws are widely used to predict loss as model size and training data grow.<n>We show that Chinchilla-style scaling laws emerge naturally as a result of loss decomposition into irreducible, approximation, and optimization errors.
  arXiv  Detail & Related papers  (2026-02-07T21:40:33Z)
• Compute Optimal Scaling of Skills: Knowledge vs Reasoning [50.76705503978189]
  We ask whether compute-optimal scaling behaviour can be skill-dependent.<n>In particular, we examine knowledge and reasoning-based skills such as knowledge-based QA and code generation.<n>We conclude with an analysis of how our findings relate to standard compute-optimal scaling using a validation set.
  arXiv  Detail & Related papers  (2025-03-13T05:21:22Z)
• Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining [59.369484219304866]
  We conduct an unprecedented empirical investigation training over 3,700 Large Language Models (LLMs) from scratch across 100 trillion tokens.<n>We establish a universal Scaling Law for hyperparameter optimization in LLM Pre-training, called Step Law.<n>Our estimated optima deviates from the global best performance found via exhaustive search by merely 0.094% on the test set.
  arXiv  Detail & Related papers  (2025-03-06T18:58:29Z)
• The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws [51.608402959163925]
  We present the first systematic exploration of optimal sparse pre-training configurations for large language models.<n>We find that initiating pruning at 25% of total training compute and concluding at 75% achieves near-optimal final evaluation loss.<n>We propose a new scaling law that modifies the Chinchilla scaling law to use the average parameter count over pre-training.
  arXiv  Detail & Related papers  (2025-01-21T20:23:22Z)
• Scaling Laws for Fine-Grained Mixture of Experts [4.412803924115907]
  Mixture of Experts (MoE) models have emerged as a primary solution for reducing the computational cost of Large Language Models. In this work, we analyze their scaling properties, incorporating an expanded range of variables. We establish scaling laws for fine-grained MoE, taking into account the number of training tokens, model size, and granularity.
  arXiv  Detail & Related papers  (2024-02-12T18:33:47Z)
• Scaling Laws for Sparsely-Connected Foundation Models [70.41266138010657]
  We explore the impact of parameter sparsity on the scaling behavior of Transformers trained on massive datasets. We identify the first scaling law describing the relationship between weight sparsity, number of non-zero parameters, and amount of training data.
  arXiv  Detail & Related papers  (2023-09-15T16:29:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.