Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search

• URL: http://arxiv.org/abs/2609.01431v2
• Date: Thu, 03 Sep 2026 15:16:24 GMT
• Title: Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search
• Abstract summary: We introduce Power-Law Entropy Search (PLES), a cost-aware acquisition function built on multi-fidelity Bayesian optimization.<n>PLES searches for candidates that reduce the overall uncertainty of a scaling law estimate, instead of optimizing a single objective function.
• Score: 51.33002870475756
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Optimal hyperparameter scaling laws describe how the best hyperparameters for large language model (LLM) training change with model and data scale, enabling practitioners to predict optimal configurations at production scales without expensive large-scale tuning. However, estimating these scaling laws conventionally requires exhaustive grid searches over thousands of training runs, consuming enormous computational resources. We introduce Power-Law Entropy Search (PLES), a computational cost-aware acquisition function built on multi-fidelity Bayesian optimization that efficiently estimates optimal hyperparameter scaling laws through adaptive experimentation. A key innovation in PLES is that it searches for candidates that reduce the overall uncertainty of a scaling law estimate, instead of optimizing a single objective function. At each iteration, PLES selects the candidate configuration that maximally reduces the uncertainty of the scaling law estimates per unit computational cost, naturally favoring informative small-scale experiments. We evaluate PLES on synthetic benchmarks, surrogate models fitted to real LLM training data, and actual LLM pre-training runs. Across all settings, PLES converges to accurate optimal hyperparameter scaling laws using less than one-tenth of the computational budget required by conventional grid search and other baselines.

Related papers

• Amortizing Scaling Law Construction Costs [19.769099912679824]
  We propose a framework for efficient scaling law construction that formulates data collection as a Bayesian optimization problem.<n>We find that progressively expanding the compute budget during acquisition, mirroring the compute-ordered evaluation of configurations in practice, substantially improves recovery efficiency.
  arXiv  Detail & Related papers  (2026-09-04T11:25:41Z)
• Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training [7.267441247692648]
  We propose a novel framework to establish relationships between compute budget and optimal hyperparameters for a given checkpoint.<n>We show that our method reduces the hyperparameter search overhead by up to 90% while achieving comparable or superior performance relative to baselines.<n>This model-agnostic framework generalizes across architectures, providing a principled and efficient methodology for diverse continued pre-training scenarios.
  arXiv  Detail & Related papers  (2026-06-04T02:32:11Z)
• Configuration-to-Performance Scaling Law with Neural Ansatz [19.686833161453464]
  We learn a textitConfiguration-to-Performance Scaling Law (CPL)<n>CPL accurately predicts how training configurations influence the final pretraining loss.<n>It achieves 20-40% lower prediction error than the configuration-agnostic Chinchilla law.
  arXiv  Detail & Related papers  (2026-02-10T21:16:59Z)
• Towards Robust Scaling Laws for Optimizers [89.21160945066737]
  Empirical scaling laws are widely used to predict loss as model size and training data grow.<n>We show that Chinchilla-style scaling laws emerge naturally as a result of loss decomposition into irreducible, approximation, and optimization errors.
  arXiv  Detail & Related papers  (2026-02-07T21:40:33Z)
• Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales [55.91454326946738]
  We study how the optimal learning rate and weight decay should scale with model width and depth for a wide range of languages.<n>We find that scaling the learning rate according to $$P improves transfer, but can still suffer from significant finite-width deviations.<n>For compute-optimal scaling, we find scaling independent weight decay as $1/mathrmwidth$ is nearly optimal across languages.
  arXiv  Detail & Related papers  (2025-12-05T11:03:41Z)
• Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining [59.369484219304866]
  We conduct an unprecedented empirical investigation training over 3,700 Large Language Models (LLMs) from scratch across 100 trillion tokens.<n>We establish a universal Scaling Law for hyperparameter optimization in LLM Pre-training, called Step Law.<n>Our estimated optima deviates from the global best performance found via exhaustive search by merely 0.094% on the test set.
  arXiv  Detail & Related papers  (2025-03-06T18:58:29Z)
• The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws [51.608402959163925]
  We present the first systematic exploration of optimal sparse pre-training configurations for large language models.<n>We find that initiating pruning at 25% of total training compute and concluding at 75% achieves near-optimal final evaluation loss.<n>We propose a new scaling law that modifies the Chinchilla scaling law to use the average parameter count over pre-training.
  arXiv  Detail & Related papers  (2025-01-21T20:23:22Z)
• Optimization Hyper-parameter Laws for Large Language Models [52.49860340549727]
  We present Opt-Laws, a framework that captures the relationship between hyper- parameters and training outcomes.<n>Our validation across diverse model sizes and data scales demonstrates Opt-Laws' ability to accurately predict training loss.<n>This approach significantly reduces computational costs while enhancing overall model performance.
  arXiv  Detail & Related papers  (2024-09-07T09:37:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.