Active Budget Allocation for Efficient Scaling Law Estimation via Surrogate-Guided Pruning

• URL: http://arxiv.org/abs/2605.17234v2
• Date: Mon, 25 May 2026 04:58:43 GMT
• Title: Active Budget Allocation for Efficient Scaling Law Estimation via Surrogate-Guided Pruning
• Abstract summary: Empirical scaling law research identifies functional forms to aid this prediction task.<n>Due to the empirical nature of this approach, the computational burden is substantial.<n>We explore the suitability of Successive Halving (SH) and SH combined with parametric and non-parametric surrogate models.
• Score: 20.863873378387865
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Predicting model performance at larger scales enables the design of training strategies and architectures tailored to specific performance targets. Empirical scaling law research identifies functional forms to aid this prediction task. These describe the relationship between loss and compute using a loss-compute frontier defined by learning curves. Due to the empirical nature of this approach, the computational burden is substantial, making strategic resource allocation essential - yet it remains surprisingly underexplored. In this work, we address this shortcoming by exploring the suitability of Successive Halving (SH) and SH combined with parametric and non-parametric surrogate models. In addition to enabling a more systematic allocation of a given compute budget, our findings show that SH paired with surrogate models yields a set of learning curves that includes one with a lower loss-compute value than what naive uniform allocation or an SH-only approach can obtain. Our experiments demonstrate mean relative improvements of up to 2.84% and 5.47% on real-world and synthetic learning curve datasets. This strategic resource allocation enables us to obtain accurate scaling laws at significantly reduced computational costs, saving up to 98.7% over the traditional exhaustive approach.

Related papers

• Direct Advantage Estimation for Scalable and Sample-efficient Deep Reinforcement Learning [54.916555668983726]
  We introduce discrete latent dynamics models that efficiently approximate transition probabilities.<n>We find that DAE scales effectively with function approximator capacity while retaining high sample efficiency.
  arXiv  Detail & Related papers  (2026-06-18T15:58:48Z)
• Observationally Informed Adaptive Causal Experimental Design [55.998153710215654]
  We propose Active Residual Learning, a new paradigm that leverages the observational model as a foundational prior.<n>This approach shifts the experimental focus from learning target causal quantities from scratch to efficiently estimating the residuals required to correct observational bias.<n> Experiments on synthetic and semi-synthetic benchmarks demonstrate that R-Design significantly outperforms baselines.
  arXiv  Detail & Related papers  (2026-03-04T06:52:37Z)
• ODAR: Principled Adaptive Routing for LLM Reasoning via Active Inference [60.958331943869126]
  ODAR-Expert is an adaptive routing framework that optimize the accuracy-efficiency trade-off via principled resource allocation.<n>We show strong and consistent gains, including 98.2% accuracy on MATH and 54.8% on Humanity's Last Exam.
  arXiv  Detail & Related papers  (2026-02-27T05:22:01Z)
• Active Learning Using Aggregated Acquisition Functions: Accuracy and Sustainability Analysis [14.398823059302279]
  Active learning (AL) is a machine learning approach that strategically selects the most informative samples for annotation during training.<n>This strategy not only reduces labeling expenses but also results in energy savings during neural network training.<n>We implement and evaluate various state-of-the-art acquisition functions, analyzing their accuracy and computational costs.
  arXiv  Detail & Related papers  (2026-02-07T08:42:12Z)
• Computational Economics in Large Language Models: Exploring Model Behavior and Incentive Design under Resource Constraints [1.00707850217229]
  Large language models (LLMs) are limited by substantial computational cost.<n>We introduce a "computational economics" framework that treats an LLM as an internal economy of resource-constrained agents.<n>We show that when computation is scarce, standard LLMs reallocate attention toward high-value tokens while preserving accuracy.
  arXiv  Detail & Related papers  (2025-08-14T07:55:45Z)
• A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search [15.387256204743407]
  Large language models (LLMs) demand considerable computational, energy, and financial resources during both training and deployment.<n>Inference costs now represent a significant and growing component of the overall resource burden.<n>We introduce directed skill search (DS3), a general framework that represents inference as expressive over a learned skill graph.
  arXiv  Detail & Related papers  (2025-06-10T14:47:48Z)
• Outcome-Based Online Reinforcement Learning: Algorithms and Fundamental Limits [58.63897489864948]
  Reinforcement learning with outcome-based feedback faces a fundamental challenge.<n>How do we assign credit to the right actions?<n>This paper provides the first comprehensive analysis of this problem in online RL with general function approximation.
  arXiv  Detail & Related papers  (2025-05-26T17:44:08Z)
• COSMOS: Predictable and Cost-Effective Adaptation of LLMs [21.91455944905485]
  Large language models (LLMs) achieve remarkable performance across numerous tasks by using a diverse array of adaptation strategies.<n>We introduce COSMOS, a unified prediction framework that efficiently estimates adaptation outcomes at minimal cost.
  arXiv  Detail & Related papers  (2025-04-30T02:06:26Z)
• Scaling Laws for Sparsely-Connected Foundation Models [70.41266138010657]
  We explore the impact of parameter sparsity on the scaling behavior of Transformers trained on massive datasets. We identify the first scaling law describing the relationship between weight sparsity, number of non-zero parameters, and amount of training data.
  arXiv  Detail & Related papers  (2023-09-15T16:29:27Z)
• Accurate and Robust Feature Importance Estimation under Distribution Shifts [49.58991359544005]
  PRoFILE is a novel feature importance estimation method. We show significant improvements over state-of-the-art approaches, both in terms of fidelity and robustness.
  arXiv  Detail & Related papers  (2020-09-30T05:29:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.