The Unreasonable Effectiveness of Scaling Laws in AI

• URL: http://arxiv.org/abs/2603.28507v1
• Date: Mon, 30 Mar 2026 14:42:53 GMT
• Title: The Unreasonable Effectiveness of Scaling Laws in AI
• Abstract summary: scaling laws describe how training loss decreases with compute in a power-law form.<n>Despite diminishing returns they predict, progress in practice has often continued through rapidly improving efficiency.<n>This paper argues that both features arise from the same source: scaling laws are unusually effective because they abstract away from many realization details.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Classical AI scaling laws, especially for pre-training, describe how training loss decreases with compute in a power-law form. Their effectiveness has a basic and very practical sense: they make progress predictable, albeit at a declining rate. Yet their effectiveness is also unreasonable in two further senses. First, these laws are largely empirical and observational, but they appear repeatedly across model families and increasingly across training-adjacent regimes. Second, despite the diminishing returns they predict, progress in practice has often continued through rapidly improving efficiency, visible for example in falling cost per token. This paper argues that both features arise from the same source: scaling laws are unusually effective because they abstract away from many realization details. The compute variable is best understood as logical compute, an implementation-agnostic notion of model-side work, while the practical burden of scaling depends on how efficiently real resources are converted into that compute. This abstraction helps explain both why the laws travel so well across settings and why they give rise to a persistent efficiency game in hardware, algorithms, and systems. Once efficiency is made explicit, the main practical question becomes how many efficiency doublings are required to keep scaling productive despite diminishing returns. Under that view, diminishing returns are not only a geometric flattening of the loss curve, but also rising pressure for cost reduction, system-level innovation, and the breakthroughs needed to sustain Moore-like efficiency doublings.

Related papers

• The Quantization Trap: Breaking Linear Scaling Laws in Multi-Hop Reasoning [10.91558162376728]
  We demonstrate that this scaling law breaks in the context of multi-hop reasoning.<n>We reveal a 'quantization trap' where reducing precision from 16-bit to 8/4-bit paradoxically increases more net energy consumption while degrading reasoning accuracy.
  arXiv  Detail & Related papers  (2026-02-14T04:25:27Z)
• AI Agents as Universal Task Solvers [94.49762121230042]
  We show that the optimal speed-up that a universal solver can achieve using past data is tightly related to their algorithmic information.<n>We argue that the key quantity to optimize when scaling reasoning models is time, whose critical role in learning has so far only been indirectly considered.
  arXiv  Detail & Related papers  (2025-10-14T02:17:54Z)
• Scaling Laws for Data-Efficient Visual Transfer Learning [14.114908296325277]
  This paper establishes the first practical framework for data-efficient scaling laws in visual transfer learning.<n>We propose the distillation boundary theory, revealing a critical turning point in distillation efficiency.<n>This work redefines scaling laws for data-limited regimes, bridging the knowledge gap between large-scale pretraining and practical downstream adaptation.
  arXiv  Detail & Related papers  (2025-04-17T07:01:01Z)
• A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules [67.87680482844884]
  We present an empirical law that describes how the pretraining loss of large language models evolves under different learning rate schedules.<n>Our proposed law takes a multi-power form, combining a power law based on the sum of learning rates and additional power laws to account for a loss reduction effect induced by learning rate decay.
  arXiv  Detail & Related papers  (2025-03-17T04:36:45Z)
• Scaling Law Phenomena Across Regression Paradigms: Multiple and Kernel Approaches [28.569601803576845]
  We show that for models with Transformer architecture, the test loss exhibits a power-law relationship with model size, dataset size, and the amount of computation used in training.<n>Our analysis provides deeper insights into the scaling law, potentially enhancing our understanding of Large Language Models.
  arXiv  Detail & Related papers  (2025-03-03T08:57:49Z)
• The Race to Efficiency: A New Perspective on AI Scaling Laws [0.0]
  We introduce a time- and efficiency-aware framework that extends classical AI scaling laws.<n>Our model shows that, without ongoing efficiency gains, advanced performance could demand millennia of training or unrealistically large GPU fleets.<n>By formalizing this race to efficiency, we offer a quantitative roadmap for balancing front-loaded GPU investments with incremental improvements across the AI stack.
  arXiv  Detail & Related papers  (2025-01-04T01:45:32Z)
• Predicting Emergent Abilities with Infinite Resolution Evaluation [85.89911520190711]
  We introduce PassUntil, an evaluation strategy with theoretically infinite resolution, through massive sampling in the decoding phase. We predict the performance of the 2.4B model on code generation with merely 0.05% deviation before training starts. We identify a kind of accelerated emergence whose scaling curve cannot be fitted by standard scaling law function.
  arXiv  Detail & Related papers  (2023-10-05T02:35:00Z)
• Scaling Laws Beyond Backpropagation [64.0476282000118]
  We study the ability of Direct Feedback Alignment to train causal decoder-only Transformers efficiently. We find that DFA fails to offer more efficient scaling than backpropagation.
  arXiv  Detail & Related papers  (2022-10-26T10:09:14Z)
• Efficient Few-Shot Object Detection via Knowledge Inheritance [62.36414544915032]
  Few-shot object detection (FSOD) aims at learning a generic detector that can adapt to unseen tasks with scarce training samples. We present an efficient pretrain-transfer framework (PTF) baseline with no computational increment. We also propose an adaptive length re-scaling (ALR) strategy to alleviate the vector length inconsistency between the predicted novel weights and the pretrained base weights.
  arXiv  Detail & Related papers  (2022-03-23T06:24:31Z)
• Powerpropagation: A sparsity inducing weight reparameterisation [65.85142037667065]
  We introduce Powerpropagation, a new weight- parameterisation for neural networks that leads to inherently sparse models. Models trained in this manner exhibit similar performance, but have a distribution with markedly higher density at zero, allowing more parameters to be pruned safely. Here, we combine Powerpropagation with a traditional weight-pruning technique as well as recent state-of-the-art sparse-to-sparse algorithms, showing superior performance on the ImageNet benchmark.
  arXiv  Detail & Related papers  (2021-10-01T10:03:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.