Small-Scale Experiments: Are We There Yet?

• URL: http://arxiv.org/abs/2608.11859v1
• Date: Wed, 12 Aug 2026 09:47:01 GMT
• Title: Small-Scale Experiments: Are We There Yet?
• Abstract summary: Researchers have found scaling laws unreliable at small scales.<n>We show this is not the case: the confounding factor is hyper parameter sensitivity.<n>We develop a new methodology for model-centric research.
• Score: 45.30146986557193
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scaling laws promised cost-effective experiments; six years later, they have yet to fully deliver. Instead, researchers have found them unreliable at small scales (starting at 4M parameters) and concluded that sizable models cannot be avoided. We show this is not the case: the confounding factor is hyperparameters. Small models are highly sensitive, but hyperparameter sensitivity fades with scale. This small-scale sensitivity makes scaling laws easy to miss because they only emerge on the fully tuned frontier, and reaching that frontier requires an extensive search far beyond what most ever run. By ablating the basic scaling law recipe, we show well-tuned hyperparameters matter more than any other ingredient. Further, we reveal why those hyperparameters become easier to find: as scale increases, the hyperparameter loss surface becomes lower dimensional. Nevertheless while scaling laws exist in small models, extrapolation hits statistical limitations. A holistic approach is required. Synthesizing our insights with the recent literature, we develop a new methodology for model-centric research and demonstrate it on a question that once took the field years to settle: where to place normalization layers in the transformer architecture. From small-scale experiments, we recover the large scale result: pre-normalization works better as models grow in size. With the right tools and a better understanding, small-scale experiments can deliver on scaling laws' long-awaited promise.

Related papers

• On the Entropy Calibration of Language Models [52.47557449370603]
  We study the problem of entropy calibration, which asks whether a language model's entropy over generations matches its log loss on human text.<n>We find that the observed scaling behavior is similar to what is predicted by the simplified setting.<n>We prove that it is possible, if we assume access to a black box which can fit models to predict the future entropy of text.
  arXiv  Detail & Related papers  (2025-11-15T00:33:03Z)
• Hyperparameter Loss Surfaces Are Simple Near their Optima [50.74035795378814]
  We develop a technique based on random search to uncover the complex loss surface.<n>Within this regime, the best scores from random search take on a new distribution we discover.<n>From these features, we derive a new law for random search that can explain and extrapolate its convergence.<n>These new tools enable new analyses, such as confidence intervals for the best possible performance.
  arXiv  Detail & Related papers  (2025-10-03T04:52:27Z)
• Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check [44.088564825871345]
  Downstream scaling laws aim to predict task performance at larger scales from the model's performance at smaller scales.<n>We find that predictable scaling only occurs in a minority of cases.<n>Seemingly benign changes to the experimental setting can completely change the scaling behavior.
  arXiv  Detail & Related papers  (2025-07-01T15:52:55Z)
• Gemstones: A Model Suite for Multi-Faceted Scaling Laws [72.09352034700738]
  We release the Gemstones: an open-source scaling law dataset.<n>This dataset consists of over 4000 checkpoints from transformers with up to 2 billion parameters.<n>We find that the prescriptions of scaling laws can be highly sensitive to the experimental design process.
  arXiv  Detail & Related papers  (2025-02-07T18:09:38Z)
• A Tale of Tails: Model Collapse as a Change of Scaling Laws [11.6055501181235]
  We ask: How will the scaling laws change in the inevitable regime where synthetic data makes its way into the training corpus? We develop a theoretical framework of model collapse through the lens of scaling laws. We discover a wide range of decay phenomena, analyzing loss of scaling, shifted scaling with number of generations, the ''un-learning" of skills, and grokking when mixing human and synthesized data.
  arXiv  Detail & Related papers  (2024-02-10T21:06:34Z)
• Predicting Emergent Abilities with Infinite Resolution Evaluation [85.89911520190711]
  We introduce PassUntil, an evaluation strategy with theoretically infinite resolution, through massive sampling in the decoding phase. We predict the performance of the 2.4B model on code generation with merely 0.05% deviation before training starts. We identify a kind of accelerated emergence whose scaling curve cannot be fitted by standard scaling law function.
  arXiv  Detail & Related papers  (2023-10-05T02:35:00Z)
• nanoLM: an Affordable LLM Pre-training Benchmark via Accurate Loss Prediction across Scales [65.01417261415833]
  We present an approach to predict the pre-training loss based on our observations that Maximal Update Parametrization (muP) enables accurate fitting of scaling laws. With around 14% of the one-time pre-training cost, we can accurately forecast the loss for models up to 52B. Our goal with nanoLM is to empower researchers with limited resources to reach meaningful conclusions on large models.
  arXiv  Detail & Related papers  (2023-04-14T00:45:01Z)
• Is the Number of Trainable Parameters All That Actually Matters? [2.624902795082451]
  We investigate ways to tentatively cheat scaling laws, and train larger models for cheaper. We find that the scaling relationship between test loss and compute depends only on the actual number of trainable parameters.
  arXiv  Detail & Related papers  (2021-09-24T12:43:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.