Skaling: Chinchilla's Exponents Meet Kaplan's Coupling

• URL: http://arxiv.org/abs/2608.07222v1
• Date: Fri, 07 Aug 2026 13:38:51 GMT
• Title: Skaling: Chinchilla's Exponents Meet Kaplan's Coupling
• Abstract summary: We introduce the Skaling law, a functional form that couples model capacity and data through a single interaction exponent.<n>This simple extension reduces the Mean Absolute Percentage Error (MAPE) by 1.5-3x across both regimes.<n>By enabling reliable performance prediction from small-scale experiments, the Skaling law provides a more robust and resource-efficient framework.
• Score: 19.21066155468739
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural scaling laws are foundational for language model development, yet standard formulations systematically under- and overestimate loss at data-scarce and overtraining extremes. This failure originates in the underlying assumption that model size and training data impact the loss independently. To address this, we introduce the Skaling law, a generalized functional form that couples model capacity and data through a single interaction exponent. This simple extension reduces the Mean Absolute Percentage Error (MAPE) by 1.5-3x across both interpolation and extrapolation regimes. When paired with a sparse grid strategy restricted to low-compute regimes, the Skaling law achieves accurate full-grid extrapolation using approximately 10x less compute than uniform sweeps. By enabling reliable performance prediction from small-scale experiments, the Skaling law provides a more robust and resource-efficient framework for allocating compute budgets in next-generation model training.

Related papers

• Explaining Data Mixing Scaling Laws [2.1378946778058796]
  We propose a unified framework to explain the underlying mechanics of data mixing.<n>We identify two key factors that govern the domain losses of models trained on different data mixtures.<n>Our model successfully extrapolates across scales, predicting highly effective mixtures for large, unseen scales.
  arXiv  Detail & Related papers  (2026-06-06T13:31:38Z)
• Asymmetric Scaling Laws from Sparse Features [2.7838875470471858]
  We introduce a model for neural scaling laws under sparse activations.<n>Test loss is often dominated by rare coordinates that are never observed in the training input.<n>We show that the sparsity-induced effect persists under nonlinear activations.
  arXiv  Detail & Related papers  (2026-05-22T13:00:52Z)
• Prescriptive Scaling Laws for Data Constrained Training [21.261009598515066]
  Training compute is increasingly outpacing the availability of high-quality data.<n>Chinchilla scaling law assumes every training token is unique.<n>Our scaling law yields qualitatively new compute-optimal allocation advice.
  arXiv  Detail & Related papers  (2026-05-02T23:14:20Z)
• Neural Neural Scaling Laws [40.38002195911611]
  We propose Neural Neural Scaling Laws (NeuNeu), a neural network that frames scaling law prediction as time-series extrapolation.<n>NeuNeu achieves 2.04% mean absolute error in predicting model accuracy on 66 downstream tasks.<n>Our work suggests that predicting downstream scaling laws directly from data outperforms parametric alternatives.
  arXiv  Detail & Related papers  (2026-01-27T17:38:11Z)
• Nonparametric Data Attribution for Diffusion Models [57.820618036556084]
  Data attribution for generative models seeks to quantify the influence of individual training examples on model outputs.<n>We propose a nonparametric attribution method that operates entirely on data, measuring influence via patch-level similarity between generated and training images.
  arXiv  Detail & Related papers  (2025-10-16T03:37:16Z)
• Unified Scaling Laws for Compressed Representations [69.72517034565467]
  We investigate whether a unified scaling framework can accurately predict model performance when training occurs over various compressed representations.<n>Our main finding is demonstrating both theoretically and empirically that there exists a simple "capacity" metric.<n>We extend our formulation to directly compare the accuracy potential of different compressed formats, and to derive better algorithms for training over sparse-quantized formats.
  arXiv  Detail & Related papers  (2025-06-02T16:52:51Z)
• The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws [51.608402959163925]
  We present the first systematic exploration of optimal sparse pre-training configurations for large language models.<n>We find that initiating pruning at 25% of total training compute and concluding at 75% achieves near-optimal final evaluation loss.<n>We propose a new scaling law that modifies the Chinchilla scaling law to use the average parameter count over pre-training.
  arXiv  Detail & Related papers  (2025-01-21T20:23:22Z)
• A Meta-Learning Approach to Predicting Performance and Data Requirements [163.4412093478316]
  We propose an approach to estimate the number of samples required for a model to reach a target performance. We find that the power law, the de facto principle to estimate model performance, leads to large error when using a small dataset. We introduce a novel piecewise power law (PPL) that handles the two data differently.
  arXiv  Detail & Related papers  (2023-03-02T21:48:22Z)
• Distributed Pruning Towards Tiny Neural Networks in Federated Learning [12.63559789381064]
  FedTiny is a distributed pruning framework for federated learning. It generates specialized tiny models for memory- and computing-constrained devices. It achieves an accuracy improvement of 2.61% while significantly reducing the computational cost by 95.91%.
  arXiv  Detail & Related papers  (2022-12-05T01:58:45Z)
• A Solvable Model of Neural Scaling Laws [72.8349503901712]
  Large language models with a huge number of parameters, when trained on near internet-sized number of tokens, have been empirically shown to obey neural scaling laws. We propose a statistical model -- a joint generative data model and random feature model -- that captures this neural scaling phenomenology. Key findings are the manner in which the power laws that occur in the statistics of natural datasets are extended by nonlinear random feature maps.
  arXiv  Detail & Related papers  (2022-10-30T15:13:18Z)
• Real-Time Regression with Dividing Local Gaussian Processes [62.01822866877782]
  Local Gaussian processes are a novel, computationally efficient modeling approach based on Gaussian process regression. Due to an iterative, data-driven division of the input space, they achieve a sublinear computational complexity in the total number of training points in practice. A numerical evaluation on real-world data sets shows their advantages over other state-of-the-art methods in terms of accuracy as well as prediction and update speed.
  arXiv  Detail & Related papers  (2020-06-16T18:43:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.