Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation

• URL: http://arxiv.org/abs/2605.08541v2
• Date: Tue, 12 May 2026 22:27:47 GMT
• Title: Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation
• Abstract summary: Neural scaling laws approximate a language model's loss as a power-law function of parameter count $N$ and token count $D$.<n>We show that a collinear design induces an inherent ill-conditioning in the Gauss-Newton least-squares problem.<n>We prove this for four scaling-law formalisms and derive a closed-form TPP-diversity threshold that is necessary and sufficient for well-conditioned estimation.
• Score: 45.56738584872585
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural scaling laws approximate a language model's loss as a power-law function of parameter count $N$ and token count $D$. Following Chinchilla-style compute-optimal training, many studies fit scaling laws from runs performed under a fixed tokens-per-parameter (TPP) ratio $k$ and set $D = kN$. We show that this collinear design, combined with the empirically common near-equality of the exponents governing $N$ and $D$, induces an inherent ill-conditioning in the Gauss-Newton least-squares problem: the condition number of the design grows as the inverse square of the gap between the $N$ and $D$-exponents. The scale coefficients become practically unidentifiable, with confidence intervals inflating by an order of magnitude or more, yielding a ``sloppy'' model whose extrapolations degrade sharply off the training ray. We prove this for four scaling-law formalisms and derive a closed-form TPP-diversity threshold that is necessary and sufficient for well-conditioned estimation. Empirically, non-collinear designs outperform collinear ones on held-out splits with a 97.3\% win rate across four laws, five corpora, multiple floating point precision modes. We further show the degeneracy is rooted in Jacobian geometry and is not an artifact of the loss function: any smooth estimation objective whose curvature involves the Jacobian inherits the same ill-conditioning.

Related papers

• Recovering Nonlinear Functions of Latent Variables: A Plausible-Value Neural Network Framework [0.0]
  When factor scores replace true latent scores in nonlinear prediction, measurement error attenuates the recoverable variance by $k$.<n>The bound recovery governs the latent-scale function, not prediction of the outcome from observed indicators.<n>An 18-condition simulation supports both predictions.
  arXiv  Detail & Related papers  (2026-08-19T08:27:55Z)
• Nishimori Threshold Estimation for Bayesian Inference and $\mathbb{Z}_q$ Surface Code Decoding [0.4910937238451485]
  We introduce an analytical estimate of error thresholds falling into the Nishimori universality class via a Fourier--Walsh projection scheme.<n>We find that the threshold values not only accurately agree with recent decohered-$mathbb Z_q$-toric-code numerics, but are found to satisfy the Gilbert-Varshamov self-dual entropy relation.<n>Our approach thereby points to a deeper connection between the clean and Born-disordered models, while allowing for instant analytical estimates of error thresholds for a variety of stabilizer codes.
  arXiv  Detail & Related papers  (2026-07-20T18:00:00Z)
• Representational Capacity: Geometric Limits on Feature Representation in Transformer Language Models [51.56484100374058]
  We develop a framework for estimating how many near-orthogonal directions a model can support.<n>Applying this metric across dozens of open-source models reveals two classes: models with high $varepsilon$ whose embeddings lack near-orthogonal structure, and models with low $varepsilon$ that maintain it.
  arXiv  Detail & Related papers  (2026-06-01T18:28:56Z)
• Polynomial Context-Truncation Sensitivity in Autoregressive Language Models: Sequential Wyner-Ziv Bounds for KV Cache Compression [0.0]
  We study the rate-distortion limits of online KV cache compression in autoregressive language models.<n>We find that the next-token distribution's sensitivity to context truncation decays emphpolynomially rather than emphgeometrically
  arXiv  Detail & Related papers  (2026-05-24T13:54:13Z)
• Effective Frontiers: A Unification of Neural Scaling Laws [19.808117554175013]
  We propose a unified framework that abstracts general learning tasks as the progressive coverage of patterns from a long-tail (Zipfian) distribution.<n>We derive the precise scaling laws for $N$, $D$, and $C$, attributing them to capacity, coverage, and optimization bottlenecks.
  arXiv  Detail & Related papers  (2026-02-01T10:44:46Z)
• Newton-Puiseux Analysis for Interpretability and Calibration of Complex-Valued Neural Networks [0.0]
  Complex neural networks (CVNNs) are suitable for handling phase-sensitive signals, including electrocardiography (ECG), radar/sonar, and wireless in-phase/quadrature (I/Q) streams.<n>We present a Newton-Puiseux framework that examines the emphlocal decision geometry of a trained CVNN by fitting a small, kink-aware surrogate.<n>Our phase-aware analysis identifies sensitive directions and enhances Expected Error in two case studies beyond a controlled $C2$ synthetic benchmark.
  arXiv  Detail & Related papers  (2025-04-27T09:37:07Z)
• Convergence Rate Analysis of LION [54.28350823319057]
  LION converges iterations of $cal(sqrtdK-)$ measured by gradient Karush-Kuhn-T (sqrtdK-)$. We show that LION can achieve lower loss and higher performance compared to standard SGD.
  arXiv  Detail & Related papers  (2024-11-12T11:30:53Z)
• Multivariate root-n-consistent smoothing parameter free matching estimators and estimators of inverse density weighted expectations [51.000851088730684]
  We develop novel modifications of nearest-neighbor and matching estimators which converge at the parametric $sqrt n $-rate.<n>We stress that our estimators do not involve nonparametric function estimators and in particular do not rely on sample-size dependent parameters smoothing.
  arXiv  Detail & Related papers  (2024-07-11T13:28:34Z)
• Universal Online Learning with Gradient Variations: A Multi-layer Online Ensemble Approach [57.92727189589498]
  We propose an online convex optimization approach with two different levels of adaptivity. We obtain $mathcalO(log V_T)$, $mathcalO(d log V_T)$ and $hatmathcalO(sqrtV_T)$ regret bounds for strongly convex, exp-concave and convex loss functions.
  arXiv  Detail & Related papers  (2023-07-17T09:55:35Z)
• Reduced Contraction Costs of Corner-Transfer Methods for PEPS [0.0]
  We propose a pair of approximations that allows the leading order computational cost of contracting an infinite projected entangled-pair state to be reduced. The improvement in computational cost enables us to perform large bond dimension calculations, extending its potential to solve challenging problems.
  arXiv  Detail & Related papers  (2023-06-14T02:54:12Z)
• Breaking the Sample Complexity Barrier to Regret-Optimal Model-Free Reinforcement Learning [52.76230802067506]
  A novel model-free algorithm is proposed to minimize regret in episodic reinforcement learning. The proposed algorithm employs an em early-settled reference update rule, with the aid of two Q-learning sequences. The design principle of our early-settled variance reduction method might be of independent interest to other RL settings.
  arXiv  Detail & Related papers  (2021-10-09T21:13:48Z)
• Robust Implicit Networks via Non-Euclidean Contractions [63.91638306025768]
  Implicit neural networks show improved accuracy and significant reduction in memory consumption. They can suffer from ill-posedness and convergence instability. This paper provides a new framework to design well-posed and robust implicit neural networks.
  arXiv  Detail & Related papers  (2021-06-06T18:05:02Z)
• Estimation in Tensor Ising Models [5.161531917413708]
  We consider the problem of estimating the natural parameter of the $p$-tensor Ising model given a single sample from the distribution on $N$ nodes. In particular, we show the $sqrt N$-consistency of the MPL estimate in the $p$-spin Sherrington-Kirkpatrick (SK) model. We derive the precise fluctuations of the MPL estimate in the special case of the $p$-tensor Curie-Weiss model.
  arXiv  Detail & Related papers  (2020-08-29T00:06:58Z)
• A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-$\ell_1$-Norm Interpolated Classifiers [3.167685495996986]
  This paper establishes a precise high-dimensional theory for boosting on separable data. Under a class of statistical models, we provide an exact analysis of the universality error of boosting. We also explicitly pin down the relation between the boosting test error and the optimal Bayes error.
  arXiv  Detail & Related papers  (2020-02-05T00:24:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.