On the Invariance and Generality of Neural Scaling Laws

• URL: http://arxiv.org/abs/2605.07546v1
• Date: Fri, 08 May 2026 10:21:09 GMT
• Title: On the Invariance and Generality of Neural Scaling Laws
• Abstract summary: Neural scaling laws establish a predictable relationship between model performance and data or compute.<n>fitting one for a new model task pair demands expensive sweeps that typically exhaust the very compute budget the law is meant to economize.<n>This paper poses the research question of how to develop generalizable scaling laws.
• Score: 30.93250651713082
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural scaling laws establish a predictable relationship between model performance and data or compute, offering crucial guidance for resource allocation in new domains and tasks. Yet such laws are most needed precisely where they are hardest to obtain: fitting one for a new model task pair demands expensive sweeps that typically exhaust the very compute budget the law is meant to economize. This paper poses the research question of how to develop generalizable scaling laws: laws fit once on a well-resourced source domain and reliably transported to new domains where running a full sweep is infeasible, which requires a fundamental understanding of when and why scaling properties change. We address this by identifying the right invariants: scaling laws are preserved under bijective (information-preserving) transformations of the data and modified in predictable, information-theoretically grounded ways under non-bijective transformations that lower its information resolution $ρ$: a single axis along which a law fit in one domain can be transported to another. We validate this across language, vision, and speech, and demonstrate two cross-domain applications: predicting scaling for language models trained on electronic health records from laws fit on general text, and predicting time-series classification scaling under varying levels of noise injection, recovering the data-scaling exponents to within $3\%$ error.

Related papers

• Towards Robust Scaling Laws for Optimizers [89.21160945066737]
  Empirical scaling laws are widely used to predict loss as model size and training data grow.<n>We show that Chinchilla-style scaling laws emerge naturally as a result of loss decomposition into irreducible, approximation, and optimization errors.
  arXiv  Detail & Related papers  (2026-02-07T21:40:33Z)
• On the origin of neural scaling laws: from random graphs to natural language [10.425020020850402]
  We study scaling laws for transformers trained to predict random walks (bigrams) on graphs with tunable complexity.<n>We consider dialing down the complexity of natural language systematically, by training on sequences sampled from increasingly simplified generative language models.<n>Our results also include scaling laws obtained from training on random walks on random graphs drawn from Erds-Renyi and scale-free Barabsi-Albert ensembles.
  arXiv  Detail & Related papers  (2026-01-15T18:46:09Z)
• Relative Scaling Laws for LLMs [91.73497548097775]
  Scaling laws describe how language models improve with additional data, parameters, and compute.<n>We introduce relative scaling laws, which track how performance gaps between test distributions evolve with scale.<n>These results show that although scaling improves overall performance, it is not a universal equalizer.
  arXiv  Detail & Related papers  (2025-10-28T16:55:22Z)
• Pretraining Scaling Laws for Generative Evaluations of Language Models [30.6654523997984]
  We show three different scaling laws for fitting pass-at-$k$ on generative evaluations and for predicting pass-at-$k$ of the most expensive model.<n>Our framework provides researchers and practitioners with insights and methodologies to forecast generative performance.
  arXiv  Detail & Related papers  (2025-09-28T18:04:18Z)
• Scaling Laws for Optimal Data Mixtures [36.161727115217964]
  We propose a systematic method to determine the optimal data mixture for any target domain using scaling laws.<n>We validate the universality of these scaling laws by demonstrating their predictive power in three distinct and large-scale settings.
  arXiv  Detail & Related papers  (2025-07-12T21:16:08Z)
• Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks [100.13335639780415]
  Scaling laws often follow the power-law and proposed several variants of power-law functions to predict the scaling behavior at larger scales.<n>Existing methods mostly rely on point estimation and do not quantify uncertainty, which is crucial for real-world applications.<n>In this work, we explore a Bayesian framework based on Prior-data Fitted Networks (PFNs) for neural scaling law extrapolation.
  arXiv  Detail & Related papers  (2025-05-29T03:19:17Z)
• ScaMo: Exploring the Scaling Law in Autoregressive Motion Generation Model [27.532993606576152]
  We introduce a scalable motion generation framework that includes the motion tokenizer MotionQ-VAE and a text FS-VAE transformer.<n>For the first time, we confirm the existence of scaling laws within the context of motion generation.<n>We predict the optimal transformer size, vocabulary size, and data requirements for a compute budget of $1e18$.
  arXiv  Detail & Related papers  (2024-12-19T06:22:19Z)
• Information-Theoretic Foundations for Neural Scaling Laws [20.617552198581024]
  We develop information-theoretic foundations for neural scaling laws. We observe that the optimal relation between data and model size is linear, up to logarithmic factors.
  arXiv  Detail & Related papers  (2024-06-28T02:20:54Z)
• A Solvable Model of Neural Scaling Laws [72.8349503901712]
  Large language models with a huge number of parameters, when trained on near internet-sized number of tokens, have been empirically shown to obey neural scaling laws. We propose a statistical model -- a joint generative data model and random feature model -- that captures this neural scaling phenomenology. Key findings are the manner in which the power laws that occur in the statistics of natural datasets are extended by nonlinear random feature maps.
  arXiv  Detail & Related papers  (2022-10-30T15:13:18Z)
• Scaling Laws Beyond Backpropagation [64.0476282000118]
  We study the ability of Direct Feedback Alignment to train causal decoder-only Transformers efficiently. We find that DFA fails to offer more efficient scaling than backpropagation.
  arXiv  Detail & Related papers  (2022-10-26T10:09:14Z)
• Learning Invariant Representations and Risks for Semi-supervised Domain Adaptation [109.73983088432364]
  We propose the first method that aims to simultaneously learn invariant representations and risks under the setting of semi-supervised domain adaptation (Semi-DA) We introduce the LIRR algorithm for jointly textbfLearning textbfInvariant textbfRepresentations and textbfRisks.
  arXiv  Detail & Related papers  (2020-10-09T15:42:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.