Effective Frontiers: A Unification of Neural Scaling Laws

• URL: http://arxiv.org/abs/2602.02593v1
• Date: Sun, 01 Feb 2026 10:44:46 GMT
• Title: Effective Frontiers: A Unification of Neural Scaling Laws
• Authors: Jiaxuan Zou, Zixuan Gong, Ye Su, Huayi Tang, Yong Liu,
• Abstract summary: We propose a unified framework that abstracts general learning tasks as the progressive coverage of patterns from a long-tail (Zipfian) distribution.<n>We derive the precise scaling laws for $N$, $D$, and $C$, attributing them to capacity, coverage, and optimization bottlenecks.
• Score: 19.808117554175013
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural scaling laws govern the prediction power-law improvement of test loss with respect to model capacity ($N$), datasize ($D$), and compute ($C$). However, existing theoretical explanations often rely on specific architectures or complex kernel methods, lacking intuitive universality. In this paper, we propose a unified framework that abstracts general learning tasks as the progressive coverage of patterns from a long-tail (Zipfian) distribution. We introduce the Effective Frontier ($k_\star$), a threshold in the pattern rank space that separates learned knowledge from the unlearned tail. We prove that reducible loss is asymptotically determined by the probability mass of the tail a resource-dependent frontier truncation. Based on our framework, we derive the precise scaling laws for $N$, $D$, and $C$, attributing them to capacity, coverage, and optimization bottlenecks, respectively. Furthermore, we unify these mechanisms via a Max-Bottleneck principle, demonstrating that the Kaplan and Chinchilla scaling laws are not contradictory, but equilibrium solutions to the same constrained optimization problem under different active bottlenecks.

Related papers

• Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA [50.494504099850325]
  We introduce the Geodesic Hypothesis, positing that token sequences trace geodesics on a smooth semantic manifold and are therefore locally linear.<n>We show this constraint improves signal-to-noise ratio, and preserves diversity by preventing collisions during trajectory.<n>We demonstrate that geometric priors can surpass brute-force scaling.
  arXiv  Detail & Related papers  (2026-02-26T04:45:07Z)
• Towards Robust Scaling Laws for Optimizers [89.21160945066737]
  Empirical scaling laws are widely used to predict loss as model size and training data grow.<n>We show that Chinchilla-style scaling laws emerge naturally as a result of loss decomposition into irreducible, approximation, and optimization errors.
  arXiv  Detail & Related papers  (2026-02-07T21:40:33Z)
• Predicting and improving test-time scaling laws via reward tail-guided search [11.49701649103495]
  Test-time scaling has emerged as a critical avenue for enhancing the reasoning capabilities of Large Language Models.<n>We propose new methodologies to predict and improve scaling properties via tail-guided search.<n>By estimating the tail distribution of rewards, our method predicts the scaling law of LLMs without the need for exhaustive evaluations.
  arXiv  Detail & Related papers  (2026-02-01T23:40:25Z)
• Learning Shrinks the Hard Tail: Training-Dependent Inference Scaling in a Solvable Linear Model [2.7074235008521246]
  We analyze neural scaling laws in a solvable model of last-layer fine-tuning where targets have intrinsic, instance-heterogeneous difficulty.<n>We show that learning shrinks the hard tail'' of the error distribution.
  arXiv  Detail & Related papers  (2026-01-07T10:00:17Z)
• Compute-Optimal LLMs Provably Generalize Better With Scale [102.29926217670926]
  We develop generalization bounds on the pretraining objective of large language models (LLMs) in the compute-optimal regime.<n>We introduce a novel, fully empirical Freedman-type martingale concentration that tightens existing bounds by accounting for the variance of the loss function.<n>We produce a scaling law for the generalization gap, with bounds that become predictably stronger with scale.
  arXiv  Detail & Related papers  (2025-04-21T16:26:56Z)
• Selecting Large Language Model to Fine-tune via Rectified Scaling Law [74.84096546112215]
  Given constrained resources, fine-tuning all models and making selections afterward is unrealistic. We find that the fine-tuning scaling curve includes not just the well-known "power phase" but also the previously unobserved "pre-power phase" By leveraging our law, we propose a novel LLM selection algorithm that selects the near-optimal model with hundreds of times less resource consumption.
  arXiv  Detail & Related papers  (2024-02-04T01:55:00Z)
• A Pseudo-Semantic Loss for Autoregressive Models with Logical Constraints [87.08677547257733]
  Neuro-symbolic AI bridges the gap between purely symbolic and neural approaches to learning. We show how to maximize the likelihood of a symbolic constraint w.r.t the neural network's output distribution. We also evaluate our approach on Sudoku and shortest-path prediction cast as autoregressive generation.
  arXiv  Detail & Related papers  (2023-12-06T20:58:07Z)
• Scaling Laws Beyond Backpropagation [64.0476282000118]
  We study the ability of Direct Feedback Alignment to train causal decoder-only Transformers efficiently. We find that DFA fails to offer more efficient scaling than backpropagation.
  arXiv  Detail & Related papers  (2022-10-26T10:09:14Z)
• High-dimensional limit theorems for SGD: Effective dynamics and critical scaling [6.950316788263433]
  We prove limit theorems for the trajectories of summary statistics of gradient descent (SGD) We show a critical scaling regime for the step-size, below which the effective ballistic dynamics matches gradient flow for the population loss. About the fixed points of this effective dynamics, the corresponding diffusive limits can be quite complex and even degenerate.
  arXiv  Detail & Related papers  (2022-06-08T17:42:18Z)
• Scaling Laws for Deep Learning [1.90365714903665]
  In this thesis we take a systematic approach to address the algorithmic and methodological limitations at the root of these costs. We first demonstrate that deep learning training and pruning are predictable and governed by scaling laws. We then show through the exploration of a noiseless realizable case that DL is in fact dominated by error sources very far from the lower error limit.
  arXiv  Detail & Related papers  (2021-08-17T15:37:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.