Towards Engineering Scaling Laws with Pretraining Data Composition

• URL: http://arxiv.org/abs/2606.19781v1
• Date: Thu, 18 Jun 2026 04:32:06 GMT
• Title: Towards Engineering Scaling Laws with Pretraining Data Composition
• Abstract summary: In particle physics, fundamental physics has high-fidelity simulators that produce synthetic data cheaply.<n>This favors scaling regimes where additional data is cheaper than additional parameters, and the pretraining dataset itself is engineered to influence the scaling.<n>For the task of classifying hadronic jets produced in collisions of high-energy particle beams, we show that the scaling behavior can be engineered towards requiring more data rather than larger models.
• Score: 0.9461871831924723
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural scaling laws describe how model performance improves as a power law in compute, model size, and dataset size. While well-established for large language models, these relationships are emerging for large models in particle physics. As with language, empirical studies show that the performance scales as a power law. However, unlike natural language or image domains, fundamental physics has high-fidelity simulators that produce synthetic data cheaply. This favors scaling regimes where additional data is cheaper than additional parameters, and allows the pretraining dataset itself to be engineered to influence the scaling. For the task of classifying hadronic jets produced in collisions of high-energy particle beams, we show that the scaling behavior can be engineered towards requiring more data rather than larger models by inclusion of pretraining data which is more diverse and better aligned with the downstream classification task.

Related papers

• Scaling Laws of Global Weather Models [57.27583619011988]
  We investigate the relationship between model performance (validation loss) and three key factors: model size, dataset size, and compute budget.<n>Across a range of models, we find that Aurora exhibits the strongest data-scaling behavior.<n>Our compute-optimal analysis indicates that, under fixed compute budgets, allocating resources to longer training durations yields greater performance gains than increasing model size.
  arXiv  Detail & Related papers  (2026-02-26T12:57:38Z)
• Neural Scaling Laws for Boosted Jet Tagging [0.22399170518036912]
  scaling compute, through joint increases in model capacity and dataset size, is the primary driver of performance in modern machine learning.<n>We derive compute optimal scaling laws and identify an effective performance limit that can be consistently approached through increased compute.<n>We then study how the scaling coefficients and performance limits vary with the choice of input features and particle multiplicity.
  arXiv  Detail & Related papers  (2026-02-17T18:13:01Z)
• On the origin of neural scaling laws: from random graphs to natural language [10.425020020850402]
  We study scaling laws for transformers trained to predict random walks (bigrams) on graphs with tunable complexity.<n>We consider dialing down the complexity of natural language systematically, by training on sequences sampled from increasingly simplified generative language models.<n>Our results also include scaling laws obtained from training on random walks on random graphs drawn from Erds-Renyi and scale-free Barabsi-Albert ensembles.
  arXiv  Detail & Related papers  (2026-01-15T18:46:09Z)
• Scaling Laws for Pre-training Agents and World Models [22.701210075508147]
  Performance of embodied agents has been shown to improve by increasing model parameters, dataset size, and compute.<n>This paper characterizes the role of scale in these tasks more precisely.
  arXiv  Detail & Related papers  (2024-11-07T04:57:40Z)
• A Hitchhiker's Guide to Scaling Law Estimation [56.06982415792523]
  Scaling laws predict the loss of a target machine learning model by extrapolating from easier-to-train models with fewer parameters or smaller training sets.<n>We estimate more than 1000 scaling laws, then derive a set of best practices for estimating scaling laws in new model families.
  arXiv  Detail & Related papers  (2024-10-15T17:59:10Z)
• Observational Scaling Laws and the Predictability of Language Model Performance [51.2336010244645]
  We propose an observational approach that bypasses model training and instead builds scaling laws from 100 publically available models. We show that several emergent phenomena follow a smooth, sigmoidal behavior and are predictable from small models. We show how to predict the impact of post-training interventions like Chain-of-Thought and Self-Consistency as language model capabilities continue to improve.
  arXiv  Detail & Related papers  (2024-05-17T17:49:44Z)
• A Tale of Tails: Model Collapse as a Change of Scaling Laws [11.6055501181235]
  We ask: How will the scaling laws change in the inevitable regime where synthetic data makes its way into the training corpus? We develop a theoretical framework of model collapse through the lens of scaling laws. We discover a wide range of decay phenomena, analyzing loss of scaling, shifted scaling with number of generations, the ''un-learning" of skills, and grokking when mixing human and synthesized data.
  arXiv  Detail & Related papers  (2024-02-10T21:06:34Z)
• Scaling Laws for Sparsely-Connected Foundation Models [70.41266138010657]
  We explore the impact of parameter sparsity on the scaling behavior of Transformers trained on massive datasets. We identify the first scaling law describing the relationship between weight sparsity, number of non-zero parameters, and amount of training data.
  arXiv  Detail & Related papers  (2023-09-15T16:29:27Z)
• Scaling Laws Do Not Scale [54.72120385955072]
  Recent work has argued that as the size of a dataset increases, the performance of a model trained on that dataset will increase. We argue that this scaling law relationship depends on metrics used to measure performance that may not correspond with how different groups of people perceive the quality of models' output. Different communities may also have values in tension with each other, leading to difficult, potentially irreconcilable choices about metrics used for model evaluations.
  arXiv  Detail & Related papers  (2023-07-05T15:32:21Z)
• A Solvable Model of Neural Scaling Laws [72.8349503901712]
  Large language models with a huge number of parameters, when trained on near internet-sized number of tokens, have been empirically shown to obey neural scaling laws. We propose a statistical model -- a joint generative data model and random feature model -- that captures this neural scaling phenomenology. Key findings are the manner in which the power laws that occur in the statistics of natural datasets are extended by nonlinear random feature maps.
  arXiv  Detail & Related papers  (2022-10-30T15:13:18Z)
• Data Scaling Laws in NMT: The Effect of Noise and Architecture [59.767899982937756]
  We study the effect of varying the architecture and training data quality on the data scaling properties of Neural Machine Translation (NMT) We find that the data scaling exponents are minimally impacted, suggesting that marginally worse architectures or training data can be compensated for by adding more data.
  arXiv  Detail & Related papers  (2022-02-04T06:53:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.