Bridging Compute- and Data-Optimal Pretraining

• URL: http://arxiv.org/abs/2607.25271v1
• Date: Tue, 28 Jul 2026 04:18:49 GMT
• Title: Bridging Compute- and Data-Optimal Pretraining
• Abstract summary: We propose a unified framework that bridges compute-optimal scaling, where data scales freely with compute, and data-optimal scaling, where the corpus is fixed while compute can grow without bound.<n>CD scaling extends classical scaling laws by introducing a token-effectiveness function, $$, which quantifies the value of a derived token-produced.
• Score: 18.911780119556663
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Classical compute-optimal scaling laws assume an unbounded supply of fresh pretraining data, yet pretraining is increasingly entering a regime in which compute grows faster than the availability of high-quality data. We propose Compute-Data (CD) scaling laws, a unified framework that bridges compute-optimal scaling, where data scales freely with compute, and data-optimal scaling, where the corpus is fixed while compute can grow without bound. CD scaling extends classical scaling laws by introducing a token-effectiveness function, $η$, which quantifies the value of a derived token-produced, for example, through multi-epoch repetition or paraphrasing-relative to a fresh token, ranging from a perfect substitute to having no value. We fit $η$ for two data-expansion strategies, multi-epoch repetition and paraphrasing, across model sizes from 14M to 600M parameters using the Dolma-3 corpus. We find that token effectiveness is far from constant: it depends jointly on model size, the tokens-per-parameter ratio, and the amount of derived data, and it saturates as the corpus is expanded. The functional form of $η$ implies diminishing returns when substituting compute for data as either model size or data availability increases. It also partitions training into three operational regimes---compute-bound, data-bound, and model-bound---and shows that classical compute-optimal allocation is suboptimal across most practically relevant settings.

Related papers

• Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws [41.26396818761427]
  We study data-constrained pretraining along two axes, regularization and scaling.<n>We propose SoftQ, a scaling law that couples model size and data size to capture their interaction under repeated data.
  arXiv  Detail & Related papers  (2026-06-05T04:10:09Z)
• Prescriptive Scaling Laws for Data Constrained Training [21.261009598515066]
  Training compute is increasingly outpacing the availability of high-quality data.<n>Chinchilla scaling law assumes every training token is unique.<n>Our scaling law yields qualitatively new compute-optimal allocation advice.
  arXiv  Detail & Related papers  (2026-05-02T23:14:20Z)
• Pre-training under infinite compute [87.02472603429936]
  We show that data-constrained approaches of increasing epoch count and parameter count eventually overfit.<n>We identify that ensembling independently trained models achieves a significantly lower loss asymptote than the regularized recipe.<n>Our results show that simple algorithmic improvements can enable significantly more data-efficient pre-training in a compute-rich future.
  arXiv  Detail & Related papers  (2025-09-18T09:36:23Z)
• Value-Based Deep RL Scales Predictably [100.21834069400023]
  We show that value-based off-policy RL methods are predictable despite community lore regarding their pathological behavior.<n>We validate our approach using three algorithms: SAC, BRO, and PQL on DeepMind Control, OpenAI gym, and IsaacGym.
  arXiv  Detail & Related papers  (2025-02-06T18:59:47Z)
• The interplay between domain specialization and model size [8.653321928148547]
  We investigate the interplay between domain and model size during continued pretraining under compute-constrained scenarios.<n>Our goal is to identify an optimal training regime for this scenario and detect patterns in this interplay that can be generalized across different model sizes and domains.
  arXiv  Detail & Related papers  (2025-01-03T19:28:53Z)
• gzip Predicts Data-dependent Scaling Laws [2.5461535398221478]
  We generate training datasets of varying complexities by modulating the syntactic properties of a PCFG. We propose a new data-dependent scaling law for LM's that accounts for the training data's gzip-compressibility.
  arXiv  Detail & Related papers  (2024-05-26T20:33:08Z)
• Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic [99.3682210827572]
  Vision-language models (VLMs) are trained for thousands of GPU hours on carefully curated web datasets. Data curation strategies are typically developed agnostic of the available compute for training. We introduce neural scaling laws that account for the non-homogeneous nature of web data.
  arXiv  Detail & Related papers  (2024-04-10T17:27:54Z)
• A Dynamical Model of Neural Scaling Laws [79.59705237659547]
  We analyze a random feature model trained with gradient descent as a solvable model of network training and generalization. Our theory shows how the gap between training and test loss can gradually build up over time due to repeated reuse of data.
  arXiv  Detail & Related papers  (2024-02-02T01:41:38Z)
• Scaling Data-Constrained Language Models [133.2083255645999]
  We investigate scaling language models in data-constrained regimes.<n>We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data.<n>We propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters.
  arXiv  Detail & Related papers  (2023-05-25T17:18:55Z)
• A Solvable Model of Neural Scaling Laws [72.8349503901712]
  Large language models with a huge number of parameters, when trained on near internet-sized number of tokens, have been empirically shown to obey neural scaling laws. We propose a statistical model -- a joint generative data model and random feature model -- that captures this neural scaling phenomenology. Key findings are the manner in which the power laws that occur in the statistics of natural datasets are extended by nonlinear random feature maps.
  arXiv  Detail & Related papers  (2022-10-30T15:13:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.