Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws

• URL: http://arxiv.org/abs/2606.06888v2
• Date: Tue, 09 Jun 2026 06:01:51 GMT
• Title: Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws
• Abstract summary: We study data-constrained pretraining along two axes, regularization and scaling.<n>We propose SoftQ, a scaling law that couples model size and data size to capture their interaction under repeated data.
• Score: 41.26396818761427
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Classical scaling laws for language model pretraining balance model size against training dataset size under a fixed compute budget, assuming abundant data and a single pass over the corpus. As training compute grows faster than the supply of natural language data, pretraining is likely to enter a data-constrained, compute-rich regime where models train for multiple epochs over a finite dataset. We study data-constrained pretraining along two axes, regularization and scaling. For regularization, we study masked-input regularization (MIR), an auxiliary next-token prediction loss on randomly masked inputs. MIR tests whether the random masking central to diffusion language models can benefit autoregressive pretraining without architectural changes or inference overhead. Across 72M to 1.4B parameter models, we find that MIR added on top of strong weight decay improves validation loss over autoregressive strong-weight-decay-only models, with downstream gains at 1.4B. For scaling, we propose SoftQ, a scaling law that couples model size and data size to capture their interaction under repeated data. Classical alternatives such as the Chinchilla law use an additive form that decouples these terms, making them misspecified in the data-constrained regime. We find that SoftQ fits data-constrained experiments substantially better than these alternatives, and estimates MIR's gains as equivalent to roughly 1.3 times as much unique training data. We release our code at https://github.com/yixinw-lab/dc_pretrain.

Related papers

• Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretraining [6.5664347332837]
  Language model pretraining is shifting toward a data-constrained, compute-abundant regime.<n>We introduce three categories of augmentation for AR pretraining: token-level noise, sequence permutations, and target offset prediction.<n>We find that individual augmentations delay overfitting and lower validation loss relative to the baseline.
  arXiv  Detail & Related papers  (2026-06-15T05:48:57Z)
• Fast Unlearning at Scale via Margin Self-Correction [52.46927918952516]
  Language-model unlearning updates a trained model to behave as if it had not seen selected training examples.<n>MASC achieves a competitive forget-retain trade-off at a fraction of the computational cost of existing baselines.
  arXiv  Detail & Related papers  (2026-06-01T21:49:54Z)
• Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World [4.6607470395685215]
  Chinchilla's $L = E + A/N+ B/D$ is the dominant scaling law form, but it diverges as unique data shrinks instead of saturating at the uninformed baseline.<n>We propose a closed-form extension that decomposes loss into undercapacity, undertraining, and overfitting terms.<n>We validate it on four multi-epoch experiments across vision, scientific ML, and language domains.
  arXiv  Detail & Related papers  (2026-05-09T22:07:01Z)
• Prescriptive Scaling Laws for Data Constrained Training [21.261009598515066]
  Training compute is increasingly outpacing the availability of high-quality data.<n>Chinchilla scaling law assumes every training token is unique.<n>Our scaling law yields qualitatively new compute-optimal allocation advice.
  arXiv  Detail & Related papers  (2026-05-02T23:14:20Z)
• Diffusion Language Models are Super Data Learners [61.721441061210896]
  When unique data is limited, diffusion language models (DLMs) consistently surpass autoregressive (AR) models by training for more epochs.<n>We attribute the gains to three compounding factors: (1) any-order modeling, (2) super-dense compute from iterative bidirectional denoising, and (3) built-in Monte Carlo augmentation.
  arXiv  Detail & Related papers  (2025-11-05T08:17:42Z)
• The interplay between domain specialization and model size [8.653321928148547]
  We investigate the interplay between domain and model size during continued pretraining under compute-constrained scenarios.<n>Our goal is to identify an optimal training regime for this scenario and detect patterns in this interplay that can be generalized across different model sizes and domains.
  arXiv  Detail & Related papers  (2025-01-03T19:28:53Z)
• A Dynamical Model of Neural Scaling Laws [79.59705237659547]
  We analyze a random feature model trained with gradient descent as a solvable model of network training and generalization. Our theory shows how the gap between training and test loss can gradually build up over time due to repeated reuse of data.
  arXiv  Detail & Related papers  (2024-02-02T01:41:38Z)
• Scaling Data-Constrained Language Models [133.2083255645999]
  We investigate scaling language models in data-constrained regimes.<n>We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data.<n>We propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters.
  arXiv  Detail & Related papers  (2023-05-25T17:18:55Z)
• AI Model Disgorgement: Methods and Choices [127.54319351058167]
  We introduce a taxonomy of possible disgorgement methods that are applicable to modern machine learning systems. We investigate the meaning of "removing the effects" of data in the trained model in a way that does not require retraining from scratch.
  arXiv  Detail & Related papers  (2023-04-07T08:50:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.