Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretraining

• URL: http://arxiv.org/abs/2606.16246v2
• Date: Fri, 19 Jun 2026 17:02:20 GMT
• Title: Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretraining
• Abstract summary: Language model pretraining is shifting toward a data-constrained, compute-abundant regime.<n>We introduce three categories of augmentation for AR pretraining: token-level noise, sequence permutations, and target offset prediction.<n>We find that individual augmentations delay overfitting and lower validation loss relative to the baseline.
• Score: 6.5664347332837
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As AI labs approach a data ceiling where compute capacity outpaces the rate of new high-quality text generation, language model pretraining is shifting toward a data-constrained, compute-abundant regime that demands productive multi-epoch training on fixed corpora. Standard autoregressive (AR) pretraining overfits severely in this setting, reaching its optimum early and then continuously deteriorating. We investigate training-time data augmentation as a regularizer to mitigate this overfitting and enable productive training for hundreds of epochs on the same data. We introduce three orthogonal categories of augmentation for AR pretraining: token-level noise (masking, random replacement), sequence permutations (right-to-left prediction, Fill-in-the-Middle), and target offset prediction ($x_{t+i}$ for $i > 1$). Through systematic ablations, we find that individual augmentations delay overfitting and lower validation loss relative to the baseline, with random token replacement achieving the best minimum loss among individual methods. Combining augmentation categories further lowers the minimum validation loss. Our experiments demonstrate that data augmentations mitigate AR pretraining's data inefficiency and offer a promising solution to the data-constrained regime~\footnote{All code and data are available at https://github.com/ michaelchen-lab/ data-augmentations-for-pretraining.

Related papers

• Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws [41.26396818761427]
  We study data-constrained pretraining along two axes, regularization and scaling.<n>We propose SoftQ, a scaling law that couples model size and data size to capture their interaction under repeated data.
  arXiv  Detail & Related papers  (2026-06-05T04:10:09Z)
• Prescriptive Scaling Laws for Data Constrained Training [21.261009598515066]
  Training compute is increasingly outpacing the availability of high-quality data.<n>Chinchilla scaling law assumes every training token is unique.<n>Our scaling law yields qualitatively new compute-optimal allocation advice.
  arXiv  Detail & Related papers  (2026-05-02T23:14:20Z)
• Pre-training under infinite compute [87.02472603429936]
  We show that data-constrained approaches of increasing epoch count and parameter count eventually overfit.<n>We identify that ensembling independently trained models achieves a significantly lower loss asymptote than the regularized recipe.<n>Our results show that simple algorithmic improvements can enable significantly more data-efficient pre-training in a compute-rich future.
  arXiv  Detail & Related papers  (2025-09-18T09:36:23Z)
• ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining [53.893792844055106]
  Large language model pretraining is compute-intensive, yet many tokens contribute marginally to learning, resulting in inefficiency.<n>We introduce Selective Efficient Language Modeling, a risk-aware algorithm that improves training efficiency and distributional robustness by performing online token-level batch selection.<n> Experiments on GPT-2 pretraining show that ESLM significantly reduces training FLOPs while maintaining or improving both perplexity and downstream performance compared to baselines.
  arXiv  Detail & Related papers  (2025-05-26T12:23:26Z)
• Reasoning to Learn from Latent Thoughts [61.2395150828168]
  We show that explicitly modeling and inferring the emphlatent thoughts that underlie the text generation process can significantly improve pretraining data efficiency.<n>We show that a 1B LM can bootstrap its performance across at least three iterations and significantly outperform baselines trained on raw data.
  arXiv  Detail & Related papers  (2025-03-24T16:41:23Z)
• Scaling Data-Constrained Language Models [133.2083255645999]
  We investigate scaling language models in data-constrained regimes.<n>We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data.<n>We propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters.
  arXiv  Detail & Related papers  (2023-05-25T17:18:55Z)
• On-the-fly Denoising for Data Augmentation in Natural Language Understanding [101.46848743193358]
  We propose an on-the-fly denoising technique for data augmentation that learns from soft augmented labels provided by an organic teacher model trained on the cleaner original data. Our method can be applied to general augmentation techniques and consistently improve the performance on both text classification and question-answering tasks.
  arXiv  Detail & Related papers  (2022-12-20T18:58:33Z)
• Efficient Augmentation for Imbalanced Deep Learning [8.38844520504124]
  We study a convolutional neural network's internal representation of imbalanced image data. We measure the generalization gap between a model's feature embeddings in the training and test sets, showing that the gap is wider for minority classes. This insight enables us to design an efficient three-phase CNN training framework for imbalanced data.
  arXiv  Detail & Related papers  (2022-07-13T09:43:17Z)
• Predicting Training Time Without Training [120.92623395389255]
  We tackle the problem of predicting the number of optimization steps that a pre-trained deep network needs to converge to a given value of the loss function. We leverage the fact that the training dynamics of a deep network during fine-tuning are well approximated by those of a linearized model. We are able to predict the time it takes to fine-tune a model to a given loss without having to perform any training.
  arXiv  Detail & Related papers  (2020-08-28T04:29:54Z)
• Self-Adaptive Training: beyond Empirical Risk Minimization [15.59721834388181]
  We propose a new training algorithm that dynamically corrects problematic labels by model predictions without incurring extra computational cost. Self-adaptive training significantly improves generalization over various levels of noises, and mitigates the overfitting issue in both natural and adversarial training. Experiments on CIFAR and ImageNet datasets verify the effectiveness of our approach in two applications.
  arXiv  Detail & Related papers  (2020-02-24T15:47:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.