Scaling Domain Data Repetition in LLM Pretraining

• URL: http://arxiv.org/abs/2608.14071v1
• Date: Fri, 14 Aug 2026 08:27:15 GMT
• Title: Scaling Domain Data Repetition in LLM Pretraining
• Abstract summary: High-quality domain data is much harder to scale than general web data.<n>As model size and the training-token budget increase, its fraction in the training mixture tends to decrease.<n>We find that repetition counts tuned on smaller proxy models with the same (mathrmTPP) can provide a practical estimate for larger models.
• Score: 18.481235801370826
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As large language models scale, their training-token budgets must also increase to maintain an appropriate tokens-per-parameter ratio (\(\mathrm{TPP}\)). However, high-quality domain data is much harder to scale than general web data. As model size and the training-token budget increase, its fraction in the training mixture tends to decrease. Repeating the available high-quality data provides an effective way to counteract this dilution, but excessive repetition may lead to overfitting. We study this trade-off under practical LLM scaling, where the training-token budget grows proportionally with model size. For a fixed domain, we first find that, surprisingly at a fixed \(\mathrm{TPP}\), the optimal repetition count mildly increases with model size. Across different domains, we find that the optimal repetition count is strongly negatively correlated with the final validation loss of a domain: domains with lower loss can generally benefit from more repetitions. In contrast, the amount of unique domain data is only weakly related to the optimal repetition count. These findings suggest that repetition counts tuned on smaller proxy models with the same \(\mathrm{TPP}\) can provide a practical estimate for larger models.

Related papers

• Internal Data Repetition Destroys Language Models [28.65775073109903]
  Language models are running out of high-quality training data, and even aggressively deduplicated corpora retain some amount of repetition.<n>We revisit repetition in the Chinchilla era, using a fitted no-repetition scaling law to report Compute-Equivalent Gain and Compute-Equivalent Loss.
  arXiv  Detail & Related papers  (2026-06-23T16:02:40Z)
• Explaining Data Mixing Scaling Laws [2.1378946778058796]
  We propose a unified framework to explain the underlying mechanics of data mixing.<n>We identify two key factors that govern the domain losses of models trained on different data mixtures.<n>Our model successfully extrapolates across scales, predicting highly effective mixtures for large, unseen scales.
  arXiv  Detail & Related papers  (2026-06-06T13:31:38Z)
• Fast Unlearning at Scale via Margin Self-Correction [52.46927918952516]
  Language-model unlearning updates a trained model to behave as if it had not seen selected training examples.<n>MASC achieves a competitive forget-retain trade-off at a fraction of the computational cost of existing baselines.
  arXiv  Detail & Related papers  (2026-06-01T21:49:54Z)
• Scaling Laws for Mixture Pretraining Under Data Constraints [20.29616657791023]
  A common strategy is to mix scarce but valuable target data with abundant generic data.<n>We study this trade-off across more than 2,000 language-model training runs.<n>We find that repetition is a central driver of target-domain performance.
  arXiv  Detail & Related papers  (2026-05-12T20:22:45Z)
• The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data [55.87500250831868]
  We study a simple strategy, specialized pretraining (SPT), where a small domain dataset is repeated starting from pretraining as a fraction of the total tokens.<n>In our experiments, SPT reduces the pretraining tokens needed to reach a given domain performance by up to 1.75x.<n>Our observations reveal the finetuner's fallacy: while finetuning may appear to be the cheapest path to domain adaptation, introducing specialized domain data during pretraining stretches its utility.
  arXiv  Detail & Related papers  (2026-03-17T06:55:53Z)
• Scaling Laws for Optimal Data Mixtures [36.161727115217964]
  We propose a systematic method to determine the optimal data mixture for any target domain using scaling laws.<n>We validate the universality of these scaling laws by demonstrating their predictive power in three distinct and large-scale settings.
  arXiv  Detail & Related papers  (2025-07-12T21:16:08Z)
• The interplay between domain specialization and model size [8.653321928148547]
  We investigate the interplay between domain and model size during continued pretraining under compute-constrained scenarios.<n>Our goal is to identify an optimal training regime for this scenario and detect patterns in this interplay that can be generalized across different model sizes and domains.
  arXiv  Detail & Related papers  (2025-01-03T19:28:53Z)
• High-Dimensional Distributed Sparse Classification with Scalable Communication-Efficient Global Updates [50.406127962933915]
  We develop solutions to problems which enable us to learn a communication-efficient distributed logistic regression model. In our experiments we demonstrate a large improvement in accuracy over distributed algorithms with only a few distributed update steps needed.
  arXiv  Detail & Related papers  (2024-07-08T19:34:39Z)
• Scaling Data-Constrained Language Models [133.2083255645999]
  We investigate scaling language models in data-constrained regimes.<n>We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data.<n>We propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters.
  arXiv  Detail & Related papers  (2023-05-25T17:18:55Z)
• To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis [50.31589712761807]
  Large language models (LLMs) are notoriously token-hungry during pre-training, and high-quality text data on the web is approaching its scaling limit for LLMs. We investigate the consequences of repeating pre-training data, revealing that the model is susceptible to overfitting. Second, we examine the key factors contributing to multi-epoch degradation, finding that significant factors include dataset size, model parameters, and training objectives.
  arXiv  Detail & Related papers  (2023-05-22T17:02:15Z)
• DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining [148.90031913522648]
  We propose Domain Reweighting with Minimax Optimization (DoReMi) DoReMi first trains a small proxy model using group distributionally robust optimization (Group DRO) over domains to produce domain weights. We then resample a dataset with these domain weights and train a larger, full-sized model.
  arXiv  Detail & Related papers  (2023-05-17T17:58:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.