Internal Data Repetition Destroys Language Models

• URL: http://arxiv.org/abs/2606.24998v1
• Date: Tue, 23 Jun 2026 16:02:40 GMT
• Title: Internal Data Repetition Destroys Language Models
• Abstract summary: Language models are running out of high-quality training data, and even aggressively deduplicated corpora retain some amount of repetition.<n>We revisit repetition in the Chinchilla era, using a fitted no-repetition scaling law to report Compute-Equivalent Gain and Compute-Equivalent Loss.
• Score: 28.65775073109903
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language models are running out of high-quality training data, and even aggressively deduplicated corpora retain some amount of repetition. Earlier controlled studies predated Chinchilla-style scaling laws and could only measure the cost of repetition indirectly. We revisit repetition in the Chinchilla era, using a fitted no-repetition scaling law to report Compute-Equivalent Gain and Compute-Equivalent Loss. We show that under this modernized paradigm, repetition damage is systematic in three ways. First, holding compute allocated to repeated data constant, eval loss peaks at an intermediate repeat count $\Rep$; repeating a moderately sized subset a moderate number of times damages performance more than repeating a large subset a few times or a small subset many times. Second, the location of this peak is well-fit by a power law in model size; this scaling law reveals that the most damaging number of repeated data grows more quickly than compute. Finally, when repeated documents consume 10\% of the FLOPs budget in a controlled exact-document repetition setting, the compute-equivalent loss can be large: on FineWeb-Edu-Dedup, the most damaging repeat count for a Qwen3-style 344M-parameter model at $\OT=1$ matches the loss of a no-repetition run using 67% of the FLOPs. We demonstrate that these phenomena are not language-model-specific, and can be analytically understood in a simple statistical model: a misspecified linear regression with verbatim duplicates reproduces the same qualitative loss peak, quantifying how such peaks can arise from a statistical tradeoff between memorization and generalization. Our findings add precision to the study of duplication in language models, allowing practitioners to quantify the wasted compute incurred by the presence and repeat structure of duplicates in pretraining corpora.

Related papers

• Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data [119.14301046837822]
  We show that Mixture-of-Experts (MoEs) degrade more rapidly under data repetition.<n>MoEs can outperform dense models even when data is repeated more than 64 times.<n>We analyze internal mechanisms correlated with MoE overfitting in high repetition regimes.
  arXiv  Detail & Related papers  (2026-09-10T17:57:33Z)
• Scaling Domain Data Repetition in LLM Pretraining [18.481235801370826]
  High-quality domain data is much harder to scale than general web data.<n>As model size and the training-token budget increase, its fraction in the training mixture tends to decrease.<n>We find that repetition counts tuned on smaller proxy models with the same (mathrmTPP) can provide a practical estimate for larger models.
  arXiv  Detail & Related papers  (2026-08-14T08:27:15Z)
• Scale Dependent Data Duplication [29.59812821602787]
  We show that semantic duplicates operate increasingly like exact duplicates during training.<n>We embedded all 192 million FineWeb-Edu-Dedup documents using EmbeddingGemma-300m.<n>We derive explicit scaling laws that allow practitioners to estimate deviation from expected scaling due to limited semantic uniqueness of the pretraining corpus.
  arXiv  Detail & Related papers  (2026-02-18T05:22:58Z)
• Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence [113.48450265178953]
  We study how to convert existing pretrained non-recurrent language models into depth-recurrent models.<n>We find that using a curriculum of recurrences to increase the effective depth of the model over the course of training preserves performance while reducing total computational cost.
  arXiv  Detail & Related papers  (2025-11-10T18:43:07Z)
• Rethinking Pruning Large Language Models: Benefits and Pitfalls of Reconstruction Error Minimization [18.24882084542254]
  We present an array of reconstruction techniques that can significantly reduce this error by more than $90%$. We find out that a strategy of self-generating calibration data can mitigate this trade-off between reconstruction and generalization.
  arXiv  Detail & Related papers  (2024-06-21T05:13:34Z)
• Deep Ensembles Meets Quantile Regression: Uncertainty-aware Imputation for Time Series [45.76310830281876]
  We propose Quantile Sub-Ensembles, a novel method to estimate uncertainty with ensemble of quantile-regression-based task networks. Our method not only produces accurate imputations that is robust to high missing rates, but also is computationally efficient due to the fast training of its non-generative model.
  arXiv  Detail & Related papers  (2023-12-03T05:52:30Z)
• Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective [91.14291142262262]
  This work presents a straightforward and fundamental explanation from the data perspective. Our preliminary investigation reveals a strong correlation between the degeneration issue and the presence of repetitions in training data. Our experiments reveal that penalizing the repetitions in training data remains critical even when considering larger model sizes and instruction tuning.
  arXiv  Detail & Related papers  (2023-10-16T09:35:42Z)
• Mitigating the Learning Bias towards Repetition by Self-Contrastive Training for Open-Ended Generation [92.42032403795879]
  We show that pretrained language models (LMs) such as GPT2 still tend to generate repetitive texts. We attribute their overestimation of token-level repetition probabilities to the learning bias. We find that LMs use longer-range dependencies to predict repetitive tokens than non-repetitive ones, which may be the cause of sentence-level repetition loops.
  arXiv  Detail & Related papers  (2023-07-04T07:53:55Z)
• Scaling Data-Constrained Language Models [133.2083255645999]
  We investigate scaling language models in data-constrained regimes.<n>We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data.<n>We propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters.
  arXiv  Detail & Related papers  (2023-05-25T17:18:55Z)
• Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation [41.3948101212288]
  We study the relationship between the probabilities of the repetitive tokens and their previous repetitions in the context. We propose a training method where the model learns to penalize probabilities of sentence-level repetitions from pseudo repetitive data.
  arXiv  Detail & Related papers  (2022-06-06T05:51:12Z)
• Scaling Laws and Interpretability of Learning from Repeated Data [4.3242395495523525]
  We train a family of models where most of the data is unique but a small fraction of it is repeated many times. We find a strong double descent phenomenon, in which repeated data can lead test loss to increase midway through training. A predictable range of repetition frequency leads to surprisingly severe degradation in performance.
  arXiv  Detail & Related papers  (2022-05-21T02:14:27Z)
• Avoiding Inference Heuristics in Few-shot Prompt-based Finetuning [57.4036085386653]
  We show that prompt-based models for sentence pair classification tasks still suffer from a common pitfall of adopting inferences based on lexical overlap. We then show that adding a regularization that preserves pretraining weights is effective in mitigating this destructive tendency of few-shot finetuning.
  arXiv  Detail & Related papers  (2021-09-09T10:10:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.