Scale Dependent Data Duplication

• URL: http://arxiv.org/abs/2603.06603v1
• Date: Wed, 18 Feb 2026 05:22:58 GMT
• Title: Scale Dependent Data Duplication
• Abstract summary: We show that semantic duplicates operate increasingly like exact duplicates during training.<n>We embedded all 192 million FineWeb-Edu-Dedup documents using EmbeddingGemma-300m.<n>We derive explicit scaling laws that allow practitioners to estimate deviation from expected scaling due to limited semantic uniqueness of the pretraining corpus.
• Score: 29.59812821602787
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Data duplication during pretraining can degrade generalization and lead to memorization, motivating aggressive deduplication pipelines. However, at web scale, it is unclear what constitutes a ``duplicate'': beyond surface-form matches, semantically equivalent documents (e.g. translations) may induce redundant training signals once models become sufficiently capable. Practically, this means that semantic duplicates operate increasingly like exact duplicates during training. We present evidence that duplication is scale-dependent in two ways. First, as model capability increases, cross-entropy loss gradients for semantically equivalent documents become more aligned. Smaller models, by contrast, produce gradients that reflect surface similarity (e.g., shared tokens) rather than semantic similarity. Second, we embedded all 192 million FineWeb-Edu-Dedup documents using EmbeddingGemma-300m. For moderate corpus sizes, the cosine similarity between nearest-neighbors follows an isotropic power law baseline. However, as corpus size grows to hundreds of billions of tokens, the nearest-neighbor similarities deviate sharply, indicating accelerated semantic collisions. Finally, controlled pretraining on data sampled with replacement from pools of finite unique documents shows that limited uniqueness yields mild degradation for small models, but rapidly increasing loss penalties for larger models, breaking naive scaling extrapolation. We derive explicit scaling laws that allow practitioners to estimate deviation from expected scaling due to limited semantic uniqueness of the pretraining corpus. Our results identify and resolve an unstudied source of scale-dependence, allowing for more accurate prediction at scale.

Related papers

• Skaling: Chinchilla's Exponents Meet Kaplan's Coupling [19.21066155468739]
  We introduce the Skaling law, a functional form that couples model capacity and data through a single interaction exponent.<n>This simple extension reduces the Mean Absolute Percentage Error (MAPE) by 1.5-3x across both regimes.<n>By enabling reliable performance prediction from small-scale experiments, the Skaling law provides a more robust and resource-efficient framework.
  arXiv  Detail & Related papers  (2026-08-07T13:38:51Z)
• Internal Data Repetition Destroys Language Models [28.65775073109903]
  Language models are running out of high-quality training data, and even aggressively deduplicated corpora retain some amount of repetition.<n>We revisit repetition in the Chinchilla era, using a fitted no-repetition scaling law to report Compute-Equivalent Gain and Compute-Equivalent Loss.
  arXiv  Detail & Related papers  (2026-06-23T16:02:40Z)
• Better and Worse with Scale: How Contextual Entrainment Diverges with Model Size [44.634649562117744]
  We formalize this apparent paradox through the first scaling laws for contextual entrainment.<n>We find entrainment follows predictable power-law scaling, but with opposite trends depending on context type.<n>Concretely, the largest models are four times more resistant to counterfactual misinformation than the smallest.
  arXiv  Detail & Related papers  (2026-04-14T20:12:05Z)
• On the Entropy Calibration of Language Models [52.47557449370603]
  We study the problem of entropy calibration, which asks whether a language model's entropy over generations matches its log loss on human text.<n>We find that the observed scaling behavior is similar to what is predicted by the simplified setting.<n>We prove that it is possible, if we assume access to a black box which can fit models to predict the future entropy of text.
  arXiv  Detail & Related papers  (2025-11-15T00:33:03Z)
• Unified Scaling Laws for Compressed Representations [69.72517034565467]
  We investigate whether a unified scaling framework can accurately predict model performance when training occurs over various compressed representations.<n>Our main finding is demonstrating both theoretically and empirically that there exists a simple "capacity" metric.<n>We extend our formulation to directly compare the accuracy potential of different compressed formats, and to derive better algorithms for training over sparse-quantized formats.
  arXiv  Detail & Related papers  (2025-06-02T16:52:51Z)
• A Hitchhiker's Guide to Scaling Law Estimation [56.06982415792523]
  Scaling laws predict the loss of a target machine learning model by extrapolating from easier-to-train models with fewer parameters or smaller training sets.<n>We estimate more than 1000 scaling laws, then derive a set of best practices for estimating scaling laws in new model families.
  arXiv  Detail & Related papers  (2024-10-15T17:59:10Z)
• Training Trajectories of Language Models Across Scales [99.38721327771208]
  Scaling up language models has led to unprecedented performance gains. How do language models of different sizes learn during pre-training? Why do larger language models demonstrate more desirable behaviors?
  arXiv  Detail & Related papers  (2022-12-19T19:16:29Z)
• Intra-class Adaptive Augmentation with Neighbor Correction for Deep Metric Learning [99.14132861655223]
  We propose a novel intra-class adaptive augmentation (IAA) framework for deep metric learning. We reasonably estimate intra-class variations for every class and generate adaptive synthetic samples to support hard samples mining. Our method significantly improves and outperforms the state-of-the-art methods on retrieval performances by 3%-6%.
  arXiv  Detail & Related papers  (2022-11-29T14:52:38Z)
• Similarity between Units of Natural Language: The Transition from Coarse to Fine Estimation [4.314956204483074]
  Capturing the similarities between human language units is crucial for explaining how humans associate different objects.<n>My research goal in this thesis is to develop regression models that account for similarities between language units in a more refined way.
  arXiv  Detail & Related papers  (2022-10-25T18:54:32Z)
• X-model: Improving Data Efficiency in Deep Learning with A Minimax Model [78.55482897452417]
  We aim at improving data efficiency for both classification and regression setups in deep learning. To take the power of both worlds, we propose a novel X-model. X-model plays a minimax game between the feature extractor and task-specific heads.
  arXiv  Detail & Related papers  (2021-10-09T13:56:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.