Scalable Frequency- and Length-Aware Subdocument Deduplication for Large Language Model Pretraining

• URL: http://arxiv.org/abs/2608.03089v1
• Date: Tue, 04 Aug 2026 04:02:28 GMT
• Title: Scalable Frequency- and Length-Aware Subdocument Deduplication for Large Language Model Pretraining
• Abstract summary: We propose a scalable subdocument deduplication framework that decouples duplicate detection from copy retention.<n>Experiments on FineWeb-Edu and a code-containing web corpus show that models trained on data processed by our method achieve the best overall performance.
• Score: 17.78093240780209
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large-scale pretraining corpora contain substantial duplicate content. Although document-level deduplication is widely used, removing subdocument-level redundancy remains challenging. At corpus scale, suffix-array-based methods are commonly applied independently within shards, leaving cross-shard duplicates undetected and making the resulting retention behavior sensitive to the sharding configuration. Hash-based methods enable global exact duplicate counting, but often rely on fixed copy-retention policies that cannot accommodate heterogeneous repetition patterns. We propose a scalable subdocument deduplication framework that decouples duplicate detection from copy retention. It identifies duplicate groups through natural-boundary segmentation, normalized exact hashing, and distributed aggregation, and then applies an explicit frequency- and length-aware retention policy that allocates an adaptive copy budget to each group, retaining more copies of low-frequency or short repetitions while more aggressively deleting high-frequency or long ones. Experiments on FineWeb-Edu and a code-containing web corpus show that models trained on data processed by our method achieve the best overall performance among the evaluated settings. These results underscore the importance of explicit copy-retention control.

Related papers

• Action-Aligned Retrieval with Pairwise Multimodal Reranking for Text-Based Person Anomaly Search [10.617164505348283]
  Text-based person anomaly search requires distinguishing individuals based on fine-grained, context-dependent behaviors rather than mere appearance.<n>Existing methods struggle to capture these context-conditioned actions.<n>We propose a unified three-stage coarse-to-fine framework that combines action-aligned retrieval with pairwise multimodal reranking.
  arXiv  Detail & Related papers  (2026-08-24T17:04:34Z)
• Generative Universal Multimodal Retrieval with Dual-role Identifiers [1.0981910170846492]
  We propose DrIG, a novel Generative framework for universal multimodal retrieval featuring Dual-role Identifiers.<n>DrIG consistently outperforms state-of-the-art generative multimodal baselines across diverse tasks, while hybrid reranking achieves a favorable efficiency-effectiveness trade-off against strong dense retrievers.
  arXiv  Detail & Related papers  (2026-08-13T09:10:09Z)
• Internal Data Repetition Destroys Language Models [28.65775073109903]
  Language models are running out of high-quality training data, and even aggressively deduplicated corpora retain some amount of repetition.<n>We revisit repetition in the Chinchilla era, using a fitted no-repetition scaling law to report Compute-Equivalent Gain and Compute-Equivalent Loss.
  arXiv  Detail & Related papers  (2026-06-23T16:02:40Z)
• Beyond the Reranker: Do RAG Retrieval Enhancements Help Once a Strong Reranker Is Present? [3.77661940418406]
  Methods for query expansion, hierarchical and cross-document summarization, graph-based expansion, per-query routing, rank fusion, and corrective re-retrieval are studied.<n>We show that a strong cross-encoder reranker accounts for most of the pipeline's quality.<n>The remaining reranking and pool-expansion methods in common use, among them hierarchical summarization, graph expansion, routing, and rank fusion, give no reliable gain once that reranker is present.
  arXiv  Detail & Related papers  (2026-06-14T20:50:41Z)
• AdaPLD: Adaptive Retrieval and Reuse for Efficient Model-Free Speculative Decoding [35.23072855575658]
  We propose emphAdaPLD, a training-free method that adaptively improves both retrieval and draft construction.<n>Across diverse benchmarks, AdaPLD reduces target-model forward passes and achieves up to $3.10times$ decoding speedup.
  arXiv  Detail & Related papers  (2026-06-04T06:09:44Z)
• Mitigating Memorization in Text-to-Image Diffusion via Region-Aware Prompt Augmentation and Multimodal Copy Detection [53.789057575175065]
  Region-Aware Prompt Augmentation (RAPTA) and Attention-Driven Multimodal Copy Detection (ADMCD) are presented.<n>RAPTA uses an object detector to find salient regions and turn them into semantically grounded prompt variants.<n>ADMCD aggregates local patch, global semantic, and texture cues with a lightweight transformer to produce a fused representation.<n>Experiments show that RAPTA reduces overfitting while maintaining high synthesis quality, and that ADMCD reliably detects copying, outperforming single-modal metrics.
  arXiv  Detail & Related papers  (2026-03-13T15:16:27Z)
• DiffuGR: Generative Document Retrieval with Diffusion Language Models [80.78126312115087]
  We propose generative document retrieval with diffusion language models, dubbed DiffuGR.<n>For inference, DiffuGR attempts to generate DocID tokens in parallel and refine them through a controllable number of denoising steps.<n>In contrast to conventional left-to-right auto-regressive decoding, DiffuGR provides a novel mechanism to first generate more confident DocID tokens.
  arXiv  Detail & Related papers  (2025-11-11T12:00:09Z)
• Learning Refined Document Representations for Dense Retrieval via Deliberate Thinking [58.69615583599489]
  Deliberate Thinking based Retriever (Debater) is a novel approach that enhances document representations by incorporating a step-by-step thinking process.<n>Debater significantly outperforms existing methods across several retrieval benchmarks.
  arXiv  Detail & Related papers  (2025-02-18T15:56:34Z)
• The Power of Summary-Source Alignments [62.76959473193149]
  Multi-document summarization (MDS) is a challenging task, often decomposed to subtasks of salience and redundancy detection. alignment of corresponding sentences between a reference summary and its source documents has been leveraged to generate training data. This paper proposes extending the summary-source alignment framework by applying it at the more fine-grained proposition span level.
  arXiv  Detail & Related papers  (2024-06-02T19:35:19Z)
• The Mosaic Memory of Large Language Models [7.405082919188384]
  We show that Large Language Models (LLMs) memorize by assembling information from similar sequences.<n>We show major LLMs to exhibit mosaic memory, with fuzzy duplicates contributing to memorization as much as 0.8 of an exact duplicate and even heavily modified sequences contributing substantially to memorization.
  arXiv  Detail & Related papers  (2024-05-24T13:05:05Z)
• Mitigating the Learning Bias towards Repetition by Self-Contrastive Training for Open-Ended Generation [92.42032403795879]
  We show that pretrained language models (LMs) such as GPT2 still tend to generate repetitive texts. We attribute their overestimation of token-level repetition probabilities to the learning bias. We find that LMs use longer-range dependencies to predict repetitive tokens than non-repetitive ones, which may be the cause of sentence-level repetition loops.
  arXiv  Detail & Related papers  (2023-07-04T07:53:55Z)
• Compressive Summarization with Plausibility and Salience Modeling [54.37665950633147]
  We propose to relax the rigid syntactic constraints on candidate spans and instead leave compression decisions to two data-driven criteria: plausibility and salience. Our method achieves strong in-domain results on benchmark summarization datasets, and human evaluation shows that the plausibility model generally selects for grammatical and factual deletions.
  arXiv  Detail & Related papers  (2020-10-15T17:07:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.