Data-efficient pre-training by scaling synthetic megadocs

• URL: http://arxiv.org/abs/2603.18534v1
• Date: Thu, 19 Mar 2026 06:30:33 GMT
• Title: Data-efficient pre-training by scaling synthetic megadocs
• Abstract summary: We show how to design synthetic data algorithms that achieve better loss scaling.<n>We first show that pre-training on web data mixed with synthetically generated rephrases improves i.i.d. validation loss.<n>We find even better loss scaling under a new perspective: synthetic generations from the same document can form a single substantially longer megadocument.
• Score: 108.28995799706763
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Synthetic data augmentation has emerged as a promising solution when pre-training is constrained by data rather than compute. We study how to design synthetic data algorithms that achieve better loss scaling: not only lowering loss at finite compute but especially as compute approaches infinity. We first show that pre-training on web data mixed with synthetically generated rephrases improves i.i.d. validation loss on the web data, despite the synthetic data coming from an entirely different distribution. With optimal mixing and epoching, loss and benchmark accuracy improve without overfitting as the number of synthetic generations grows, plateauing near $1.48\times$ data efficiency at 32 rephrases per document. We find even better loss scaling under a new perspective: synthetic generations from the same document can form a single substantially longer megadocument instead of many short documents. We show two ways to construct megadocs: stitching synthetic rephrases from the same web document or stretching a document by inserting rationales. Both methods improve i.i.d. loss, downstream benchmarks, and especially long-context loss relative to simple rephrasing, increasing data efficiency from $1.48\times$ to $1.80\times$ at $32$ generations per document. Importantly, the improvement of megadocs over simple rephrasing widens as more synthetic data is generated. Our results show how to design synthetic data algorithms that benefit more from increasing compute when data-constrained.

Related papers

• Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls [25.294408301653576]
  Training data plays a crucial role in Large Language Models (LLM) scaling, yet high quality data is of limited supply.<n>We compare natural web data, diverse synthetic types (rephrased text, generated textbooks), and mixtures of natural and synthetic data.<n>We find pre-training on rephrased synthetic data textitalone is not faster than pre-training on natural web texts.
  arXiv  Detail & Related papers  (2025-10-02T03:24:42Z)
• Pre-training under infinite compute [87.02472603429936]
  We show that data-constrained approaches of increasing epoch count and parameter count eventually overfit.<n>We identify that ensembling independently trained models achieves a significantly lower loss asymptote than the regularized recipe.<n>Our results show that simple algorithmic improvements can enable significantly more data-efficient pre-training in a compute-rich future.
  arXiv  Detail & Related papers  (2025-09-18T09:36:23Z)
• BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining [12.935276089987662]
  We introduce BeyondWeb, a synthetic data generation framework that produces high-quality synthetic data for pretraining.<n>BeyondWeb significantly extends the capabilities of traditional web-scale datasets.<n>It delivers up to 7.7x faster training than open web data and 2.7x faster than Nemotron- Synth.
  arXiv  Detail & Related papers  (2025-08-14T17:55:47Z)
• Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models [92.85086256871027]
  We propose REWIRE, REcycling the Web with guIded REwrite, to enrich low-quality documents so that they could become useful for training.<n>We demonstrate that about 82% of the mixed in texts come from transforming lower-quality documents that would otherwise be discarded.
  arXiv  Detail & Related papers  (2025-06-05T07:12:12Z)
• Scaling Laws of Synthetic Data for Language Models [125.41600201811417]
  We introduce SynthLLM, a scalable framework that transforms pre-training corpora into diverse, high-quality synthetic datasets.<n>Our approach achieves this by automatically extracting and recombining high-level concepts across multiple documents using a graph algorithm.
  arXiv  Detail & Related papers  (2025-03-25T11:07:12Z)
• Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching [19.8751746334929]
  We present an algorithm that remains effective as the size of the synthetic dataset grows. We experimentally find that the training stage of the trajectories we choose to match greatly affects the effectiveness of the distilled dataset. In doing so, we successfully scale trajectory matching-based methods to larger synthetic datasets.
  arXiv  Detail & Related papers  (2023-10-09T14:57:41Z)
• Synthetic data, real errors: how (not) to publish and use synthetic data [86.65594304109567]
  We show how the generative process affects the downstream ML task. We introduce Deep Generative Ensemble (DGE) to approximate the posterior distribution over the generative process model parameters.
  arXiv  Detail & Related papers  (2023-05-16T07:30:29Z)
• Communication-Efficient Adam-Type Algorithms for Distributed Data Mining [93.50424502011626]
  We propose a class of novel distributed Adam-type algorithms (emphi.e., SketchedAMSGrad) utilizing sketching. Our new algorithm achieves a fast convergence rate of $O(frac1sqrtnT + frac1(k/d)2 T)$ with the communication cost of $O(k log(d))$ at each iteration.
  arXiv  Detail & Related papers  (2022-10-14T01:42:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.