Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data

• URL: http://arxiv.org/abs/2605.10129v1
• Date: Mon, 11 May 2026 07:40:39 GMT
• Title: Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data
• Abstract summary: Large language models rely on web-scale corpora for pre-training.<n>Data curation mitigates but cannot eliminate such noise, so pre-training corpora remain noisy in practice.<n>We study whether a lightweight pre-pre-training stage helps resist noisy data during the pre-training stage.
• Score: 32.68918472543787
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) rely on web-scale corpora for pre-training. The noise inherent in these datasets tends to obscure meaningful patterns and ultimately degrade model performance. Data curation mitigates but cannot eliminate such noise, so pre-training corpora remain noisy in practice. We therefore study whether a lightweight pre-pre-training (PPT) stage based on synthetic data with learnable temporal structure helps resist noisy data during the pre-training (PT) stage. Across various corruption settings, our method consistently improves robustness to noise during PT, with larger relative gains at higher noise levels. For a 1B-parameter model, a synthetic PPT stage with only 65M tokens achieves the same final loss as the baseline while using up to 49\% fewer natural-text PT tokens across different noise levels. Mechanistic analyses suggest PPT does not immediately suppress attention to noisy tokens. Rather, PPT-initialized models gradually downweight attention between corrupted tokens during noisy PT. This indicates that synthetic PPT inhibits noise self-modeling and shapes the subsequent optimization trajectory. Code is available at https://github.com/guox18/formal-language-prepretraining.

Related papers

• Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners [33.10721304996431]
  Self-supervised learning has driven substantial progress in audio representation learning.<n>We introduce NAPE, a self-supervised framework in which a causal Transformer predicts each next patch embedding of a spectrogram from the previous ones.<n>It achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results.
  arXiv  Detail & Related papers  (2026-08-20T10:16:07Z)
• DropoutTS: Sample-Adaptive Dropout for Robust Time Series Forecasting [59.868414584142336]
  DropoutTS is a model-agnostic plugin that shifts the paradigm from "what" to "how much" to learn.<n>It maps noise to adaptive dropout rates - selectively suppressing spurious fluctuations while preserving fine-grained fidelity.
  arXiv  Detail & Related papers  (2026-01-29T13:49:20Z)
• PDR: A Plug-and-Play Positional Decay Framework for LLM Pre-training Data Detection [30.13331191100816]
  We introduce Positional Decay Reweighting (PDR), a training-free and plug-and-play framework to detect pre-training data in Large Language Models (LLMs)<n>PDR explicitly reweights token-level scores to amplify distinct signals from early positions while suppressing noise from later ones.
  arXiv  Detail & Related papers  (2026-01-11T09:32:13Z)
• Mixture of Noise for Pre-Trained Model-Based Class-Incremental Learning [59.635264288605946]
  Class Incremental Learning (CIL) aims to continuously learn new categories while retaining the knowledge of old ones.<n>Existing approaches that apply lightweight fine-tuning to backbones still induce drift.<n>We propose Mixture of Noise (Min) to mitigate the degradation of backbone generalization from adapting new tasks.
  arXiv  Detail & Related papers  (2025-09-20T16:07:20Z)
• Impact of Noisy Supervision in Foundation Model Learning [91.56591923244943]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets.<n>We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• Noise-Robust Fine-Tuning of Pretrained Language Models via External Guidance [61.809732058101304]
  We introduce an innovative approach for fine-tuning PLMs using noisy labels. This approach incorporates the guidance of Large Language Models (LLMs) like ChatGPT. This guidance assists in accurately distinguishing between clean and noisy samples.
  arXiv  Detail & Related papers  (2023-11-02T09:20:38Z)
• Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks [91.15120211190519]
  This paper aims to understand the nature of noise in pre-training datasets and to mitigate its impact on downstream tasks. We propose a light-weight black-box tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise.
  arXiv  Detail & Related papers  (2023-09-29T06:18:15Z)
• Latent Class-Conditional Noise Model [54.56899309997246]
  We introduce a Latent Class-Conditional Noise model (LCCN) to parameterize the noise transition under a Bayesian framework. We then deduce a dynamic label regression method for LCCN, whose Gibbs sampler allows us efficiently infer the latent true labels. Our approach safeguards the stable update of the noise transition, which avoids previous arbitrarily tuning from a mini-batch of samples.
  arXiv  Detail & Related papers  (2023-02-19T15:24:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.