Can Generative Artificial Intelligence Survive Data Contamination? Theoretical Guarantees under Contaminated Recursive Training

• URL: http://arxiv.org/abs/2602.16065v1
• Date: Tue, 17 Feb 2026 22:38:18 GMT
• Title: Can Generative Artificial Intelligence Survive Data Contamination? Theoretical Guarantees under Contaminated Recursive Training
• Authors: Kevin Wang, Hongqian Niu, Didong Li,
• Abstract summary: Generative Artificial Intelligence (AI) has become a transformative force across science, industry, and society.<n>As these systems grow in popularity, web data becomes increasingly interwoven with this AI-generated material.<n>As generative models are updated regularly, later models will inevitably be trained on mixtures of human-generated data and AI-generated data from earlier versions.
• Score: 11.253812961752958
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Generative Artificial Intelligence (AI), such as large language models (LLMs), has become a transformative force across science, industry, and society. As these systems grow in popularity, web data becomes increasingly interwoven with this AI-generated material and it is increasingly difficult to separate them from naturally generated content. As generative models are updated regularly, later models will inevitably be trained on mixtures of human-generated data and AI-generated data from earlier versions, creating a recursive training process with data contamination. Existing theoretical work has examined only highly simplified settings, where both the real data and the generative model are discrete or Gaussian, where it has been shown that such recursive training leads to model collapse. However, real data distributions are far more complex, and modern generative models are far more flexible than Gaussian and linear mechanisms. To fill this gap, we study recursive training in a general framework with minimal assumptions on the real data distribution and allow the underlying generative model to be a general universal approximator. In this framework, we show that contaminated recursive training still converges, with a convergence rate equal to the minimum of the baseline model's convergence rate and the fraction of real data used in each iteration. To the best of our knowledge, this is the first (positive) theoretical result on recursive training without distributional assumptions on the data. We further extend the analysis to settings where sampling bias is present in data collection and support all theoretical results with empirical studies.

Related papers

• Universal pre-training by iterated random computation [0.0]
  We show that synthetically generated data can be used to pre-train a model before the data is seen.<n>We extend earlier results to real-world data, and show that finetuning a model after pre-training offers faster convergence and better generalization.
  arXiv  Detail & Related papers  (2025-06-24T23:36:35Z)
• A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops [55.07063067759609]
  High-quality data is essential for training large generative models, yet the vast reservoir of real data available online has become nearly depleted.<n>Models increasingly generate their own data for further training, forming Self-consuming Training Loops (STLs)<n>Some models degrade or even collapse, while others successfully avoid these failures, leaving a significant gap in theoretical understanding.
  arXiv  Detail & Related papers  (2025-02-26T06:18:13Z)
• Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World [19.266191284270793]
  generative machine learning models are pretrained on web-scale datasets containing data generated by earlier models.<n>Some prior work warns of "model collapse" as the web is overwhelmed by synthetic data.<n>We report experiments on three ways of using data (training-workflows) across three generative model task-settings.
  arXiv  Detail & Related papers  (2024-10-22T05:49:24Z)
• Learning by Surprise: Surplexity for Mitigating Model Collapse in Generative AI [1.6545633988217645]
  As synthetic content infiltrates the web, generative AI models may be retrained on their own outputs.<n>This leads to model collapse: a progressive loss of performance and diversity across generations.<n>We introduce new measures that characterise collapse directly from a model's next-token probability distributions.
  arXiv  Detail & Related papers  (2024-10-16T08:02:48Z)
• Heat Death of Generative Models in Closed-Loop Learning [63.83608300361159]
  We study the learning dynamics of generative models that are fed back their own produced content in addition to their original training dataset. We show that, unless a sufficient amount of external data is introduced at each iteration, any non-trivial temperature leads the model to degenerate.
  arXiv  Detail & Related papers  (2024-04-02T21:51:39Z)
• Towards Theoretical Understandings of Self-Consuming Generative Models [56.84592466204185]
  This paper tackles the emerging challenge of training generative models within a self-consuming loop. We construct a theoretical framework to rigorously evaluate how this training procedure impacts the data distributions learned by future models. We present results for kernel density estimation, delivering nuanced insights such as the impact of mixed data training on error propagation.
  arXiv  Detail & Related papers  (2024-02-19T02:08:09Z)
• Learning Defect Prediction from Unrealistic Data [57.53586547895278]
  Pretrained models of code have become popular choices for code understanding and generation tasks. Such models tend to be large and require commensurate volumes of training data. It has become popular to train models with far larger but less realistic datasets, such as functions with artificially injected bugs. Models trained on such data tend to only perform well on similar data, while underperforming on real world programs.
  arXiv  Detail & Related papers  (2023-11-02T01:51:43Z)
• On the Stability of Iterative Retraining of Generative Models on their own Data [56.153542044045224]
  We study the impact of training generative models on mixed datasets. We first prove the stability of iterative training under the condition that the initial generative models approximate the data distribution well enough. We empirically validate our theory on both synthetic and natural images by iteratively training normalizing flows and state-of-the-art diffusion models.
  arXiv  Detail & Related papers  (2023-09-30T16:41:04Z)
• Synthetic data, real errors: how (not) to publish and use synthetic data [86.65594304109567]
  We show how the generative process affects the downstream ML task. We introduce Deep Generative Ensemble (DGE) to approximate the posterior distribution over the generative process model parameters.
  arXiv  Detail & Related papers  (2023-05-16T07:30:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.