The Fairness Collapse Phenomenon: Bias Amplification in Language Models Trained on Synthetic Data

• URL: http://arxiv.org/abs/2608.04268v1
• Date: Tue, 04 Aug 2026 22:56:39 GMT
• Title: The Fairness Collapse Phenomenon: Bias Amplification in Language Models Trained on Synthetic Data
• Abstract summary: Generative models trained on artificially generated data have been shown to exhibit model collapse.<n>As synthetic content increasingly contaminates the training corpora, this raises critical concerns about the use of open data in continued pretraining.
• Score: 4.774098410526357
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative models trained on artificially generated data have been shown to exhibit model collapse, resulting in significant performance degradation. As synthetic content increasingly contaminates the training corpora of language models, this raises critical concerns about the use of open data in continued pretraining. Although previous work has demonstrated model collapse in language models, it remains unclear whether exposure to synthetic data amplifies or attenuates the social biases already present in pretrained models. Because language models are known to reproduce and amplify demographic stereotypes, recursive training on self-generated data may create a self-reinforcing feedback loop in which biased associations become progressively stronger across generations. We call this hypothesized phenomenon fairness collapse. In this work, we construct controlled training regimes in which models are repeatedly trained on synthetic data using the Bias in Bios dataset. Across experiments, we observe a consistent and concerning pattern: fairness degradation emerges before substantial degradation is reflected by standard language-modeling metrics. This result highlights a critical risk associated with synthetic data contamination in language model training: bias can increase silently before strong indicators of model collapse become apparent.

Related papers

• When Models Don't Collapse: On the Consistency of Iterative MLE [26.518651842868135]
  We study model collapse for maximum likelihood estimation (MLE) in a natural setting.<n>We establish non-asymptotic bounds showing that collapse can be avoided even as the fraction of real data vanishes.<n>We prove that some assumptions (beyond MLE consistency) are indeed necessary.
  arXiv  Detail & Related papers  (2025-05-25T08:50:46Z)
• A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops [55.07063067759609]
  High-quality data is essential for training large generative models, yet the vast reservoir of real data available online has become nearly depleted.<n>Models increasingly generate their own data for further training, forming Self-consuming Training Loops (STLs)<n>Some models degrade or even collapse, while others successfully avoid these failures, leaving a significant gap in theoretical understanding.
  arXiv  Detail & Related papers  (2025-02-26T06:18:13Z)
• How to Synthesize Text Data without Model Collapse? [37.219627817995054]
  Model collapse in synthetic data indicates that iterative training on self-generated data leads to a gradual decline in performance.<n>We propose token editing on human-produced data to obtain semi-synthetic data.
  arXiv  Detail & Related papers  (2024-12-19T09:43:39Z)
• Will the Inclusion of Generated Data Amplify Bias Across Generations in Future Image Classification Models? [29.71939692883025]
  We investigate the effects of generated data on image classification tasks, with a specific focus on bias. Hundreds of experiments are conducted on Colorized MNIST, CIFAR-20/100, and Hard ImageNet datasets. Our findings contribute to the ongoing debate on the implications of synthetic data for fairness in real-world applications.
  arXiv  Detail & Related papers  (2024-10-14T05:07:06Z)
• How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse [9.59833542807268]
  Model collapse occurs when new models are trained on synthetic data generated from previously trained models. We show that model collapse cannot be avoided when training solely on synthetic data. We estimate a maximal amount of synthetic data below which model collapse can eventually be avoided.
  arXiv  Detail & Related papers  (2024-04-07T22:15:13Z)
• Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data [49.73114504515852]
  We show that replacing the original real data by each generation's synthetic data does indeed tend towards model collapse. We demonstrate that accumulating the successive generations of synthetic data alongside the original real data avoids model collapse.
  arXiv  Detail & Related papers  (2024-04-01T18:31:24Z)
• Fairness Feedback Loops: Training on Synthetic Data Amplifies Bias [47.79659355705916]
  Model-induced distribution shifts (MIDS) occur as previous model outputs pollute new model training sets over generations of models. We introduce a framework that allows us to track multiple MIDS over many generations, finding that they can lead to loss in performance, fairness, and minoritized group representation. Despite these negative consequences, we identify how models might be used for positive, intentional, interventions in their data ecosystems.
  arXiv  Detail & Related papers  (2024-03-12T17:48:08Z)
• Towards Theoretical Understandings of Self-Consuming Generative Models [56.84592466204185]
  This paper tackles the emerging challenge of training generative models within a self-consuming loop. We construct a theoretical framework to rigorously evaluate how this training procedure impacts the data distributions learned by future models. We present results for kernel density estimation, delivering nuanced insights such as the impact of mixed data training on error propagation.
  arXiv  Detail & Related papers  (2024-02-19T02:08:09Z)
• On the Stability of Iterative Retraining of Generative Models on their own Data [56.153542044045224]
  We study the impact of training generative models on mixed datasets. We first prove the stability of iterative training under the condition that the initial generative models approximate the data distribution well enough. We empirically validate our theory on both synthetic and natural images by iteratively training normalizing flows and state-of-the-art diffusion models.
  arXiv  Detail & Related papers  (2023-09-30T16:41:04Z)
• Debiasing Vision-Language Models via Biased Prompts [79.04467131711775]
  We propose a general approach for debiasing vision-language foundation models by projecting out biased directions in the text embedding. We show that debiasing only the text embedding with a calibrated projection matrix suffices to yield robust classifiers and fair generative models.
  arXiv  Detail & Related papers  (2023-01-31T20:09:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.