Model Collapse as Cultural Evolution

• URL: http://arxiv.org/abs/2605.23054v1
• Date: Thu, 21 May 2026 21:36:26 GMT
• Title: Model Collapse as Cultural Evolution
• Abstract summary: We show that iterated learning theory from cultural evolution fills this gap.<n>We derive five falsifiable predictions, distinguish those uniquely discriminative for the theory from confirmatory ones, and test them by self-training.<n>The critical discriminative finding: compositionality follows a non-monotonic trajectory under unfiltered self-training.
• Score: 13.891522069967507
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Model collapse, the progressive degradation of LLMs trained on their own outputs, has been characterized statistically but lacks a linguistic explanation for which structures degrade, in what order, and why. We show that iterated learning theory from cultural evolution fills this gap. We derive five falsifiable predictions, distinguish those uniquely discriminative for the theory from confirmatory ones, and test them by self-training LLaMA-2-7B and Mistral-7B over 10 generations in English, German, and Turkish. The critical discriminative finding: compositionality follows a non-monotonic trajectory (initially rising, then falling) under unfiltered self-training. This signature persists with maximally regular seed data (ruling out noise removal) and is sustained only by task-grounded filtering, not random filtering, providing the first LLM-scale evidence for the compression-communication tradeoff. All predictions are confirmed with large effect sizes (Hedges' $g > 1.6$; $\mathrm{BF}_{10} > 100$), and LLM regularization gradients closely match human behavioral data ($R^2 = 0.94$). These results reframe model collapse as a cultural transmission phenomenon and yield concrete principles for self-training pipeline design.

Related papers

• Analysis of the Neglect-Zero Effect in Large Language Models [17.164287400700815]
  We investigate the extent to which the language processing of LLMs resembles human cognitive processes.<n>We focus on two types of inferences driven by the neglect-zero effect.<n>The results suggest that the neglect-zero effect may not occur in the LLMs analyzed in this study.
  arXiv  Detail & Related papers  (2026-06-04T08:39:39Z)
• The Oracle's Fingerprint: Correlated AI Forecasting Errors and the Limits of Bias Transmission [0.0]
  We show that GPT-4o, Claude, and Gemini exhibit highly correlated forecasting errors on binary prediction questions.<n>In Study 2, we test whether this correlated bias has propagated into human crowd forecasts.<n>In Study 3, we examine whether the category-level pattern of human forecasting errors increasingly resembles the LLM bias fingerprint.
  arXiv  Detail & Related papers  (2026-04-07T19:45:40Z)
• Self-Calibrating Language Models via Test-Time Discriminative Distillation [18.46710400838861]
  Large language models (LLMs) are systematically overconfident on questions they often answer incorrectly.<n>We introduce $textbfSECL$ ($textbfSE$lf-$textbfC$alibrating $textbfL$anguage Models), a test-time training (TTT) pipeline that exploits this gap as label-free self-supervision.
  arXiv  Detail & Related papers  (2026-03-18T13:28:50Z)
• Deriving Decoder-Free Sparse Autoencoders from First Principles [0.0]
  Theory predicts collapse without volume control analogous to the log-determinant in Gaussian mixture models.<n>We instantiate the theory in a single-layer encoder with an LSE objective and InfoMax regularization for volume control.<n>Experiments confirm the theory's predictions.
  arXiv  Detail & Related papers  (2026-01-10T08:13:58Z)
• LLMs Can Get "Brain Rot"! [68.08198331505695]
  Continual exposure to junk web text induces lasting cognitive decline in large language models (LLMs)<n>We run controlled experiments on real Twitter/X corpora, constructing junk and reversely controlled datasets.<n>Results provide significant, multi-perspective evidence that data quality is a causal driver of LLM capability decay.
  arXiv  Detail & Related papers  (2025-10-15T13:28:49Z)
• Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning [77.120955854093]
  We show that data diversity can be a strong predictor of generalization in language models.<n>We introduce G-Vendi, a metric that quantifies diversity via the entropy of model-induced gradients.<n>We present Prismatic Synthesis, a framework for generating diverse synthetic data.
  arXiv  Detail & Related papers  (2025-05-26T16:05:10Z)
• Context-Free Synthetic Data Mitigates Forgetting [13.825822994127943]
  We show that augmenting a fine-tuning dataset with context-free generations mitigates forgetting.<n>We present our results for OLMo-1B for pretrained-only setting and R1-Distill-Llama-8B for the reasoning setting.
  arXiv  Detail & Related papers  (2025-05-20T01:47:31Z)
• S$^2$R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning [51.84977135926156]
  We introduce S$2$R, an efficient framework that enhances LLM reasoning by teaching models to self-verify and self-correct during inference.<n>Our results demonstrate that Qwen2.5-math-7B achieves an accuracy improvement from 51.0% to 81.6%, outperforming models trained on an equivalent amount of long-CoT distilled data.
  arXiv  Detail & Related papers  (2025-02-18T13:40:22Z)
• Rejection via Learning Density Ratios [50.91522897152437]
  Classification with rejection emerges as a learning paradigm which allows models to abstain from making predictions.<n>We propose a different distributional perspective, where we seek to find an idealized data distribution which maximizes a pretrained model's performance.<n>Our framework is tested empirically over clean and noisy datasets.
  arXiv  Detail & Related papers  (2024-05-29T01:32:17Z)
• Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experiment [82.60594940370919]
  We propose the FlipFlop experiment to study the multi-turn behavior of Large Language Models (LLMs) We show that models flip their answers on average 46% of the time and that all models see a deterioration of accuracy between their first and final prediction, with an average drop of 17% (the FlipFlop effect) We conduct finetuning experiments on an open-source LLM and find that finetuning on synthetically created data can mitigate - reducing performance deterioration by 60% - but not resolve sycophantic behavior entirely.
  arXiv  Detail & Related papers  (2023-11-14T23:40:22Z)
• Do pretrained Transformers Learn In-Context by Gradient Descent? [21.23795112800977]
  In this paper, we investigate the emergence of In-Context Learning (ICL) in language models pre-trained on natural data (LLaMa-7B) We find that ICL and Gradient Descent (GD) modify the output distribution of language models differently. These results indicate that emphthe equivalence between ICL and GD remains an open hypothesis and calls for further studies.
  arXiv  Detail & Related papers  (2023-10-12T17:32:09Z)
• Can Large Language Models Infer Causation from Correlation? [104.96351414570239]
  We test the pure causal inference skills of large language models (LLMs) We formulate a novel task Corr2Cause, which takes a set of correlational statements and determines the causal relationship between the variables. We show that these models achieve almost close to random performance on the task.
  arXiv  Detail & Related papers  (2023-06-09T12:09:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.