A Fragility Spectrum for Recursive Language-Model Training

• URL: http://arxiv.org/abs/2609.11149v1
• Date: Thu, 10 Sep 2026 06:56:22 GMT
• Title: A Fragility Spectrum for Recursive Language-Model Training
• Abstract summary: We show that different models behave very differently under the same process.<n>The unique 4-gram outcome after five generations ranges from 0.187 to 0.940 across checkpoints.<n>Changing the composition of the shared pool or mixing in human text keeps the Spearman correlation of the ordering at 0.91--0.97.
• Score: 9.563027159959587
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Model-generated text is finding its way back into training corpora, and there is plenty of evidence that training on such data over and over collapses output diversity. Prior work has studied the phenomenon itself: which protocols and which data mixtures cause collapse. But different models behave very differently under the same process. We fix one recursive contamination protocol and let 13 publicly released checkpoints form an ecosystem that shares a common corpus for five generations. The unique 4-gram outcome after five generations ranges from 0.187 to 0.940 across checkpoints, a roughly five-fold spread: some models are barely touched, others degenerate into repetitive fragments. Changing the composition of the shared pool or mixing in human text keeps the Spearman correlation of the ordering at 0.91--0.97, and changing the random seed keeps it at 0.93--0.98. Whether a model collapses easily under recursive training is, then, a property of the checkpoint itself, and one that has gone largely unexamined. Parameter scale alone does not explain it, since a three-size ladder within one family is not monotonic in size, and none of the static indicators we tested predicts it either. What does work is cheap: let a model iterate on its own output for two or three generations, and its fragility in the larger ecosystem can be inferred from that alone. Collapse speed also responds to intervention. Tightening top-p, which cuts the low-probability tail at generation time, nearly stops collapse within three generations and stabilizes six checkpoints spanning the whole spectrum together, while data-side filtering slows collapse without stopping it.

Related papers

• SynthSentry: Detecting Synthetic Data Contamination in Language Model Training Data [2.998247765202193]
  We introduce a corpus-level, model-agnostic contamination signal requiring no access to the generating model, no generation history, and no synthetic labels.<n>We evaluate on corpora contaminated by small open-weight generators and an instruction-tuned open-weight model under a leave-one-generator-out protocol.<n>The score ranks corpora by severity with little loss when whole generator families are held out.
  arXiv  Detail & Related papers  (2026-09-11T02:25:18Z)
• The Oligarch Barely Steers Model Collapse in Multi-Model Ecosystems [9.563027159959587]
  Concentration raises two worries: fewer, more uniform sources may make collapse faster, and later models may be dragged toward the oligarch's output.<n>We test both in controlled ecosystems: 13 open 1--4B models form natural ecosystems of 3 to 13 players, plus an injected probe that pushes the top share to 90%.<n>Within the tested range, concentration sets neither the destination nor the pace of collapse; the pace follows whose text fills the pool.
  arXiv  Detail & Related papers  (2026-09-10T06:48:46Z)
• A Later Test Set Is Not a New Domain: Pretraining Familiarity Survives a Contamination-Free Hold-Out [0.0]
  Time-series foundation models are evaluated almost exclusively on public archives that predate them, so a strong score cannot be separated from having seen the test set during pretraining.<n>We build one: thirteen forecasters -- four classical, three trained per dataset, six pretrained -- on seven groups drawn from five domains.<n>Under this protocol pretrained models win 5 of 7 groups, lose one to a Theta baseline, and on daily exchange rates are indistinguishable from a seasonal naive forecast, along with every other method tested.
  arXiv  Detail & Related papers  (2026-09-09T15:52:55Z)
• A Calibration Audit of Confidence in Feed-Forward 3D Reconstruction [45.496856835431885]
  3D reconstruction models emit per-pixel confidence that downstream systems read as a reliability signal.<n>We score the confidence on four properties, how well it ranks error, whether its level is right on average, whether it holds across the confidence range, and whether its intervals cover the truth.<n>The confidence ranks error well, but the predicted uncertainty is too low when it is read under conditions that are not exactly those of training.
  arXiv  Detail & Related papers  (2026-08-30T10:21:34Z)
• Prompt-Model Interaction Reaches the Fixed Points: A deterministic, task-free structural readout -- and the factorizations of it that failed [0.0]
  We ask a readout with no task in it: the fixed-point structure of the short-window argmax map x_t+1 = argmax_x p(x | x_t-1, x_t), censused from 96 starts.<n>It is deterministic, so nothing can be helped or hurt, and it exists only at short windows -- four of six models lose it entirely by window 16 -- so everything here concerns how a model reads a fragment.<n>First, the interaction reaches this readout at full magnitude: nine tokens of conditioning move the fixed-point fraction across most of its
  arXiv  Detail & Related papers  (2026-08-21T17:25:18Z)
• Explanatory Engagement Under Rare Anomalous Failure: Asymptotic Rarity in Model Behavior (or: The Asymptotic AI) [0.0]
  We build a local, zero-cost harness on three open-weight models (qwen3:8b, llama3.1:8b, mistral:7b) running a repeated tool-call task.<n>We hypothesized a rise in engagement as failures grew rarer, then a collapse near a detectability threshold.<n>A companion guaranteed-failure run shows models differ in whether they recognize an anomaly, distinct from engagement once recognized.
  arXiv  Detail & Related papers  (2026-08-13T10:25:40Z)
• Evaluating Generative Time-Series Models on Data with Point Masses [2.5536942228520814]
  We report what happens when such data is evaluated carefully.<n>The standard rolling-origin protocol can score a model on a window whose atom structure bears no resemblance to the dataset.<n>An autoregressive hurdle beats a conditional flow on five of six datasets by up to a factor of $153$.
  arXiv  Detail & Related papers  (2026-08-10T14:57:36Z)
• Multi-Level Collaboration in Model Merging [56.31088116526825]
  This paper explores the intrinsic connections between model merging and model ensembling.<n>We find that even when previous restrictions are not met, there is still a way for model merging to attain a near-identical and superior performance similar to that of ensembling.
  arXiv  Detail & Related papers  (2025-03-03T07:45:04Z)
• Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data [49.73114504515852]
  We show that replacing the original real data by each generation's synthetic data does indeed tend towards model collapse. We demonstrate that accumulating the successive generations of synthetic data alongside the original real data avoids model collapse.
  arXiv  Detail & Related papers  (2024-04-01T18:31:24Z)
• LARA: A Light and Anti-overfitting Retraining Approach for Unsupervised Time Series Anomaly Detection [49.52429991848581]
  We propose a Light and Anti-overfitting Retraining Approach (LARA) for deep variational auto-encoder based time series anomaly detection methods (VAEs) This work aims to make three novel contributions: 1) the retraining process is formulated as a convex problem and can converge at a fast rate as well as prevent overfitting; 2) designing a ruminate block, which leverages the historical data without the need to store them; and 3) mathematically proving that when fine-tuning the latent vector and reconstructed data, the linear formations can achieve the least adjusting errors between the ground truths and the fine-tuned ones.
  arXiv  Detail & Related papers  (2023-10-09T12:36:16Z)
• SurvivalGAN: Generating Time-to-Event Data for Survival Analysis [121.84429525403694]
  Imbalances in censoring and time horizons cause generative models to experience three new failure modes specific to survival analysis. We propose SurvivalGAN, a generative model that handles survival data by addressing the imbalance in the censoring and event horizons. We evaluate this method via extensive experiments on medical datasets.
  arXiv  Detail & Related papers  (2023-02-24T17:03:51Z)
• Mitigating Generation Shifts for Generalized Zero-Shot Learning [52.98182124310114]
  Generalized Zero-Shot Learning (GZSL) is the task of leveraging semantic information (e.g., attributes) to recognize the seen and unseen samples, where unseen classes are not observable during training. We propose a novel Generation Shifts Mitigating Flow framework for learning unseen data synthesis efficiently and effectively. Experimental results demonstrate that GSMFlow achieves state-of-the-art recognition performance in both conventional and generalized zero-shot settings.
  arXiv  Detail & Related papers  (2021-07-07T11:43:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.