The Oligarch Barely Steers Model Collapse in Multi-Model Ecosystems

• URL: http://arxiv.org/abs/2609.11146v1
• Date: Thu, 10 Sep 2026 06:48:46 GMT
• Title: The Oligarch Barely Steers Model Collapse in Multi-Model Ecosystems
• Abstract summary: Concentration raises two worries: fewer, more uniform sources may make collapse faster, and later models may be dragged toward the oligarch's output.<n>We test both in controlled ecosystems: 13 open 1--4B models form natural ecosystems of 3 to 13 players, plus an injected probe that pushes the top share to 90%.<n>Within the tested range, concentration sets neither the destination nor the pace of collapse; the pace follows whose text fills the pool.
• Score: 9.563027159959587
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: AI-generated text is flowing back into the training corpora of the next generation of models. Recursive training on it drives model collapse, and recent work extends the setting to many models feeding one another -- but almost always with the market split evenly, while real generative AI is an oligopoly. Concentration raises two worries: fewer, more uniform sources may make collapse faster, and later models may be dragged toward the oligarch's output. We test both in controlled ecosystems: 13 open 1--4B models form natural ecosystems of 3 to 13 players, plus an injected probe that pushes the top share to 90%; each generation, every model's output is mixed into a shared pool by market share and every model is retrained on that pool from clean base weights, for five generations. Yet within the range we test, neither worry materializes; what emerges instead is an invariance. Making the split more unequal barely changes the speed of collapse. Destinations move even less: the share and identity knobs shift five-generation endpoints by only a few percent of the drift common to all arms -- the ecosystems collapse to nearly the same place. An extreme share paired with the strongest injected bias still does not guarantee steering, and the topic shifts it does produce leave only a faint trace on the ruler that measures collapse. What sets the speed is who supplies the pool and how readily those suppliers are carried along: with every share held fixed, swapping the members of a K=3 ecosystem changes five-generation drift by 2.8x; a share-weighted index of each member's susceptibility explains the speed differences across nineteen arms with R^2 = 0.68; and replacing half the pool with human text roughly halves drift without changing its course. Within the tested range, concentration sets neither the destination nor the pace of collapse; the pace follows whose text fills the pool.

Related papers

• A Fragility Spectrum for Recursive Language-Model Training [9.563027159959587]
  We show that different models behave very differently under the same process.<n>The unique 4-gram outcome after five generations ranges from 0.187 to 0.940 across checkpoints.<n>Changing the composition of the shared pool or mixing in human text keeps the Spearman correlation of the ordering at 0.91--0.97.
  arXiv  Detail & Related papers  (2026-09-10T06:56:22Z)
• Abliteration Is Not a Scalpel: Off-Target Effects of Refusal Removal on Decision Disposition Across Model Families [51.56484100374058]
  Abliteration - deleting a model's refusal direction from its weights - is the standard recipe behind popular "uncensored" open-weight models.<n>As a disposition probe we use 21,600 decisions under uncertainty, replayed through a frozen pipeline so the decision-layer model is the only variable.<n>Three effects replicate across both families (weeks-clustered CIs excluding zero)<n>A fourth effect reverses sign: the same operation makes Gemma-abliterated less confident and Qwen-abliterated more (family CIs non-overlapping)
  arXiv  Detail & Related papers  (2026-07-19T22:27:00Z)
• Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models [50.733497395381164]
  Agora combines bandwidth-efficient pipeline-parallel model sharding over internet-grade links with multi-party, fault-tolerant collective operations.<n>Report culminates in the first demonstration of its kind: Pluralis-8B, an open, permissionless pretraining run of an 8.6B- parameter model on 500B tokens of FineWeb-Edu.
  arXiv  Detail & Related papers  (2026-07-14T23:32:18Z)
• CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression [77.41779716950387]
  We present Cavewoman, a two-channel evaluation protocol that scores every generation on task accuracy, realized per-item cost, and reference-text agreement.<n>We evaluate eight models on five datasets at five reduction levels, with both channels measured on the same items.
  arXiv  Detail & Related papers  (2026-06-23T02:56:48Z)
• Linear Ensembles Wash Away Watermarks: On the Fragility of Distributional Perturbations in LLMs [13.891118064010898]
  Watermarking embeds statistical signatures in AI-generated text for detection and attribution.<n>We reveal a fundamental vulnerability: when users access multiple models, watermarks trivially fail.<n>We introduce WASH (Watermark Attenuation via Statistical Hybridisation), which solves practical challenges in ensemble generation.
  arXiv  Detail & Related papers  (2026-05-28T19:34:58Z)
• Coupling Models for One-Step Discrete Generation [50.37847093567075]
  Coupling Models is a one-step discrete generative model that learns a direct coupling between discrete sequences and Gaussian latents.<n>It avoids complex continuous flows over the simplex and hand-specified data-to-noise couplings.<n>It reduces LM1B text-generation perplexity by 33% at its lowest-perplexity operating point.
  arXiv  Detail & Related papers  (2026-05-08T03:40:39Z)
• e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddings [91.3041144166326]
  We propose e5-omni, a lightweight explicit alignment recipe that adapts off-the-shelf vision-language models into robust omni-modal embedding models.<n>e5-omni combines modality-aware temperature calibration to align similarity scales, a controllable negative curriculum with debiasing to focus on confusing negatives, and batch whitening to better match cross-modal geometry.<n> Experiments on MMEB-V2 and AudioCaps show consistent gains over strong bi-modal and omni-modal baselines.
  arXiv  Detail & Related papers  (2026-01-07T07:39:40Z)
• Epistemic diversity across language models mitigates knowledge collapse [0.4941630596191806]
  Inspired by ecology, we ask whether AI ecosystem diversity, that is, diversity among models, can mitigate such a collapse.<n>To study the effect of diversity on model performance, we segment the training data across language models and evaluate the resulting ecosystems over ten, self-training iterations.<n>Our results suggest that an ecosystem containing only a few diverse models fails to express the rich mixture of the full, true distribution, resulting in rapid performance decay.
  arXiv  Detail & Related papers  (2025-12-17T02:03:28Z)
• Consensus Is All You Need: Gossip-Based Reasoning Among Large Language Models [0.0]
  Instead of relying on one model alone, we take inspiration from gossip protocols in distributed systems.<n>We show that this "gossip-based consensus" leads to robust, resilient, and accurate multi-agent AI reasoning.
  arXiv  Detail & Related papers  (2025-08-22T05:49:27Z)
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses [97.50616524350123]
  We build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch. The second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level. The third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal.
  arXiv  Detail & Related papers  (2020-01-15T18:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.