The Randomness Floor: Measuring Intrinsic Non-Randomness in Language Model Token Distributions

• URL: http://arxiv.org/abs/2604.22771v1
• Date: Sun, 29 Mar 2026 21:17:46 GMT
• Title: The Randomness Floor: Measuring Intrinsic Non-Randomness in Language Model Token Distributions
• Abstract summary: This paper introduces Entropic Deviation (ED), the normalised KL divergence between a model's token distribution and the uniform distribution.<n>Three transformer families converge on nearly identical ED values despite different training data and vocabularies.<n>Cross-lingual experiments with Qwen-32B show a stable gradient across five languages that does not correlate with token fertility.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language models cannot be random. This paper introduces Entropic Deviation (ED), the normalised KL divergence between a model's token distribution and the uniform distribution, and measures it systematically across 31,200 generations spanning seven models, two architectures (transformer and state space), nine prompt categories, three temperatures, and five languages. Under semantically neutral prompts (empty strings, random characters, nonsense syllables) transformers still exhibit ED of approximately 0.30, meaning that 88-93% of the non-randomness observed under semantic prompts is intrinsic to the learned weights rather than induced by context. Three transformer families (Gemma, Llama, Qwen) converge on nearly identical ED values despite different training data and vocabularies. A state space model (Mamba2) reveals a qualitatively different regime: twice the ED, three times lower within-sequence variance, and massive sensitivity to temperature (r = -0.78) where transformers are nearly immune (r < 0.05). Cross-lingual experiments with Qwen-32B show a stable gradient across five languages (English, Japanese, Chinese, Polish, Arabic) that does not correlate with token fertility and persists when two languages sharing an identical tokeniser subset are compared. These findings establish a structural lower bound on randomness in pretrained language models, characterise how this bound differs across architectures, and demonstrate that language itself modulates the bound independently of tokenisation.

Related papers

• A Declarative-Procedural Perspective on Expert Routing in Bilingual Mixture-of-Experts Language Models [0.0]
  We investigate whether Mixture-of-Experts (MoE) language models develop linguistically structured expert routing during bilingual language acquisition.<n>Inspired by the Declarative-Procedural framework, we analyze lexical, grammatical, and syntactic processing in a decoder-only English-German MoE Transformer trained under sequential language exposure.
  arXiv  Detail & Related papers  (2026-08-15T07:53:12Z)
• Does Slightly Mean Somewhat? Measuring Vague Intensity Words in LLM Numeric Actions [0.0]
  I study a researcher-constructed scale of 10 English degree modifier, from slightly to drastically.<n>The only variable that changes between runs is the intensity word or the starting system state.<n>Across 6,620 runs at T=0.7, three patterns emerge: four lower-tier words all map to the same value, while stronger words break into higher regimes.
  arXiv  Detail & Related papers  (2026-05-20T23:49:44Z)
• Phase transition on a context-sensitive random language model with short range interactions [1.4874449172133888]
  We numerically investigate the statistical properties of a random language model with short-range interactions.<n>We find that finite-temperature phase transitions in language models are genuinely induced by the intrinsic nature of language.
  arXiv  Detail & Related papers  (2026-04-01T14:19:11Z)
• Internal Knowledge Without External Expression: Probing the Generalization Boundary of a Classical Chinese Language Model [2.096023520617773]
  We train a language model from scratch on a curated corpus of 1.56 billion tokens of pure Classical Chinese.<n>We investigate whether the model can distinguish known from unknown inputs.<n>We find a clear dissociation between internal and external uncertainty.
  arXiv  Detail & Related papers  (2026-03-31T07:37:15Z)
• Alignment Backfire: Language-Dependent Reversal of Safety Interventions Across 16 Languages in LLM Multi-Agent Systems [0.0]
  In perpetrator treatment, offenders articulate remorse yet behavioral change does not follow.<n>We show that alignment interventions produce a structurally analogous phenomenon: surface safety that masks or generates collective pathology and internal dissociation.<n>These findings reframe alignment as a behavioral intervention subject to risk homeostasis and iatrogenesis.
  arXiv  Detail & Related papers  (2026-03-05T07:46:59Z)
• Cross-Lingual Stability of LLM Judges Under Controlled Generation: Evidence from Finno-Ugric Languages [0.22009842278462158]
  Cross-lingual evaluation of large language models (LLMs) typically conflates two sources of variance: genuine model performance differences and measurement instability.<n>We investigate evaluation reliability by holding generation conditions constant while varying target language.<n>Our findings suggest that zero-shot judge transfer is unreliable for discourse-level assessment in morphologically rich languages.
  arXiv  Detail & Related papers  (2026-02-02T16:27:32Z)
• Information Locality as an Inductive Bias for Neural Language Models [52.92279412466086]
  We show that $m$local entropy are difficult for Transformer and LSTM LMs to learn languages.<n>These results suggest that neurals are highly sensitive to the statistical structure of a language.
  arXiv  Detail & Related papers  (2025-06-05T15:21:05Z)
• First numerical observation of the Berezinskii-Kosterlitz-Thouless transition in language models [1.4061979259370274]
  We numerically demonstrate an unambiguous phase transition in the framework of a natural language model.<n>We identify the phase transition as a variant of the Berezinskii-Kosterlitz-Thouless transition.
  arXiv  Detail & Related papers  (2024-12-02T07:32:32Z)
• PanGu-{\Sigma}: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing [64.53242758625922]
  PanGu-Sigma is trained on a cluster of Ascend 910 AI processors and MindSpore framework. It provides state-of-the-art performance in zero-shot learning of various Chinese NLP downstream tasks.
  arXiv  Detail & Related papers  (2023-03-20T03:39:27Z)
• Scaling Laws for Generative Mixed-Modal Language Models [103.25737824352949]
  We report new mixed-modal scaling laws that unify the contributions of individual modalities and the interactions between them. Specifically, we explicitly model the optimal synergy and competition due to data and model size as an additive term to previous uni-modal scaling laws. We also find four empirical phenomena observed during the training, such as emergent coordinate-ascent style training that naturally alternates between modalities.
  arXiv  Detail & Related papers  (2023-01-10T00:20:06Z)
• Evaluating Distributional Distortion in Neural Language Modeling [81.83408583979745]
  A heavy-tail of rare events accounts for a significant amount of the total probability mass of distributions in language. Standard language modeling metrics such as perplexity quantify the performance of language models (LM) in aggregate. We develop a controlled evaluation scheme which uses generative models trained on natural data as artificial languages.
  arXiv  Detail & Related papers  (2022-03-24T01:09:46Z)
• Bridging the Data Gap between Training and Inference for Unsupervised Neural Machine Translation [49.916963624249355]
  A UNMT model is trained on the pseudo parallel data with translated source, and natural source sentences in inference. The source discrepancy between training and inference hinders the translation performance of UNMT models. We propose an online self-training approach, which simultaneously uses the pseudo parallel data natural source, translated target to mimic the inference scenario.
  arXiv  Detail & Related papers  (2022-03-16T04:50:27Z)
• Parameter Space Factorization for Zero-Shot Learning across Tasks and Languages [112.65994041398481]
  We propose a Bayesian generative model for the space of neural parameters. We infer the posteriors over such latent variables based on data from seen task-language combinations. Our model yields comparable or better results than state-of-the-art, zero-shot cross-lingual transfer methods.
  arXiv  Detail & Related papers  (2020-01-30T16:58:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.