Turbulence-like 5/3 spectral scaling in contextual representations of language as a complex system

• URL: http://arxiv.org/abs/2604.05536v1
• Date: Tue, 07 Apr 2026 07:36:12 GMT
• Title: Turbulence-like 5/3 spectral scaling in contextual representations of language as a complex system
• Abstract summary: We quantify scale-dependent fluctuations along the token sequence using an embedding-step signal.<n>Across multiple languages and corpora, the resulting power spectrum exhibits a robust power law with an exponent close to $5/3$.<n>By analogy with the Kolmogorov spectrum in turbulence, our findings suggest that semantic information is integrated in a scale-free, self-similar manner across linguistic scales.
• Score: 1.4363133217553712
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Natural language is a complex system that exhibits robust statistical regularities. Here, we represent text as a trajectory in a high-dimensional embedding space generated by transformer-based language models, and quantify scale-dependent fluctuations along the token sequence using an embedding-step signal. Across multiple languages and corpora, the resulting power spectrum exhibits a robust power law with an exponent close to $5/3$ over an extended frequency range. This scaling is observed consistently in contextual embeddings from both human-written and AI-generated text, but is absent in static word embeddings and is disrupted by randomization of token order. These results show that the observed scaling reflects multiscale, context-dependent organization rather than lexical statistics alone. By analogy with the Kolmogorov spectrum in turbulence, our findings suggest that semantic information is integrated in a scale-free, self-similar manner across linguistic scales, and provide a quantitative, model-agnostic benchmark for studying complex structure in language representations.

Related papers

• Repeated Sequences Reveal Gaps between Large Language Models and Natural Language [8.883733362171036]
  We show that GPT-generated texts show systematic and statistically significant shifts in estimated exponents with model size.<n>These results demonstrate that repeated-subsequence entropy provides a quantitative structural diagnostic.
  arXiv  Detail & Related papers  (2026-05-24T03:49:20Z)
• The Statistical Signature of LLMs [1.3135750017147134]
  We show that a simple, model-agnostic measure of statistical regularity differentiates generative regimes directly from surface text.<n>Across settings, compression reveals a persistent structural signature of probabilistic generation.<n>Our findings introduce a simple and robust framework for quantifying how generative systems reshape textual production.
  arXiv  Detail & Related papers  (2026-02-20T11:33:37Z)
• Scale-Dependent Semantic Dynamics Revealed by Allan Deviation [0.0]
  We analyze the stability of meaning by treating ordered sentence embeddings as a displacement signal.<n>We find that while large language models successfully mimic the local scaling statistics of human text, they exhibit a systematic reduction in their stability horizon.
  arXiv  Detail & Related papers  (2026-01-29T13:10:59Z)
• From Linear Input to Hierarchical Structure: Function Words as Statistical Cues for Language Learning [2.893006778402251]
  We argue that function words play a crucial role in language acquisition due to their distinctive distributional properties.<n>We show that language variants preserving all three properties are more easily acquired by neural learners.
  arXiv  Detail & Related papers  (2026-01-29T02:42:12Z)
• Correlation Dimension of Auto-Regressive Large Language Models [11.183390901786659]
  Large language models (LLMs) have achieved remarkable progress in natural language generation.<n>They continue to display puzzling behaviors, such as repetition and incoherence, even when exhibiting low perplexity.<n>We introduce correlation dimension, a fractal-geometric measure of self-similarity, to quantify complexity of text.
  arXiv  Detail & Related papers  (2025-10-24T08:42:23Z)
• The Harmonic Structure of Information Contours [54.38365999922221]
  We find consistent evidence of periodic patterns in information rate in texts in English, Spanish, German, Dutch, Basque, and Brazilian Portuguese.<n>Many dominant frequencies align with discourse structure, suggesting these oscillations reflect meaningful linguistic organization.
  arXiv  Detail & Related papers  (2025-06-04T12:56:30Z)
• Towards Understanding Text Hallucination of Diffusion Models via Local Generation Bias [76.85949078144098]
  This paper focuses on textual hallucinations, where diffusion models correctly generate individual symbols but assemble them in a nonsensical manner.<n>We observe that such phenomenon is attributed it to the network's local generation bias.<n>We also theoretically analyze the training dynamics for a specific case involving a two-layer learning parity points on a hypercube.
  arXiv  Detail & Related papers  (2025-03-05T15:28:50Z)
• How Compositional Generalization and Creativity Improve as Diffusion Models are Trained [82.08869888944324]
  How many samples do generative models need in order to learn composition rules?<n>What signal in the data is exploited to learn those rules?<n>We discuss connections between the hierarchical clustering mechanism we introduce here and the renormalization group in physics.
  arXiv  Detail & Related papers  (2025-02-17T18:06:33Z)
• DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion [66.21290235237808]
  We introduce an energy constrained diffusion model which encodes a batch of instances from a dataset into evolutionary states. We provide rigorous theory that implies closed-form optimal estimates for the pairwise diffusion strength among arbitrary instance pairs. Experiments highlight the wide applicability of our model as a general-purpose encoder backbone with superior performance in various tasks.
  arXiv  Detail & Related papers  (2023-01-23T15:18:54Z)
• Model Criticism for Long-Form Text Generation [113.13900836015122]
  We apply a statistical tool, model criticism in latent space, to evaluate the high-level structure of generated text. We perform experiments on three representative aspects of high-level discourse -- coherence, coreference, and topicality. We find that transformer-based language models are able to capture topical structures but have a harder time maintaining structural coherence or modeling coreference.
  arXiv  Detail & Related papers  (2022-10-16T04:35:58Z)
• Learning Multiscale Transformer Models for Sequence Generation [33.73729074207944]
  We build a multiscale Transformer model by establishing relationships among scales based on word-boundary information and phrase-level prior knowledge. Notably, it yielded consistent performance gains over the strong baseline on several test sets without sacrificing the efficiency.
  arXiv  Detail & Related papers  (2022-06-19T07:28:54Z)
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck [52.08901549360262]
  Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
  arXiv  Detail & Related papers  (2020-04-22T14:41:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.