Phase transition on a context-sensitive random language model with short range interactions

• URL: http://arxiv.org/abs/2604.00947v1
• Date: Wed, 01 Apr 2026 14:19:11 GMT
• Title: Phase transition on a context-sensitive random language model with short range interactions
• Abstract summary: We numerically investigate the statistical properties of a random language model with short-range interactions.<n>We find that finite-temperature phase transitions in language models are genuinely induced by the intrinsic nature of language.
• Score: 1.4874449172133888
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Since the random language model was proposed by E. DeGiuli [Phys. Rev. Lett. 122, 128301], language models have been investigated intensively from the viewpoint of statistical mechanics. Recently, the existence of a Berezinskii--Kosterlitz--Thouless transition was numerically demonstrated in models with long-range interactions between symbols. In statistical mechanics, it has long been known that long-range interactions can induce phase transitions. Therefore, it has remained unclear whether phase transitions observed in language models originate from genuinely linguistic properties that are absent in conventional spin models. In this study, we construct a random language model with short-range interactions and numerically investigate its statistical properties. Our model belongs to the class of context-sensitive grammars in the Chomsky hierarchy and allows explicit reference to contexts. We find that a phase transition occurs even when the model refers only to contexts whose length remains constant with respect to the sentence length. This result indicates that finite-temperature phase transitions in language models are genuinely induced by the intrinsic nature of language, rather than by long-range interactions.

Related papers

• The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context [76.36635039699112]
  Large language models (LLMs) are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context.<n>We find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level.<n>This aggregate stability, however, masks significant per-example instability.
  arXiv  Detail & Related papers  (2026-07-14T17:01:12Z)
• Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMs [50.07451351559251]
  We present a study across five typologically distinct languages (English, Russian, German, Hindi, and Vietnamese)<n>We examine how position bias interacts with prompt strategies and affects output entropy.
  arXiv  Detail & Related papers  (2025-05-22T02:23:00Z)
• First numerical observation of the Berezinskii-Kosterlitz-Thouless transition in language models [1.4061979259370274]
  We numerically demonstrate an unambiguous phase transition in the framework of a natural language model.<n>We identify the phase transition as a variant of the Berezinskii-Kosterlitz-Thouless transition.
  arXiv  Detail & Related papers  (2024-12-02T07:32:32Z)
• Learning thresholds lead to stable language coexistence [0.0]
  We introduce a language competition model that incorporates the effects of memory and learning in the language shift dynamics. On a coarse grained time scale, the effects of memory and learning can be expressed as thresholds on the speakers fractions of the competing languages.
  arXiv  Detail & Related papers  (2024-06-14T14:24:02Z)
• Modeling language contact with the Iterated Learning Model [0.0]
  Iterated learning models are agent-based models of language change. A recently introduced type of iterated learning model, the Semi-Supervised ILM is used to simulate language contact.
  arXiv  Detail & Related papers  (2024-06-11T01:43:23Z)
• Robustness of the Random Language Model [0.0]
  The model suggests a simple picture of first language learning as a type of annealing in the vast space of potential languages. It implies a single continuous transition to grammatical syntax, at which the symmetry among potential words and categories is spontaneously broken. Results are discussed in light of theory of first-language acquisition in linguistics, and recent successes in machine learning.
  arXiv  Detail & Related papers  (2023-09-26T13:14:35Z)
• Quantum Effects on the Synchronization Dynamics of the Kuramoto Model [62.997667081978825]
  We show that quantum fluctuations hinder the emergence of synchronization, albeit not entirely suppressing it. We derive an analytical expression for the critical coupling, highlighting its dependence on the model parameters.
  arXiv  Detail & Related papers  (2023-06-16T16:41:16Z)
• Model Criticism for Long-Form Text Generation [113.13900836015122]
  We apply a statistical tool, model criticism in latent space, to evaluate the high-level structure of generated text. We perform experiments on three representative aspects of high-level discourse -- coherence, coreference, and topicality. We find that transformer-based language models are able to capture topical structures but have a harder time maintaining structural coherence or modeling coreference.
  arXiv  Detail & Related papers  (2022-10-16T04:35:58Z)
• Multi-timescale Representation Learning in LSTM Language Models [69.98840820213937]
  Language models must capture statistical dependencies between words at timescales ranging from very short to very long. We derived a theory for how the memory gating mechanism in long short-term memory language models can capture power law decay. Experiments showed that LSTM language models trained on natural English text learn to approximate this theoretical distribution.
  arXiv  Detail & Related papers  (2020-09-27T02:13:38Z)
• Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans [75.15855405318855]
  We studied whether a modern artificial neural network trained with "deep learning" methods mimics a central aspect of human sentence processing. Although the network was solely trained to predict the next word in a large corpus, analysis showed the emergence of specialized units that successfully handled local and long-distance syntactic agreement. We tested the model's predictions in a behavioral experiment where humans detected violations in number agreement in sentences with systematic variations in the singular/plural status of multiple nouns.
  arXiv  Detail & Related papers  (2020-06-19T12:00:05Z)
• Limits of Detecting Text Generated by Large-Scale Language Models [65.46403462928319]
  Some consider large-scale language models that can generate long and coherent pieces of text as dangerous, since they may be used in misinformation campaigns. Here we formulate large-scale language model output detection as a hypothesis testing problem to classify text as genuine or generated.
  arXiv  Detail & Related papers  (2020-02-09T19:53:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.