Scaling limit of the Random Language Model

• URL: http://arxiv.org/abs/2606.28105v1
• Date: Fri, 26 Jun 2026 14:00:30 GMT
• Title: Scaling limit of the Random Language Model
• Abstract summary: We develop a theory of the Random Language Model (RLM), an ensemble of context-free grammars.<n>We show that the RLM exhibits a condensation transition at a critical value $x_c=1/8$, below which rule usage concentrates and language statistics acquire a nontrivial dependence on corpus length.<n>Across these regimes, we derive explicit scaling laws for the number of distinct rules, entropy, and related observables.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We develop a quantitative theory of the Random Language Model (RLM), an ensemble of stochastic context-free grammars, in a scaling limit where the number of hidden symbols $N \to \infty$ while the grammar temperature $\tildeε_d \to 0$ at fixed $x = {\tildeε}_d \log N$. In this limit, the model admits a controlled description based on a large-deviation principle over rule-usage patterns. A semi-annealed approximation maps the problem to a class of Random Energy Models with nontrivial combinatorics. We show that the RLM exhibits a condensation transition at a critical value $x_c=1/8$, below which rule usage concentrates and language statistics acquire a nontrivial dependence on corpus length. A second characteristic scale at $x=1/2$ marks the onset of entropy reduction from its maximal value. Across these regimes, we derive explicit scaling laws for the number of distinct rules, entropy, and related observables, identifying distinct scaling, saturation, and critical regimes controlled by the interplay of grammar size, corpus length, and temperature. The theory resolves previous ambiguities regarding the existence of a thermodynamic transition and explains the slow approach to the large-$N$ limit as a consequence of the dependence on $\log N$. It further provides a unified framework in which universal statistical properties of language emerge from typical realizations of generative grammars, with implications for both natural language statistics and the behavior of large language models.

Related papers

• Generative Criticality in Large Language Model Temperature Scaling [3.472734054649125]
  We propose a statistical-field framework for text generated by large language models (LLMs)<n>We define a susceptibility from the connected two-point correlator and an order parameter from the ensemble-averaged embedding field.<n>We observe a sharp susceptibility peak near a characteristic $T_c$ with power-law-like scaling, a concurrent rapid change in the order parameter, and a collapse onto a single semantic direction below $T_c$.
  arXiv  Detail & Related papers  (2026-06-04T14:43:32Z)
• Information Hierarchy in Many-Body Berry Phase [0.0]
  Many-body topology is a central concept in modern theories of solids.<n>This work studies the extent to which geometric information of an interacting many-body ground state can be inferred from a finite number of local correlations.
  arXiv  Detail & Related papers  (2026-06-01T12:20:41Z)
• Asymptotic distinguishability of Haar-averaged measurement models [0.0]
  We study discrimination problems generated by the same basic Haar-random measurement mechanism.<n>We derive expressions in the fixed-$N$, large-$d$, the fixed-$d$, large-$N$ regime, the sparse joint-scaling regime $N=o(sqrt d)$, and the critical scaling regime $N/sqrt dto c$.
  arXiv  Detail & Related papers  (2026-05-29T11:20:52Z)
• Fine-Tuning Improves Information Conveyance in Language Models [34.13231597052981]
  We propose Canopy Entropy ($mathrmCEstar$), a measure that views language generation from a tree perspective.<n>$mathrmCEstar$ captures uncertainty in both the output length $N$ and the generated sequence $Y_1:N$.<n>We find that fine-tuning nearly triples the correlation strength between entropy rate and semantic diversity.
  arXiv  Detail & Related papers  (2026-05-29T05:05:01Z)
• Self-Attention as a Covariance Readout: A Unified View of In-Context Learning and Repetition [8.250374560598495]
  Large language models (LLMs) exhibit two behaviours: in-context learning (ICL) and repetitive generation.<n>In both, the model behaves as though it had summarised the context into a population-level statistic and discarded token-level detail.<n>We ask whether this summarisation and forgetting' can be derived from the attention mechanism itself, and answer in the affirmative.
  arXiv  Detail & Related papers  (2026-05-11T12:33:15Z)
• Information Locality as an Inductive Bias for Neural Language Models [52.92279412466086]
  We show that $m$local entropy are difficult for Transformer and LSTM LMs to learn languages.<n>These results suggest that neurals are highly sensitive to the statistical structure of a language.
  arXiv  Detail & Related papers  (2025-06-05T15:21:05Z)
• Compute-Optimal LLMs Provably Generalize Better With Scale [102.29926217670926]
  We develop generalization bounds on the pretraining objective of large language models (LLMs) in the compute-optimal regime.<n>We introduce a novel, fully empirical Freedman-type martingale concentration that tightens existing bounds by accounting for the variance of the loss function.<n>We produce a scaling law for the generalization gap, with bounds that become predictably stronger with scale.
  arXiv  Detail & Related papers  (2025-04-21T16:26:56Z)
• Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Sampling [90.86991492288487]
  evaluating constraint on every token can be prohibitively expensive.<n> LCD can distort the global distribution over strings, sampling tokens based only on local information.<n>We show that our approach is superior to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-04-07T18:30:18Z)
• Non-asymptotic bounds for forward processes in denoising diffusions: Ornstein-Uhlenbeck is hard to beat [49.1574468325115]
  This paper presents explicit non-asymptotic bounds on the forward diffusion error in total variation (TV)<n>We parametrise multi-modal data distributions in terms of the distance $R$ to their furthest modes and consider forward diffusions with additive and multiplicative noise.
  arXiv  Detail & Related papers  (2024-08-25T10:28:31Z)
• A Pseudo-Semantic Loss for Autoregressive Models with Logical Constraints [87.08677547257733]
  Neuro-symbolic AI bridges the gap between purely symbolic and neural approaches to learning. We show how to maximize the likelihood of a symbolic constraint w.r.t the neural network's output distribution. We also evaluate our approach on Sudoku and shortest-path prediction cast as autoregressive generation.
  arXiv  Detail & Related papers  (2023-12-06T20:58:07Z)
• Theory of free fermions under random projective measurements [43.04146484262759]
  We develop an analytical approach to the study of one-dimensional free fermions subject to random projective measurements of local site occupation numbers. We derive a non-linear sigma model (NLSM) as an effective field theory of the problem.
  arXiv  Detail & Related papers  (2023-04-06T15:19:33Z)
• Minimax Optimal Quantization of Linear Models: Information-Theoretic Limits and Efficient Algorithms [59.724977092582535]
  We consider the problem of quantizing a linear model learned from measurements. We derive an information-theoretic lower bound for the minimax risk under this setting. We show that our method and upper-bounds can be extended for two-layer ReLU neural networks.
  arXiv  Detail & Related papers  (2022-02-23T02:39:04Z)
• Many-Body Quantum Chaos and Space-time Translational Invariance [0.0]
  We study the consequences of having translational invariance in space and in time in many-body quantum chaotic systems. We consider an ensemble of random quantum circuits, composed of single-site random unitaries and nearest neighbour couplings. We numerically demonstrate, with simulations of two distinct circuit models, that in such a scaling limit, most microscopic details become unimportant.
  arXiv  Detail & Related papers  (2021-09-09T18:00:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.