Generative Criticality in Large Language Model Temperature Scaling

• URL: http://arxiv.org/abs/2606.06238v1
• Date: Thu, 04 Jun 2026 14:43:32 GMT
• Title: Generative Criticality in Large Language Model Temperature Scaling
• Abstract summary: We propose a statistical-field framework for text generated by large language models (LLMs)<n>We define a susceptibility from the connected two-point correlator and an order parameter from the ensemble-averaged embedding field.<n>We observe a sharp susceptibility peak near a characteristic $T_c$ with power-law-like scaling, a concurrent rapid change in the order parameter, and a collapse onto a single semantic direction below $T_c$.
• Score: 3.472734054649125
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We propose a statistical-field framework for text generated by large language models (LLMs), treating token embeddings as continuous spin variables on a one-dimensional chain. Defining a susceptibility from the connected two-point correlator and an order parameter from the ensemble-averaged embedding field, we vary the \texttt{softmax} temperature $T$ and observe a sharp susceptibility peak near a characteristic $T_c$ with power-law-like scaling, a concurrent rapid change in the order parameter, and a collapse onto a single semantic direction below $T_c$. The intrinsic dimension estimated by the two nearest neighbor (TwoNN) method independently corroborates these findings, reaching a minimum near $T_c$. Results are robust across model scales (Qwen3: 0.6B--32B) and prompt categories. While the phenomenology closely resembles a continuous phase transition, the non-equilibrium nature of autoregressive generation warrants further investigation. Our framework provides quantitative tools for probing the collective statistical structure of LLM outputs and suggests connections between decoding strategies and critical phenomena.

Related papers

• The Geometry of Semantic Space: A Continuous Geometric Framework for the Transformer Architecture [0.7310043452300737]
  We present a continuous framework that models the discrete operations of the Transformer architecture as an integro-differential equation (IDE) on a fiber bundle.<n>We conduct a six-part experimental campaign spanning $124$M to $8$B parameters.<n>The results demonstrate that analyzing Transformers through the lens of differential geometry provides a descriptive vocabulary for large language models.
  arXiv  Detail & Related papers  (2026-07-19T09:08:51Z)
• Collective Alignment in LLM Multi-Agent Systems: Disentangling Bias from Cooperation via Statistical Physics [0.0]
  We present a model-agnostic statistical-physics method to disentangle social conformity from intrinsic bias, compute critical exponents, and probe the collective behavior and possible phase transitions of multi-agent systems.<n>We find that collective alignment is dominated by an intrinsic bias rather than by cooperative neighbor coupling, producing field-driven crossovers instead of genuine phase transitions.
  arXiv  Detail & Related papers  (2026-05-11T13:13:44Z)
• Self-Attention as a Covariance Readout: A Unified View of In-Context Learning and Repetition [8.250374560598495]
  Large language models (LLMs) exhibit two behaviours: in-context learning (ICL) and repetitive generation.<n>In both, the model behaves as though it had summarised the context into a population-level statistic and discarded token-level detail.<n>We ask whether this summarisation and forgetting' can be derived from the attention mechanism itself, and answer in the affirmative.
  arXiv  Detail & Related papers  (2026-05-11T12:33:15Z)
• Towards a Data-Parameter Correspondence for LLMs: A Preliminary Discussion [10.237445112597095]
  Large language model optimization has historically bifurcated into isolated data-centric and model-centric paradigms.<n>This paper establishes a unified emphdata- parameter correspondence, revealing these seemingly disparate operations as dual manifestations of the same geometric structure.
  arXiv  Detail & Related papers  (2026-04-19T11:18:03Z)
• Phase Transition for Budgeted Multi-Agent Synergy [41.486076708302456]
  Multi-agent systems can improve reliability, yet under a fixed inference budget they often help, saturate, or even collapse.<n>We develop a minimal and calibratable theory that predicts these regimes from three binding constraints of modern agent stacks.
  arXiv  Detail & Related papers  (2026-01-24T05:32:50Z)
• Test time training enhances in-context learning of nonlinear functions [51.56484100374058]
  Test-time training (TTT) enhances model performance by explicitly updating designated parameters prior to each prediction.<n>We investigate the combination of TTT with in-context learning (ICL), where the model is given a few examples from the target distribution at inference time.
  arXiv  Detail & Related papers  (2025-09-30T03:56:44Z)
• Beyond Softmax: A Natural Parameterization for Categorical Random Variables [61.709831225296305]
  We introduce the $textitcatnat$ function, a function composed of a sequence of hierarchical binary splits.<n>A rich set of experiments show that the proposed function improves the learning efficiency and yields models characterized by consistently higher test performance.
  arXiv  Detail & Related papers  (2025-09-29T12:55:50Z)
• InfiGFusion: Graph-on-Logits Distillation via Efficient Gromov-Wasserstein for Model Fusion [36.27704594180795]
  InfiGFusion is a structure-aware fusion framework with a novel textitGraph-on-Logits Distillation (GLD) loss.<n>We show that GLD consistently improves fusion quality and stability.<n>It shows particular strength in complex reasoning tasks, with +35.6 improvement on Multistep Arithmetic and +37.06 on Causal Judgement over SFT.
  arXiv  Detail & Related papers  (2025-05-20T03:55:35Z)
• Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers [54.20763128054692]
  We study how a two-attention-layer transformer is trained to perform ICL on $n$-gram Markov chain data. We prove that the gradient flow with respect to a cross-entropy ICL loss converges to a limiting model.
  arXiv  Detail & Related papers  (2024-09-09T18:10:26Z)
• Latent Semantic Consensus For Deterministic Geometric Model Fitting [109.44565542031384]
  We propose an effective method called Latent Semantic Consensus (LSC) LSC formulates the model fitting problem into two latent semantic spaces based on data points and model hypotheses. LSC is able to provide consistent and reliable solutions within only a few milliseconds for general multi-structural model fitting.
  arXiv  Detail & Related papers  (2024-03-11T05:35:38Z)
• Towards Faster Non-Asymptotic Convergence for Diffusion-Based Generative Models [49.81937966106691]
  We develop a suite of non-asymptotic theory towards understanding the data generation process of diffusion models. In contrast to prior works, our theory is developed based on an elementary yet versatile non-asymptotic approach.
  arXiv  Detail & Related papers  (2023-06-15T16:30:08Z)
• On the Discrepancy between Density Estimation and Sequence Generation [92.70116082182076]
  log-likelihood is highly correlated with BLEU when we consider models within the same family. We observe no correlation between rankings of models across different families.
  arXiv  Detail & Related papers  (2020-02-17T20:13:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.