Phase Transitions in the Output Distribution of Large Language Models

• URL: http://arxiv.org/abs/2405.17088v1
• Date: Mon, 27 May 2024 12:04:36 GMT
• Title: Phase Transitions in the Output Distribution of Large Language Models
• Authors: Julian Arnold, Flemming Holtorf, Frank Schäfer, Niels Lörch,
• Abstract summary: In a physical system, changing parameters such as temperature can induce a phase transition: an abrupt change from one state of matter to another. The task of identifying phase transitions requires human analysis and some prior understanding of the system to narrow down which low-dimensional properties to monitor and analyze. Statistical methods for the automated detection of phase transitions from data have recently been proposed within the physics community. We quantify distributional changes in the generated output via statistical distances, which can be efficiently estimated with access to the probability distribution over next-tokens.
• Score: 0.9374652839580183
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In a physical system, changing parameters such as temperature can induce a phase transition: an abrupt change from one state of matter to another. Analogous phenomena have recently been observed in large language models. Typically, the task of identifying phase transitions requires human analysis and some prior understanding of the system to narrow down which low-dimensional properties to monitor and analyze. Statistical methods for the automated detection of phase transitions from data have recently been proposed within the physics community. These methods are largely system agnostic and, as shown here, can be adapted to study the behavior of large language models. In particular, we quantify distributional changes in the generated output via statistical distances, which can be efficiently estimated with access to the probability distribution over next-tokens. This versatile approach is capable of discovering new phases of behavior and unexplored transitions -- an ability that is particularly exciting in light of the rapid development of language models and their emergent capabilities.

Related papers

• Phase Transitions in Large Language Models and the $O(N)$ Model [0.0]
  We reformulated the Transformer architecture as an $O(N)$ model to investigate phase transitions in large language models. Our study reveals two distinct phase transitions corresponding to the temperature used in text generation. As an application, the energy of the $O(N)$ model can be used to evaluate whether an LLM's parameters are sufficient to learn the training data.
  arXiv  Detail & Related papers  (2025-01-27T17:36:06Z)
• First numerical observation of the Berezinskii-Kosterlitz-Thouless transition in language models [1.4061979259370274]
  We numerically demonstrate an unambiguous phase transition in the framework of a natural language model. We identify the phase transition as a variant of the Berezinskii-Kosterlitz-Thouless transition.
  arXiv  Detail & Related papers  (2024-12-02T07:32:32Z)
• Synthetic location trajectory generation using categorical diffusion models [50.809683239937584]
  Diffusion models (DPMs) have rapidly evolved to be one of the predominant generative models for the simulation of synthetic data. We propose using DPMs for the generation of synthetic individual location trajectories (ILTs) which are sequences of variables representing physical locations visited by individuals.
  arXiv  Detail & Related papers  (2024-02-19T15:57:39Z)
• Neural-Network Decoders for Measurement Induced Phase Transitions [0.0]
  Measurement-induced entanglement phase transitions in monitored quantum systems are a striking example. We propose a neural network decoder to determine the state of the reference qubits conditioned on the measurement outcomes. We show that the entanglement phase transition manifests itself as a stark change in the learnability of the decoder function.
  arXiv  Detail & Related papers  (2022-04-22T19:40:26Z)
• Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion [88.45326906116165]
  We present a new framework to formulate the trajectory prediction task as a reverse process of motion indeterminacy diffusion (MID) We encode the history behavior information and the social interactions as a state embedding and devise a Transformer-based diffusion model to capture the temporal dependencies of trajectories. Experiments on the human trajectory prediction benchmarks including the Stanford Drone and ETH/UCY datasets demonstrate the superiority of our method.
  arXiv  Detail & Related papers  (2022-03-25T16:59:08Z)
• Replacing neural networks by optimal analytical predictors for the detection of phase transitions [0.10152838128195464]
  We derive analytical expressions for the optimal output of three widely used NN-based methods for detecting phase transitions. The inner workings of the considered methods are revealed through the explicit dependence of the optimal output on the input data. Our theoretical results are supported by extensive numerical simulations covering, e.g., topological, quantum, and many-body localization phase transitions.
  arXiv  Detail & Related papers  (2022-02-14T19:00:03Z)
• Finite-size scalings in measurement-induced dynamical phase transition [0.0]
  We study the fate of the many-body quantum Zeno transition if the system is allowed to evolve repetitively under unitary dynamics. We use different diagnostics, such as long-time evolved entanglement entropy, purity and their fluctuations in order to characterize the transition.
  arXiv  Detail & Related papers  (2021-07-30T14:11:22Z)
• Probing the topological Anderson transition with quantum walks [48.7576911714538]
  We consider one-dimensional quantum walks in optical linear networks with synthetically introduced disorder and tunable system parameters. The option to directly monitor the walker's probability distribution makes this optical platform ideally suited for the experimental observation of the unique signatures of the one-dimensional topological Anderson transition.
  arXiv  Detail & Related papers  (2021-02-01T21:19:15Z)
• Unsupervised machine learning of topological phase transitions from experimental data [52.77024349608834]
  We apply unsupervised machine learning techniques to experimental data from ultracold atoms. We obtain the topological phase diagram of the Haldane model in a completely unbiased fashion. Our work provides a benchmark for unsupervised detection of new exotic phases in complex many-body systems.
  arXiv  Detail & Related papers  (2021-01-14T16:38:21Z)
• Unsupervised machine learning of quantum phase transitions using diffusion maps [77.34726150561087]
  We show that the diffusion map method, which performs nonlinear dimensionality reduction and spectral clustering of the measurement data, has significant potential for learning complex phase transitions unsupervised. This method works for measurements of local observables in a single basis and is thus readily applicable to many experimental quantum simulators.
  arXiv  Detail & Related papers  (2020-03-16T18:40:13Z)
• Transformer Hawkes Process [79.16290557505211]
  We propose a Transformer Hawkes Process (THP) model, which leverages the self-attention mechanism to capture long-term dependencies. THP outperforms existing models in terms of both likelihood and event prediction accuracy by a notable margin. We provide a concrete example, where THP achieves improved prediction performance for learning multiple point processes when incorporating their relational information.
  arXiv  Detail & Related papers  (2020-02-21T13:48:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.