Phase Transitions in Large Language Models and the $O(N)$ Model

• URL: http://arxiv.org/abs/2501.16241v1
• Date: Mon, 27 Jan 2025 17:36:06 GMT
• Title: Phase Transitions in Large Language Models and the $O(N)$ Model
• Authors: Youran Sun, Babak Haghighat,
• Abstract summary: We reformulated the Transformer architecture as an $O(N)$ model to investigate phase transitions in large language models. Our study reveals two distinct phase transitions corresponding to the temperature used in text generation. As an application, the energy of the $O(N)$ model can be used to evaluate whether an LLM's parameters are sufficient to learn the training data.
• Score: 0.0
• License:
• Abstract: Large language models (LLMs) exhibit unprecedentedly rich scaling behaviors. In physics, scaling behavior is closely related to phase transitions, critical phenomena, and field theory. To investigate the phase transition phenomena in LLMs, we reformulated the Transformer architecture as an $O(N)$ model. Our study reveals two distinct phase transitions corresponding to the temperature used in text generation and the model's parameter size, respectively. The first phase transition enables us to estimate the internal dimension of the model, while the second phase transition is of \textit{higher-depth} and signals the emergence of new capabilities. As an application, the energy of the $O(N)$ model can be used to evaluate whether an LLM's parameters are sufficient to learn the training data.

Related papers

• Towards Neural Scaling Laws for Time Series Foundation Models [63.5211738245487]
  We examine two common TSFM architectures, encoder-only and decoder-only Transformers, and investigate their scaling behavior on both ID and OOD data. Our experiments reveal that the log-likelihood loss of TSFMs exhibits similar scaling behavior in both OOD and ID settings. We provide practical guidelines for designing and scaling larger TSFMs with enhanced model capabilities.
  arXiv  Detail & Related papers  (2024-10-16T08:23:39Z)
• Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers [54.20763128054692]
  We study how a two-attention-layer transformer is trained to perform ICL on $n$-gram Markov chain data. We prove that the gradient flow with respect to a cross-entropy ICL loss converges to a limiting model.
  arXiv  Detail & Related papers  (2024-09-09T18:10:26Z)
• SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models [85.67096251281191]
  We present an innovative approach to model fusion called zero-shot Sparse MIxture of Low-rank Experts (SMILE) construction. SMILE allows for the upscaling of source models into an MoE model without extra data or further training. We conduct extensive experiments across diverse scenarios, such as image classification and text generation tasks, using full fine-tuning and LoRA fine-tuning.
  arXiv  Detail & Related papers  (2024-08-19T17:32:15Z)
• Phase Transitions in the Output Distribution of Large Language Models [0.9374652839580183]
  In a physical system, changing parameters such as temperature can induce a phase transition: an abrupt change from one state of matter to another. The task of identifying phase transitions requires human analysis and some prior understanding of the system to narrow down which low-dimensional properties to monitor and analyze. Statistical methods for the automated detection of phase transitions from data have recently been proposed within the physics community. We quantify distributional changes in the generated output via statistical distances, which can be efficiently estimated with access to the probability distribution over next-tokens.
  arXiv  Detail & Related papers  (2024-05-27T12:04:36Z)
• Cascade of phase transitions in the training of Energy-based models [9.945465034701288]
  We investigate the feature encoding process in a prototypical energy-based generative model, the Bernoulli-Bernoulli RBM. Our study tracks the evolution of the model's weight matrix through its singular value decomposition. We validate our theoretical results by training the Bernoulli-Bernoulli RBM on real data sets.
  arXiv  Detail & Related papers  (2024-05-23T15:25:56Z)
• Dynamics Reflects Quantum Phase Transition of Rabi Model [0.0]
  A breakdown in the rotating wave approximation of the Rabi model leads to phase transition versus coupling strength. We show that the dynamics of physical quantities can reflect such a phase transition for this model. This work offers an idea to explore phase transitions by non-equilibrium process for open quantum systems.
  arXiv  Detail & Related papers  (2023-09-13T14:45:07Z)
• 2-D SSM: A General Spatial Layer for Visual Transformers [79.4957965474334]
  A central objective in computer vision is to design models with appropriate 2-D inductive bias. We leverage an expressive variation of the multidimensional State Space Model. Our approach introduces efficient parameterization, accelerated computation, and a suitable normalization scheme.
  arXiv  Detail & Related papers  (2023-06-11T09:41:37Z)
• Learning Bounded Context-Free-Grammar via LSTM and the Transformer:Difference and Explanations [51.77000472945441]
  Long Short-Term Memory (LSTM) and Transformers are two popular neural architectures used for natural language processing tasks. In practice, it is often observed that Transformer models have better representation power than LSTM. We study such practical differences between LSTM and Transformer and propose an explanation based on their latent space decomposition patterns.
  arXiv  Detail & Related papers  (2021-12-16T19:56:44Z)
• MoEfication: Conditional Computation of Transformer Models for Efficient Inference [66.56994436947441]
  Transformer-based pre-trained language models can achieve superior performance on most NLP tasks due to large parameter capacity, but also lead to huge computation cost. We explore to accelerate large-model inference by conditional computation based on the sparse activation phenomenon. We propose to transform a large model into its mixture-of-experts (MoE) version with equal model size, namely MoEfication.
  arXiv  Detail & Related papers  (2021-10-05T02:14:38Z)
• Kosterlitz-Thouless phase and $Z_d$ topological quantum phase [0.0]
  We find a corresponding quantum model constructed by applying a local invertible transformation on a d-level version of Kitaev's Toric code. We identify an extended topological phase transition in our model in a sense that, for $d geq 5$, a KT-like quantum phase emerges between a $Z_d$ topological phase and a trivial phase.
  arXiv  Detail & Related papers  (2020-04-30T10:16:59Z)
• Discrete truncated Wigner approach to dynamical phase transitions in Ising models after a quantum quench [0.0]
  We study dynamical phase transitions arising in the steady state of transverse-field Ising models after a quantum quench. We find identical exponents for $alpha lesssim 0.5$, suggesting that the dynamical transitions in this regime fall into the same universality class as the nonergodic mean-field limit.
  arXiv  Detail & Related papers  (2020-04-21T08:20:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.