From Density Matrices to Phase Transitions in Deep Learning: Spectral Early Warnings and Interpretability

• URL: http://arxiv.org/abs/2603.29805v2
• Date: Wed, 01 Apr 2026 10:40:57 GMT
• Title: From Density Matrices to Phase Transitions in Deep Learning: Spectral Early Warnings and Interpretability
• Abstract summary: We show that this object provides a computationally efficient, unified observable of phase transitions during training.<n>Remarkably, the top eigenvectors of the 2RDM are directly interpretable making it straightforward to study the nature of the transitions.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A key problem in the modern study of AI is predicting and understanding emergent capabilities in models during training. Inspired by methods for studying reactions in quantum chemistry, we present the ``2-datapoint reduced density matrix". We show that this object provides a computationally efficient, unified observable of phase transitions during training. By tracking the eigenvalue statistics of the 2RDM over a sliding window, we derive two complementary signals: the spectral heat capacity, which we prove provides early warning of second-order phase transitions via critical slowing down, and the participation ratio, which reveals the dimensionality of the underlying reorganization. Remarkably, the top eigenvectors of the 2RDM are directly interpretable making it straightforward to study the nature of the transitions. We validate across four distinct settings: deep linear networks, induction head formation, grokking, and emergent misalignment. We then discuss directions for future work using the 2RDM.

Related papers

• Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence [6.025438902954768]
  We show how the copy subcircuit in the first layer of an induction head is learned by analyzing a single-layer softmax attention network trained on a copy task.<n>Our work provides a first-principles theoretical account of the abrupt emergence of the copy subcircuit, reminiscent of the one observed in training large language models.
  arXiv  Detail & Related papers  (2026-06-10T13:26:56Z)
• Random Matrix Theory of Early-Stopped Gradient Flow: A Transient BBP Scenario [1.2744523252873352]
  We show that learning occurs when an isolated eigenvalue separates from a noisy bulk, before disappearing in the overfitting regime.<n>We map the corresponding phase diagrams and validate the theory against finite-size simulations.
  arXiv  Detail & Related papers  (2026-04-20T16:05:58Z)
• Emergence of Superposition: Unveiling the Training Dynamics of Chain of Continuous Thought [64.43689151961054]
  We theoretically analyze the training dynamics of a simplified two-layer transformer on the directed graph reachability problem.<n>Our analysis reveals that during training using continuous thought, the index-matching logit will first increase and then remain bounded under mild assumptions.
  arXiv  Detail & Related papers  (2025-09-27T15:23:46Z)
• Phase Transitions between Accuracy Regimes in L2 regularized Deep Neural Networks [0.0]
  Increasing the L2 regularization of Deep Neural Networks (DNNs) causes a first-order phase transition into the under-parametrized phase.<n>We predict new transition points as the data complexity is increased and, in accordance with the theory of phase transitions, the existence of effects.
  arXiv  Detail & Related papers  (2025-05-10T11:02:30Z)
• How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Bias [48.9399496805422]
  We focus on two representative tasks in the category of regular language recognition, known as even pairs' and parity check'<n>Our goal is to explore how a one-layer transformer, consisting of an attention layer followed by a linear layer, learns to solve these tasks.
  arXiv  Detail & Related papers  (2025-05-02T00:07:35Z)
• A Unified Perspective on the Dynamics of Deep Transformers [24.094975798576783]
  We study the evolution of data anisotropy through a deep Transformer.<n>We highlight a clustering phenomenon that parallels previous results in the non-normalized discrete case.
  arXiv  Detail & Related papers  (2025-01-30T13:04:54Z)
• Training Dynamics of Transformers to Recognize Word Co-occurrence via Gradient Flow Analysis [97.54180451650122]
  We study the dynamics of training a shallow transformer on a task of recognizing co-occurrence of two designated words. We analyze the gradient flow dynamics of simultaneously training three attention matrices and a linear layer. We prove a novel property of the gradient flow, termed textitautomatic balancing of gradients, which enables the loss values of different samples to decrease almost at the same rate and further facilitates the proof of near minimum training loss.
  arXiv  Detail & Related papers  (2024-10-12T17:50:58Z)
• Non-asymptotic Convergence of Training Transformers for Next-token Prediction [48.9399496805422]
  Transformers have achieved extraordinary success in modern machine learning due to their excellent ability to handle sequential data. This paper provides a fine-grained non-asymptotic analysis of the training dynamics of a one-layer transformer. We show that the trained transformer presents non-token prediction ability with dataset shift.
  arXiv  Detail & Related papers  (2024-09-25T20:22:06Z)
• Cascade of phase transitions in the training of Energy-based models [9.945465034701288]
  We investigate the feature encoding process in a prototypical energy-based generative model, the Bernoulli-Bernoulli RBM.<n>Our study tracks the evolution of the model's weight matrix through its singular value decomposition.<n>We validate our theoretical results by training the Bernoulli-Bernoulli RBM on real data sets.
  arXiv  Detail & Related papers  (2024-05-23T15:25:56Z)
• Assessing Neural Network Representations During Training Using Noise-Resilient Diffusion Spectral Entropy [55.014926694758195]
  Entropy and mutual information in neural networks provide rich information on the learning process. We leverage data geometry to access the underlying manifold and reliably compute these information-theoretic measures. We show that they form noise-resistant measures of intrinsic dimensionality and relationship strength in high-dimensional simulated data.
  arXiv  Detail & Related papers  (2023-12-04T01:32:42Z)
• In-Context Convergence of Transformers [63.04956160537308]
  We study the learning dynamics of a one-layer transformer with softmax attention trained via gradient descent. For data with imbalanced features, we show that the learning dynamics take a stage-wise convergence process.
  arXiv  Detail & Related papers  (2023-10-08T17:55:33Z)
• 2-D SSM: A General Spatial Layer for Visual Transformers [79.4957965474334]
  A central objective in computer vision is to design models with appropriate 2-D inductive bias. We leverage an expressive variation of the multidimensional State Space Model. Our approach introduces efficient parameterization, accelerated computation, and a suitable normalization scheme.
  arXiv  Detail & Related papers  (2023-06-11T09:41:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.