Multiple Descents in Deep Learning as a Sequence of Order-Chaos Transitions

• URL: http://arxiv.org/abs/2505.20030v1
• Date: Mon, 26 May 2025 14:18:22 GMT
• Title: Multiple Descents in Deep Learning as a Sequence of Order-Chaos Transitions
• Abstract summary: We observe a novel'multiple-descent' phenomenon during the training process of LSTM, in which the test loss goes through long cycles of up and down trend multiple times after the model is overtrained.<n>By carrying out stability analysis of the models, we found that the cycles in test loss are closely associated with the phase transition process between order and chaos.
• Score: 4.424479010649862
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We observe a novel 'multiple-descent' phenomenon during the training process of LSTM, in which the test loss goes through long cycles of up and down trend multiple times after the model is overtrained. By carrying out asymptotic stability analysis of the models, we found that the cycles in test loss are closely associated with the phase transition process between order and chaos, and the local optimal epochs are consistently at the critical transition point between the two phases. More importantly, the global optimal epoch occurs at the first transition from order to chaos, where the 'width' of the 'edge of chaos' is the widest, allowing the best exploration of better weight configurations for learning.

Related papers

• Grokking and epoch-wise double descent in quantum neural networks [0.08739101659113156]
  We report the empirical observation of both the grokking transition and epoch-wise double descent in a two-qubit quantum neural network (QNN)<n>We identify a generalization decay in late-stage training, where the test error increases significantly despite a stagnant training loss.<n>Our analysis reveals that this decay correlates with an unconstrained increase of the weight-norm, drifting away from sparse, phase-aligned harmonic solutions.
  arXiv  Detail & Related papers  (2026-07-09T10:55:16Z)
• The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training [48.37150760049573]
  We identify an underlying spectral phenomenon, Stability of Singular Distribution (SoSD), where the trace-normalized singular value spectrum stabilizes early.<n>We demonstrate that synchronization between SoSD and the slow-descent regime is widely observed across diverse architectures.<n>We further interpret strategies like WSD and Muon through their ability to modulate the SoSD scale, offering a spectral lens for understanding efficient pre-training dynamics.
  arXiv  Detail & Related papers  (2026-05-26T03:03:24Z)
• Attention to Order: Transformers Discover Phase Transitions via Learnability [0.0]
  We introduce learnability as a universal criterion, defined as the ability of a transformer model containing attention mechanism to extract structure from microscopic states.<n>Using self-supervised learning and Monte Carlo generated configurations of the two-dimensional Ising model, we show that ordered phases correspond to enhanced learnability.<n>Our results establish learnability as a data-driven marker of phase transitions and highlight deep parallels between long-range order in condensed matter and the emergence of structure in modern language models.
  arXiv  Detail & Related papers  (2025-10-08T18:00:59Z)
• New Evidence of the Two-Phase Learning Dynamics of Neural Networks [59.55028392232715]
  We introduce an interval-wise perspective that compares network states across a time window.<n>We show that the response of the network to a perturbation exhibits a transition from chaotic to stable.<n>We also find that after this transition point the model's functional trajectory is confined to a narrow cone-shaped subset.
  arXiv  Detail & Related papers  (2025-05-20T04:03:52Z)
• On Multi-Stage Loss Dynamics in Neural Networks: Mechanisms of Plateau and Descent Stages [1.5235340620594793]
  We identify three distinct stages observed in the loss curve during training: the initial plateau stage, the initial descent stage, and the secondary plateau stage. Through rigorous analysis, we reveal the underlying challenges contributing to slow training during the plateau stages.
  arXiv  Detail & Related papers  (2024-10-26T08:16:00Z)
• Training Dynamics of Transformers to Recognize Word Co-occurrence via Gradient Flow Analysis [97.54180451650122]
  We study the dynamics of training a shallow transformer on a task of recognizing co-occurrence of two designated words. We analyze the gradient flow dynamics of simultaneously training three attention matrices and a linear layer. We prove a novel property of the gradient flow, termed textitautomatic balancing of gradients, which enables the loss values of different samples to decrease almost at the same rate and further facilitates the proof of near minimum training loss.
  arXiv  Detail & Related papers  (2024-10-12T17:50:58Z)
• Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective [66.80315289020487]
  Warmup-Stable-Decay (WSD) schedule uses a constant learning rate to produce a main branch of iterates that can continue indefinitely without a pre-specified compute budget.<n>We show that pretraining loss exhibits a river valley landscape, which resembles a deep valley with a river at its bottom.<n>Inspired by the theory, we introduce WSD-S, a variant of WSD that reuses previous checkpoints' decay phases and keeps only one main branch.
  arXiv  Detail & Related papers  (2024-10-07T16:49:39Z)
• Improving Diffusion Inverse Problem Solving with Decoupled Noise Annealing [84.97865583302244]
  We propose a new method called Decoupled Annealing Posterior Sampling (DAPS)<n>DAPS relies on a novel noise annealing process.<n>We demonstrate that DAPS significantly improves sample quality and stability across multiple image restoration tasks.
  arXiv  Detail & Related papers  (2024-07-01T17:59:23Z)
• Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimization [41.20978920228298]
  We show that the second phase begins once the empirical risk falls below a certain threshold, dependent on the stepsize. We also show that the normalized margin grows nearly monotonically in the second phase, demonstrating an implicit bias of GD in training non-homogeneous predictors. Our analysis applies to networks of any width, beyond the well-known neural tangent kernel and mean-field regimes.
  arXiv  Detail & Related papers  (2024-06-12T21:33:22Z)
• Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality [54.20763128054692]
  We study the dynamics of gradient flow for training a multi-head softmax attention model for in-context learning of multi-task linear regression. We prove that an interesting "task allocation" phenomenon emerges during the gradient flow dynamics.
  arXiv  Detail & Related papers  (2024-02-29T18:43:52Z)
• Attractor Memory for Long-Term Time Series Forecasting: A Chaos Perspective [63.60312929416228]
  textbftextitAttraos incorporates chaos theory into long-term time series forecasting. We show that Attraos outperforms various LTSF methods on mainstream datasets and chaotic datasets with only one-twelfth of the parameters compared to PatchTST.
  arXiv  Detail & Related papers  (2024-02-18T05:35:01Z)
• Exact Phase Transitions in Deep Learning [5.33024001730262]
  We prove that the competition between prediction error and model complexity in the training loss leads to the second-order phase transition for nets with one hidden layer and the first-order phase transition for nets with more than one hidden layer. The proposed theory is directly relevant to the optimization of neural networks and points to an origin of the posterior collapse problem in Bayesian deep learning.
  arXiv  Detail & Related papers  (2022-05-25T06:00:34Z)
• On regularization of gradient descent, layer imbalance and flat minima [9.08659783613403]
  We analyze the training dynamics for deep linear networks using a new metric - imbalance - which defines the flatness of a solution. We demonstrate that different regularization methods, such as weight decay or noise data augmentation, behave in a similar way.
  arXiv  Detail & Related papers  (2020-07-18T00:09:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.