A dynamic view of some anomalous phenomena in SGD

• URL: http://arxiv.org/abs/2505.01751v3
• Date: Sun, 14 Sep 2025 15:54:59 GMT
• Title: A dynamic view of some anomalous phenomena in SGD
• Abstract summary: It has been observed that over-parametrized neural networks exhibit a double descent' phenomenon.<n>Another anomalous phenomenon is that of itgrokking wherein two regimes of descent are interrupted by a third regime wherein the mean loss remains almost constant.<n>This note presents a plausible explanation for these and related phenomena by using the theory of two time scale approximation, applied to the continuous time limit dynamics.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: It has been observed by Belkin et al.\ that over-parametrized neural networks exhibit a `double descent' phenomenon. That is, as the model complexity (as reflected in the number of features) increases, the test error initially decreases, then increases, and then decreases again. A counterpart of this phenomenon in the time domain has been noted in the context of epoch-wise training, viz., the test error decreases with the number of iterates, then increases, then decreases again. Another anomalous phenomenon is that of \textit{grokking} wherein two regimes of descent are interrupted by a third regime wherein the mean loss remains almost constant. This note presents a plausible explanation for these and related phenomena by using the theory of two time scale stochastic approximation, applied to the continuous time limit of the gradient dynamics. This gives a novel perspective for an already well studied theme.

Related papers

• Adynamical systems view of training generativemodels and the memorization phenomenon [0.0]
  We give a system theoretic explanation of the phenomenon in generative models.<n>Specifically, we use a result of Austin [2016] to motivate a stylized model for the loss function for gradient descent (SGD)<n>This naturally leads to two distinct time scales in the constant step size SGD that is commonly used in machine learning.
  arXiv  Detail & Related papers  (2026-05-19T07:34:58Z)
• New Evidence of the Two-Phase Learning Dynamics of Neural Networks [59.55028392232715]
  We introduce an interval-wise perspective that compares network states across a time window.<n>We show that the response of the network to a perturbation exhibits a transition from chaotic to stable.<n>We also find that after this transition point the model's functional trajectory is confined to a narrow cone-shaped subset.
  arXiv  Detail & Related papers  (2025-05-20T04:03:52Z)
• On the Dynamics Under the Unhinged Loss and Beyond [104.49565602940699]
  We introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze closed-form dynamics. The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization.
  arXiv  Detail & Related papers  (2023-12-13T02:11:07Z)
• A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning [68.76846801719095]
  We show that double descent appears exactly when and where it occurs, and that its location is not inherently tied to the threshold p=n. This provides a resolution to tensions between double descent and statistical intuition.
  arXiv  Detail & Related papers  (2023-10-29T12:05:39Z)
• Intensity Profile Projection: A Framework for Continuous-Time Representation Learning for Dynamic Networks [50.2033914945157]
  We present a representation learning framework, Intensity Profile Projection, for continuous-time dynamic network data. The framework consists of three stages: estimating pairwise intensity functions, learning a projection which minimises a notion of intensity reconstruction error. Moreoever, we develop estimation theory providing tight control on the error of any estimated trajectory, indicating that the representations could even be used in quite noise-sensitive follow-on analyses.
  arXiv  Detail & Related papers  (2023-06-09T15:38:25Z)
• The Closed and Open Unbalanced Dicke Trimer Model: Critical Properties and Nonlinear Semiclassical Dynamics [5.824077816472029]
  We study a generalization of the recently introduced Dicke trimer model. In the extreme unbalanced limit, the symmetry of the Tavis-Cummings model is restored. We observe the emergence of nonequilibrium phases characterized by trivial and non-trivial dynamical signatures.
  arXiv  Detail & Related papers  (2023-03-21T11:23:18Z)
• Learning time-scales in two-layers neural networks [11.878594839685471]
  We study the gradient flow dynamics of a wide two-layer neural network in high-dimension.<n>Based on new rigorous results, we propose a scenario for the learning dynamics in this setting.
  arXiv  Detail & Related papers  (2023-02-28T19:52:26Z)
• Losing momentum in continuous-time stochastic optimisation [42.617042045455506]
  momentum-based optimisation algorithms have become particularly widespread. In this work, we analyse a continuous-time model for gradient descent with momentum. We also train a convolutional neural network in an image classification problem.
  arXiv  Detail & Related papers  (2022-09-08T10:46:05Z)
• Multi-scale Feature Learning Dynamics: Insights for Double Descent [71.91871020059857]
  We study the phenomenon of "double descent" of the generalization error. We find that double descent can be attributed to distinct features being learned at different scales.
  arXiv  Detail & Related papers  (2021-12-06T18:17:08Z)
• On Convergence of Training Loss Without Reaching Stationary Points [62.41370821014218]
  We show that Neural Network weight variables do not converge to stationary points where the gradient the loss function vanishes. We propose a new perspective based on ergodic theory dynamical systems.
  arXiv  Detail & Related papers  (2021-10-12T18:12:23Z)
• When and how epochwise double descent happens [7.512375012141203]
  An epochwise double descent' effect exists in which the generalization error initially drops, then rises, and finally drops again with increasing training time. This presents a practical problem in that the amount of time required for training is long, and early stopping based on validation performance may result in suboptimal generalization. We show that epochwise double descent requires a critical amount of noise to occur, but above a second critical noise level early stopping remains effective.
  arXiv  Detail & Related papers  (2021-08-26T19:19:17Z)
• Optimization Variance: Exploring Generalization Properties of DNNs [83.78477167211315]
  The test error of a deep neural network (DNN) often demonstrates double descent. We propose a novel metric, optimization variance (OV), to measure the diversity of model updates.
  arXiv  Detail & Related papers  (2021-06-03T09:34:17Z)
• Noise-induced degeneration in online learning [0.0]
  We show that a plateau phenomenon emerges as a noise-induced degeneration, which is not observed in deterministic gradient descent. The noise-induced degeneration observed herein is expected to be found in a broad class of machine learning via neural networks.
  arXiv  Detail & Related papers  (2020-08-24T15:03:58Z)
• Early Stopping in Deep Networks: Double Descent and How to Eliminate it [30.61588337557343]
  We show that epoch-wise double descent arises because different parts of the network are learned at different epochs. We study two standard convolutional networks empirically and show that eliminating epoch-wise double descent through adjusting stepsizes of different layers improves the early stopping performance significantly.
  arXiv  Detail & Related papers  (2020-07-20T13:43:33Z)
• Feedback-induced instabilities and dynamics in the Jaynes-Cummings model [62.997667081978825]
  We investigate the coherence and steady-state properties of the Jaynes-Cummings model subjected to time-delayed coherent feedback. The introduced feedback qualitatively modifies the dynamical response and steady-state quantum properties of the system.
  arXiv  Detail & Related papers  (2020-06-20T10:07:01Z)
• Data-Dependence of Plateau Phenomenon in Learning with Neural Network --- Statistical Mechanical Analysis [5.299781502918034]
  plateau phenomenon, wherein the loss value stops decreasing during the process of learning, has been reported by various researchers. In this paper, using statistical mechanical formulation, we clarified the relationship between the plateau phenomenon and the statistical property of the data learned.
  arXiv  Detail & Related papers  (2020-01-10T10:17:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.