Random Matrix Theory of Early-Stopped Gradient Flow: A Transient BBP Scenario

• URL: http://arxiv.org/abs/2604.18450v1
• Date: Mon, 20 Apr 2026 16:05:58 GMT
• Title: Random Matrix Theory of Early-Stopped Gradient Flow: A Transient BBP Scenario
• Abstract summary: We show that learning occurs when an isolated eigenvalue separates from a noisy bulk, before disappearing in the overfitting regime.<n>We map the corresponding phase diagrams and validate the theory against finite-size simulations.
• Score: 1.2744523252873352
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Empirical studies of trained models often report a transient regime in which signal is detectable in a finite gradient descent time window before overfitting dominates. We provide an analytically tractable random-matrix model that reproduces this phenomenon for gradient flow in a linear teacher--student setting. In this framework, learning occurs when an isolated eigenvalue separates from a noisy bulk, before eventually disappearing in the overfitting regime. The key ingredient is anisotropy in the input covariance, which induces fast and slow directions in the learning dynamics. In a two-block covariance model, we derive the full time-dependent bulk spectrum of the symmetrized weight matrix through a $2\times 2$ Dyson equation, and we obtain an explicit outlier condition for a rank-one teacher via a rank-two determinant formula. This yields a transient Baik-Ben Arous-Péché (BBP) transition: depending on signal strength and covariance anisotropy, the teacher spike may never emerge, emerge and persist, or emerge only during an intermediate time interval before being reabsorbed into the bulk. We map the corresponding phase diagrams and validate the theory against finite-size simulations. Our results provide a minimal solvable mechanism for early stopping as a transient spectral effect driven by anisotropy and noise.

Related papers

• Posterior Information Dynamics of Diffusion Models for Linear Inverse Problems [11.548853370822343]
  We study the process through the smoothed likelihood force, the difference between exact posterior and prior scores at each noise level.<n>Averaging over measurements yields an information--minimum mean-square error (I-MMSE) identity linking information gain to denoising-error reduction.
  arXiv  Detail & Related papers  (2026-08-22T00:53:42Z)
• From Atoms to Entropy: Optimal Noise Allocation for Diffusion Training in the Convex Regime [52.459513910242485]
  We develop a general statistical framework for studying optimal noise-level allocation in diffusion training.<n>We show that the optimized training schedule admits an atomic minimizer, concentrated on finitely many noise levels.<n>We then evaluate the entropic schedule in larger-scale experiments, where full schedule optimization is currently intractable.
  arXiv  Detail & Related papers  (2026-07-10T20:59:06Z)
• Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reduction [18.72807692009739]
  Grokking suggests that fitting training data and learning a simple underlying rule may occur on different time scales.<n>We formalize this phenomenon by separating the fast decay of the classification loss from the slower simplification of the learned representation.<n>We show that a post-margin gap-growth or one-step-contraction condition reduces the cross-entropy loss to level epsilon on a logarithmic time scale.
  arXiv  Detail & Related papers  (2026-06-04T08:39:04Z)
• The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training [48.37150760049573]
  We identify an underlying spectral phenomenon, Stability of Singular Distribution (SoSD), where the trace-normalized singular value spectrum stabilizes early.<n>We demonstrate that synchronization between SoSD and the slow-descent regime is widely observed across diverse architectures.<n>We further interpret strategies like WSD and Muon through their ability to modulate the SoSD scale, offering a spectral lens for understanding efficient pre-training dynamics.
  arXiv  Detail & Related papers  (2026-05-26T03:03:24Z)
• Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics [92.39053980710702]
  Gradient Descent (SGD) is commonly modeled as a Langevin process, assuming that minibatch noise acts as Brownian motion.<n>This approximation relies on a continuous-time limit and a sqrt(eta) noise scaling that does not match the discrete SGD update at finite learning rate.<n>We propose an alternative formulation of SGD as deterministic dynamics in a fluctuating loss landscape induced by minibatch sampling.
  arXiv  Detail & Related papers  (2026-05-21T15:50:40Z)
• Plug-and-Play Diffusion Meets ADMM: Dual-Variable Coupling for Robust Medical Image Reconstruction [45.25461515976432]
  Plug-and-Play diffusion prior (DP) frameworks have emerged as a powerful paradigm for imaging reconstruction.<n>We present a novel approach to resolving bias-hallucination trade-off, achieving state-of-the-art gradients with significantly accelerated convergence.
  arXiv  Detail & Related papers  (2026-02-26T16:58:43Z)
• Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrieval [13.218607858857295]
  Spectral gradient methods modify gradient updates by preserving directional information while discarding scale.<n>We investigate the mechanisms underlying these gains through a dynamical analysis of a nonlinear phase retrieval model.
  arXiv  Detail & Related papers  (2026-01-30T07:12:58Z)
• The Geometry of Grokking: Norm Minimization on the Zero-Loss Manifold [5.076419064097734]
  We argue that post-memorization learning can be understood through the lens of constrained optimization.<n>We show that gradient descent effectively minimizes the weight norm on the zero-loss manifold.<n>Experiments confirm that simulating the training process using our predicted gradients reproduces both the delayed generalization and representation learning characteristic of grokking.
  arXiv  Detail & Related papers  (2025-11-02T18:44:42Z)
• Contraction and entropy production in continuous-time Sinkhorn dynamics [0.6423239719448169]
  We give an exact identity for the entropy production rate of the Sinkhorn flow, which was previously known only to be nonpositive.<n>We show that the flow induces a reversible Markov dynamics on the target marginal as an Onsager gradient flow.<n>We give for illustration two immediate practical use-cases for the Sinkhorn LSI: as a design principle for the latent space in which generative models are trained, and as a stopping algorithm for discrete-time algorithms.
  arXiv  Detail & Related papers  (2025-10-14T15:32:15Z)
• Learning What Matters: Steering Diffusion via Spectrally Anisotropic Forward Noise [43.07594740645669]
  Diffusion Probabilistic Models (DPMs) have achieved strong generative performance, yet their inductive biases remain largely implicit.<n>In this work, we aim to build inductive biases into the training and sampling of diffusion models to better accommodate the target distribution of the data to model.<n>We introduce an anisotropic noise operator that shapes these biases by replacing the isotropic forward covariance with a structured, frequency-diagonal covariance.
  arXiv  Detail & Related papers  (2025-10-07T16:08:39Z)
• Kernel-Smoothed Scores for Denoising Diffusion: A Bias-Variance Study [3.265950484493743]
  Diffusion models can be prone to memorization.<n>Regularization on the score has the same effect as increasing the size of the training dataset.<n>This perspective highlights two regularization mechanisms taking place in denoising diffusions.
  arXiv  Detail & Related papers  (2025-05-28T20:22:18Z)
• Sixth-order time-convolutionless master equation and beyond: Late-time resummations, two types of divergences, and the limits of validity [1.7620619500719317]
  We analyze the time-convolutionless (TCL) master equation, expanded to order 2n.<n>We show that while van Kampensants suppress early-time secular growth, they ultimately diverge at long times.<n>For exponentially decaying correlations, the method recovers a proper Markovian limit below a critical coupling threshold.
  arXiv  Detail & Related papers  (2024-06-16T22:15:07Z)
• Machine learning in and out of equilibrium [58.88325379746631]
  Our study uses a Fokker-Planck approach, adapted from statistical physics, to explore these parallels. We focus in particular on the stationary state of the system in the long-time limit, which in conventional SGD is out of equilibrium. We propose a new variation of Langevin dynamics (SGLD) that harnesses without replacement minibatching.
  arXiv  Detail & Related papers  (2023-06-06T09:12:49Z)
• Restoration-Degradation Beyond Linear Diffusions: A Non-Asymptotic Analysis For DDIM-Type Samplers [90.45898746733397]
  We develop a framework for non-asymptotic analysis of deterministic samplers used for diffusion generative modeling. We show that one step along the probability flow ODE can be expressed as two steps: 1) a restoration step that runs ascent on the conditional log-likelihood at some infinitesimally previous time, and 2) a degradation step that runs the forward process using noise pointing back towards the current gradient.
  arXiv  Detail & Related papers  (2023-03-06T18:59:19Z)
• Beyond the Edge of Stability via Two-step Gradient Updates [49.03389279816152]
  Gradient Descent (GD) is a powerful workhorse of modern machine learning. GD's ability to find local minimisers is only guaranteed for losses with Lipschitz gradients. This work focuses on simple, yet representative, learning problems via analysis of two-step gradient updates.
  arXiv  Detail & Related papers  (2022-06-08T21:32:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.