Correlated Noise in Epoch-Based Stochastic Gradient Descent: Implications for Weight Variances

• URL: http://arxiv.org/abs/2306.05300v2
• Date: Mon, 15 Jul 2024 12:21:02 GMT
• Title: Correlated Noise in Epoch-Based Stochastic Gradient Descent: Implications for Weight Variances
• Authors: Marcel Kühn, Bernd Rosenow,
• Abstract summary: gradient descent (SGD) has become a cornerstone of neural network optimization. We investigate the effects of epoch-based noise correlations on the stationary distribution of discrete-time SGD with momentum.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Stochastic gradient descent (SGD) has become a cornerstone of neural network optimization, yet the noise introduced by SGD is often assumed to be uncorrelated over time, despite the ubiquity of epoch-based training. In this work, we challenge this assumption and investigate the effects of epoch-based noise correlations on the stationary distribution of discrete-time SGD with momentum, limited to a quadratic loss. Our main contributions are twofold: first, we calculate the exact autocorrelation of the noise for training in epochs under the assumption that the noise is independent of small fluctuations in the weight vector, and find that SGD noise is anti-correlated in time. Second, we explore the influence of these anti-correlations on SGD dynamics. We find that for directions with a curvature greater than a hyperparameter-dependent crossover value, the results for uncorrelated noise are recovered. However, for relatively flat directions, the weight variance is significantly reduced, and our variance prediction leads to a considerable reduction in loss fluctuations as compared to the constant weight variance assumption.

Related papers

• Algorithmic Stability of Stochastic Gradient Descent with Momentum under Heavy-Tailed Noise [20.922456964393213]
  We establish generalization bounds for SGD with momentum (SGDm) under heavy-tailed noise. For quadratic loss functions, we show that SGDm admits a worse generalization bound in the presence of momentum and heavy tails. We develop a uniform-in-time discretization error bound, which to our knowledge, is the first result of its kind for SDEs with degenerate noise.
  arXiv  Detail & Related papers  (2025-02-02T19:25:48Z)
• Exact Risk Curves of signSGD in High-Dimensions: Quantifying Preconditioning and Noise-Compression Effects [6.653325043862049]
  We present an analysis of signSGD in a high dimensional limit. We quantify four effects of signSGD: effective learning rate, noise compression, diagonal preconditioning, gradient and noise reshaping. We conclude with a conjecture on how these results might be extended to Adam.
  arXiv  Detail & Related papers  (2024-11-19T00:24:50Z)
• Momentum Does Not Reduce Stochastic Noise in Stochastic Gradient Descent [0.6906005491572401]
  In neural deep networks, gradient descent (SGD) with momentum is said to converge faster and have better generalizability than SGD without momentum. In particular, adding momentum is thought to reduce this batch noise. We analyzed the effect of search direction noise, which is noise defined as the error between the search direction and the steepest descent direction.
  arXiv  Detail & Related papers  (2024-02-04T02:48:28Z)
• Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregression [70.78523583702209]
  We study training instabilities of behavior cloning with deep neural networks. We observe that minibatch SGD updates to the policy network during training result in sharp oscillations in long-horizon rewards.
  arXiv  Detail & Related papers  (2023-10-17T17:39:40Z)
• Doubly Stochastic Models: Learning with Unbiased Label Noises and Inference Stability [85.1044381834036]
  We investigate the implicit regularization effects of label noises under mini-batch sampling settings of gradient descent. We find such implicit regularizer would favor some convergence points that could stabilize model outputs against perturbation of parameters. Our work doesn't assume SGD as an Ornstein-Uhlenbeck like process and achieve a more general result with convergence of approximation proved.
  arXiv  Detail & Related papers  (2023-04-01T14:09:07Z)
• A Data-driven Loss Weighting Scheme across Heterogeneous Tasks for Image Denoising [67.02529586335473]
  In variational denoising models, weight in the data fidelity term plays the role of enhancing the noise-removal capability.<n>In this work, we propose a data-driven loss weighting scheme to address these issues.<n> Numerical results verify the remarkable performance of DLW on improving the ability of various variational denoising models to handle different complex noise.
  arXiv  Detail & Related papers  (2022-12-09T03:28:07Z)
• High-Order Qubit Dephasing at Sweet Spots by Non-Gaussian Fluctuators: Symmetry Breaking and Floquet Protection [55.41644538483948]
  We study the qubit dephasing caused by the non-Gaussian fluctuators. We predict a symmetry-breaking effect that is unique to the non-Gaussian noise.
  arXiv  Detail & Related papers  (2022-06-06T18:02:38Z)
• The effective noise of Stochastic Gradient Descent [9.645196221785694]
  Gradient Descent (SGD) is the workhorse algorithm of deep learning technology. We characterize the parameters of SGD and a recently-introduced variant, persistent SGD, in a neural network model. We find that noisier algorithms lead to wider decision boundaries of the corresponding constraint satisfaction problem.
  arXiv  Detail & Related papers  (2021-12-20T20:46:19Z)
• Analyzing and Improving the Optimization Landscape of Noise-Contrastive Estimation [50.85788484752612]
  Noise-contrastive estimation (NCE) is a statistically consistent method for learning unnormalized probabilistic models. It has been empirically observed that the choice of the noise distribution is crucial for NCE's performance. In this work, we formally pinpoint reasons for NCE's poor performance when an inappropriate noise distribution is used.
  arXiv  Detail & Related papers  (2021-10-21T16:57:45Z)
• Revisiting the Characteristics of Stochastic Gradient Noise and Dynamics [25.95229631113089]
  We show that the gradient noise possesses finite variance, and therefore the Central Limit Theorem (CLT) applies. We then demonstrate the existence of the steady-state distribution of gradient descent and approximate the distribution at a small learning rate.
  arXiv  Detail & Related papers  (2021-09-20T20:39:14Z)
• Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections [73.95786440318369]
  We focus on the so-called implicit effect' of GNIs, which is the effect of the injected noise on the dynamics of gradient descent (SGD) We show that this effect induces an asymmetric heavy-tailed noise on gradient updates. We then formally prove that GNIs induce an implicit bias', which varies depending on the heaviness of the tails and the level of asymmetry.
  arXiv  Detail & Related papers  (2021-02-13T21:28:09Z)
• Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descent [3.0079490585515343]
  gradient descent (SGD) is relatively well understood in the vanishing learning rate regime. We propose to study the basic properties of SGD and its variants in the non-vanishing learning rate regime.
  arXiv  Detail & Related papers  (2020-12-07T12:31:43Z)
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance [76.54300276636982]
  Noise in gradient descent provides a crucial implicit regularization effect for training over parameterized models. We show that parameter-dependent noise -- induced by mini-batches or label perturbation -- is far more effective than Gaussian noise. Our analysis reveals that parameter-dependent noise introduces a bias towards local minima with smaller noise variance, whereas spherical Gaussian noise does not.
  arXiv  Detail & Related papers  (2020-06-15T18:31:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.