Mini-batch Noise Lowers Sharpness via Dominant-Subspace Fluctuations

• URL: http://arxiv.org/abs/2607.23012v1
• Date: Sat, 25 Jul 2026 03:06:30 GMT
• Title: Mini-batch Noise Lowers Sharpness via Dominant-Subspace Fluctuations
• Abstract summary: We show how the gradient averaged over fluctuations in the dominant directions produces a sharpness correction term.<n>We derive a sharpness correction term induced by mini-batch noise in the dominant directions.<n> Experimental results show that adding the derived correction term to GD brings the sharpness evolution of GD closer to that of SGD.
• Score: 0.08594140167290099
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: During SGD training, the gradients often align strongly with the dominant subspace spanned by the top-$k$ eigenvectors of the Hessian of the loss. While this seems to naturally imply that loss reduction mainly occurs within this space, prior work has shown that updates within this dominant subspace make no meaningful progress in reducing the loss. In this work, we argue that the dominant subspace is better understood not as the main space for loss reduction, but as a key subspace for explaining the sharpness dynamics of mini-batch SGD. To explain the role of the dominant subspace in reducing top-$k$ sharpness, we show how the averaged gradient over fluctuations in the dominant directions produces a sharpness correction term, and derive a sharpness correction term induced by mini-batch noise in the dominant directions. Experimental results show that adding the derived correction term to GD brings the sharpness evolution of GD closer to that of SGD.

Related papers

• SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation [64.33799467286265]
  Low-Rank Adaptation (LoRA) enables efficient adaptation of large pre-trained models to downstream tasks.<n>We show that when a full fine-tuning gradient is backpropagated to the low-rank matrices, it undergoes anisotropic scaling driven by their singular values.<n>We propose a new low-rank parameterization, SDS-LoRA, which structurally decouples singular values from the backward pass.
  arXiv  Detail & Related papers  (2026-06-15T09:27:50Z)
• Suspicious Alignment of SGD: A Fine-Grained Step Size Condition Analysis [30.6120085647449]
  This paper explores the suspicious alignment phenomenon in gradient descent (SGD) under ill-conditioned optimization.<n>Specifically, during the initial phase of SGD updates, the alignment between the gradient and the dominant subspace tends to decrease.<n>We show that under sufficient ill-conditioning, a step size interval exists where projecting the SGD updates to the bulk space decreases the loss while projecting them to the dominant space increases the loss.
  arXiv  Detail & Related papers  (2026-01-16T21:32:48Z)
• Training Instabilities Induce Flatness Bias in Gradient Descent [6.628332915214955]
  Modern deep networks often achieve their best performance beyond a stability threshold.<n>We show that training instabilities induce an implicit bias in GD, driving parameters toward flatter regions of the loss landscape.<n>We also show that restoring instabilities in Adam further improves generalization.
  arXiv  Detail & Related papers  (2025-11-16T11:26:25Z)
• Accelerating Neural Network Training Along Sharp and Flat Directions [6.576051895863941]
  We study Bulk-SGD, a variant of SGD that restricts updates to the complement of the Dominant subspace.<n>We show that updates along the Bulk subspace, corresponding to flatter directions in the loss landscape, can accelerate convergence but may compromise stability.<n>Our findings suggest a principled approach to designing curvature-awares.
  arXiv  Detail & Related papers  (2025-05-17T12:13:05Z)
• Does SGD really happen in tiny subspaces? [18.283839252425803]
  Recent studies have revealed that, along the training trajectory, the gradient aligns with a low-rank top eigenspace of the training loss Hessian.<n>This paper explores whether neural networks can be trained within the dominant subspace, which could lead to more efficient training methods.
  arXiv  Detail & Related papers  (2024-05-25T01:44:35Z)
• Risk Bounds of Accelerated SGD for Overparameterized Linear Regression [75.27846230182885]
  Accelerated gradient descent (ASGD) is a workhorse in deep learning. Existing optimization theory can only explain the faster convergence of ASGD, but cannot explain its better generalization.
  arXiv  Detail & Related papers  (2023-11-23T23:02:10Z)
• Inference and Interference: The Role of Clipping, Pruning and Loss Landscapes in Differentially Private Stochastic Gradient Descent [13.27004430044574]
  Differentially private gradient descent (DP-SGD) is known to have poorer training and test performance on large neural networks. We compare the behavior of the two processes separately in early and late epochs. We find that while DP-SGD makes slower progress in early stages, it is the behavior in the later stages that determines the end result.
  arXiv  Detail & Related papers  (2023-11-12T13:31:35Z)
• Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregression [70.78523583702209]
  We study training instabilities of behavior cloning with deep neural networks. We observe that minibatch SGD updates to the policy network during training result in sharp oscillations in long-horizon rewards.
  arXiv  Detail & Related papers  (2023-10-17T17:39:40Z)
• Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stability [69.01076284478151]
  In machine learning optimization, gradient descent (GD) often operates at the edge of stability (EoS) This paper studies the convergence and implicit bias of constant-stepsize GD for logistic regression on linearly separable data in the EoS regime.
  arXiv  Detail & Related papers  (2023-05-19T16:24:47Z)
• Direction Matters: On the Implicit Bias of Stochastic Gradient Descent with Moderate Learning Rate [105.62979485062756]
  This paper attempts to characterize the particular regularization effect of SGD in the moderate learning rate regime. We show that SGD converges along the large eigenvalue directions of the data matrix, while GD goes after the small eigenvalue directions.
  arXiv  Detail & Related papers  (2020-11-04T21:07:52Z)
• The Break-Even Point on Optimization Trajectories of Deep Neural Networks [64.7563588124004]
  We argue for the existence of the "break-even" point on this trajectory. We show that using a large learning rate in the initial phase of training reduces the variance of the gradient. We also show that using a low learning rate results in bad conditioning of the loss surface even for a neural network with batch normalization layers.
  arXiv  Detail & Related papers  (2020-02-21T22:55:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.