Gradient Perturbation: Learning to Perturb Gradients for Adaptive Training

• URL: http://arxiv.org/abs/2605.29494v1
• Date: Thu, 28 May 2026 07:19:36 GMT
• Title: Gradient Perturbation: Learning to Perturb Gradients for Adaptive Training
• Abstract summary: gradient perturbation has received little systematic investigation.<n>We propose Learning to Perturb Gradients (LPG), which adaptively perturbs logit-level gradients at the class level to achieve category-aware training.<n> Experiments on balanced classification, long-tail classification, and noisy label learning demonstrate LPG consistently outperforms existing methods and can be combined with them as a plug-in module.
• Score: 3.2608238154676843
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural network training involves both forward propagation (from features through logits to loss) and backward propagation (from loss through gradients to parameter updates). While perturbations along the forward chain, including feature perturbation, logit perturbation, and label perturbation, have been extensively studied, the backward chain's gradient perturbation has received little systematic investigation. In this paper, we establish a unified framework for gradient perturbation, revealing that existing methods such as Sharpness-Aware Minimization (SAM), gradient clipping, and gradient noise injection can all be interpreted as imposing specific forms of gradient perturbation. Analogous to the recently proposed Logit Perturbation Learning (LPL), we conjecture that amplifying the gradient norm for a class acts as positive augmentation (enhancing learning), while dampening it acts as negative augmentation (suppressing overfitting). Based on these observations, we propose Learning to Perturb Gradients (LPG), which adaptively perturbs logit-level gradients at the class level to achieve category-aware training. We also establish theoretical connections between gradient perturbation bounds and generalization guarantees via PAC-Bayesian analysis. Experiments on balanced classification, long-tail classification, and noisy label learning demonstrate that LPG consistently outperforms existing methods and can be combined with them as a plug-in module.

Related papers

• Focus, Align, and Sustain: Counteracting Gradient Dilution in Incremental Object Detection [34.23489857539403]
  We propose a unified framework thatFocuses, Aligns, and Sustains gradient flow throughout incremental learning.<n>Experiments show that FAS restores robust optimization dynamics and outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-06-13T11:20:16Z)
• SURGE: Surrogate Gradient Adaptation in Binary Neural Networks [41.424349870612716]
  We propose SURrogate GradiEnt Adaptation (SURGE), a learnable gradient compensation framework with theoretical grounding.<n> SURGE mitigates gradient mismatch through auxiliary backpropagation.<n> Experiments on image classification, object detection, and language understanding tasks demonstrate that SURGE performs best over state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-09T09:52:38Z)
• The Devil Is in Gradient Entanglement: Energy-Aware Gradient Coordinator for Robust Generalized Category Discovery [76.35318747478522]
  Energy-Aware Gradient Coordinator (EAGC) is a plug-and-play gradient module that explicitly regulates the optimization process.<n>EAGC comprises two components: Anchor-based Gradient Alignment (AGA) and Energy-aware Elastic Projection (EEP)
  arXiv  Detail & Related papers  (2026-03-25T14:58:42Z)
• Gradient Normalization Provably Benefits Nonconvex SGD under Heavy-Tailed Noise [60.92029979853314]
  We investigate the roles of gradient normalization and clipping in ensuring the convergence of Gradient Descent (SGD) under heavy-tailed noise. Our work provides the first theoretical evidence demonstrating the benefits of gradient normalization in SGD under heavy-tailed noise. We introduce an accelerated SGD variant incorporating gradient normalization and clipping, further enhancing convergence rates under heavy-tailed noise.
  arXiv  Detail & Related papers  (2024-10-21T22:40:42Z)
• Mjolnir: Breaking the Shield of Perturbation-Protected Gradients via Adaptive Diffusion [13.764770382623812]
  We present the first attempt to break the shield of gradient perturbation protection in Federated Learning.<n>We introduce Mjolnir, a perturbation-resilient gradient leakage attack.<n>Mjolnir is capable of removing perturbations from gradients without requiring additional access to the original model structure or external data.
  arXiv  Detail & Related papers  (2024-07-07T07:06:49Z)
• Expected Grad-CAM: Towards gradient faithfulness [7.2203673761998495]
  gradient-weighted CAM approaches still rely on vanilla gradients. Our work proposes a gradient-weighted CAM augmentation that tackles the saturation and sensitivity problem.
  arXiv  Detail & Related papers  (2024-06-03T12:40:30Z)
• A Learning Paradigm for Interpretable Gradients [9.074325843851726]
  We present a novel training approach to improve the quality of gradients for interpretability. We find that the resulting gradient is qualitatively less noisy and improves quantitatively the interpretability properties of different networks.
  arXiv  Detail & Related papers  (2024-04-23T13:32:29Z)
• The Equalization Losses: Gradient-Driven Training for Long-tailed Object Recognition [84.51875325962061]
  We propose a gradient-driven training mechanism to tackle the long-tail problem. We introduce a new family of gradient-driven loss functions, namely equalization losses. Our method consistently outperforms the baseline models.
  arXiv  Detail & Related papers  (2022-10-11T16:00:36Z)
• A Study of Gradient Variance in Deep Learning [56.437755740715396]
  We introduce a method, Gradient Clustering, to minimize the variance of average mini-batch gradient with stratified sampling. We measure the gradient variance on common deep learning benchmarks and observe that, contrary to common assumptions, gradient variance increases during training.
  arXiv  Detail & Related papers  (2020-07-09T03:23:10Z)
• Unbiased Risk Estimators Can Mislead: A Case Study of Learning with Complementary Labels [92.98756432746482]
  We study a weakly supervised problem called learning with complementary labels. We show that the quality of gradient estimation matters more in risk minimization. We propose a novel surrogate complementary loss(SCL) framework that trades zero bias with reduced variance.
  arXiv  Detail & Related papers  (2020-07-05T04:19:37Z)
• The Break-Even Point on Optimization Trajectories of Deep Neural Networks [64.7563588124004]
  We argue for the existence of the "break-even" point on this trajectory. We show that using a large learning rate in the initial phase of training reduces the variance of the gradient. We also show that using a low learning rate results in bad conditioning of the loss surface even for a neural network with batch normalization layers.
  arXiv  Detail & Related papers  (2020-02-21T22:55:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.