Orthogonal Gradient Constraints Shape Noisy-Label Memorization Dynamics

• URL: http://arxiv.org/abs/2607.16231v1
• Date: Wed, 24 Jun 2026 04:34:31 GMT
• Title: Orthogonal Gradient Constraints Shape Noisy-Label Memorization Dynamics
• Abstract summary: We evaluate OrthoGrad, which removes the component of each weight gradient parallel to the current weight vector, in noisy-label image classification.<n>On MNIST with small-data regimes, OrthoGrad improves test accuracy most clearly for CNNs while reducing corrupted-label fitting.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern neural networks can fit corrupted training labels, making noisy-label learning a useful setting for studying memorization-driven overfitting. Most regularization methods modify the objective, architecture, or data distribution; here we instead study a geometric intervention on the optimizer update itself. We evaluate OrthoGrad, which removes the component of each weight gradient parallel to the current weight vector, in noisy-label image classification. On MNIST with small-data regimes, OrthoGrad improves test accuracy most clearly for CNNs while reducing corrupted-label fitting. Mechanism diagnostics based on weight norms and gradient-weight cosine similarity suggest that the projection has the strongest effect when the raw gradient contains a nontrivial radial component, and becomes weaker in larger-data regimes where gradients are already nearly orthogonal to weights. Additional CIFAR-10 ResNet-18 experiments show that the method can alter memorization trajectories but does not prevent eventual noisy-label memorization. These results support orthogonal update constraints as a useful diagnostic for studying learning dynamics, while showing that OrthoGrad is regime-dependent rather than universally regularizing.

Related papers

• Variational Rectification Inference for Learning with Noisy Labels [74.85528327499662]
  We propose variational rectification inference (VRI) to formulate the adaptive rectification for loss functions.<n>VRI is constructed as a hierarchical Bayes by treating the rectifying vector as a latent variable.<n>By introducing the variational term in VRI, the conditional posterior is estimated accurately and avoids collapsing to a Dirac delta function.
  arXiv  Detail & Related papers  (2026-03-18T01:25:08Z)
• Statistical Roughness-Informed Machine Unlearning [3.8218584696400484]
  Machine unlearning aims to remove the influence of a designated forget set from a trained model while preserving utility on the retained data.<n>We propose Statistical-Roughness Adaptive Gradient Unlearning (SRAGU), a mechanism-first unlearning algorithm that reallocates unlearning updates.
  arXiv  Detail & Related papers  (2026-02-10T00:40:36Z)
• Geometric-Disentangelment Unlearning [106.99160454669902]
  gradient ascent on forget samples often harms retained knowledge.<n>We propose the Geometric-disment Unlearning (GU) that decomposes any candidate forget gradient update into tangential and normal components to retain space and executes only the normal component.<n>Our method is plug-and-play and can be attached to existing gradient-based unlearning procedures to mitigate side effects.
  arXiv  Detail & Related papers  (2025-11-21T09:58:25Z)
• Reliable Active Learning from Unreliable Labels via Neural Collapse Geometry [5.1511135538176]
  Active Learning (AL) promises to reduce annotation cost by prioritizing informative samples, yet its reliability is undermined when labels are noisy or when the data distribution shifts.<n>We propose Active Learning via Neural Collapse Geometry (NCAL-R), a framework that leverages the emergent geometric regularities of deep networks to counteract unreliable supervision.
  arXiv  Detail & Related papers  (2025-10-10T17:50:31Z)
• Stochastic Resetting Mitigates Latent Gradient Bias of SGD from Label Noise [2.048226951354646]
  We show that resetting from a checkpoint can significantly improve generalization performance when training deep neural networks (DNNs) with noisy labels.<n>In the presence of noisy labels, DNNs initially learn the general patterns of the data but then gradually memorize the corrupted data, leading to overfitting.<n>By deconstructing the dynamics of gradient descent (SGD), we identify the behavior of a latent gradient bias induced by noisy labels, which harms generalization.
  arXiv  Detail & Related papers  (2024-06-01T10:45:41Z)
• ERASE: Error-Resilient Representation Learning on Graphs for Label Noise Tolerance [53.73316938815873]
  We propose a method called ERASE (Error-Resilient representation learning on graphs for lAbel noiSe tolerancE) to learn representations with error tolerance. ERASE combines prototype pseudo-labels with propagated denoised labels and updates representations with error resilience. Our method can outperform multiple baselines with clear margins in broad noise levels and enjoy great scalability.
  arXiv  Detail & Related papers  (2023-12-13T17:59:07Z)
• All Points Matter: Entropy-Regularized Distribution Alignment for Weakly-supervised 3D Segmentation [67.30502812804271]
  Pseudo-labels are widely employed in weakly supervised 3D segmentation tasks where only sparse ground-truth labels are available for learning. We propose a novel learning strategy to regularize the generated pseudo-labels and effectively narrow the gaps between pseudo-labels and model predictions.
  arXiv  Detail & Related papers  (2023-05-25T08:19:31Z)
• Gradient Descent in RKHS with Importance Labeling [58.79085525115987]
  We study importance labeling problem, in which we are given many unlabeled data. We propose a new importance labeling scheme that can effectively select an informative subset of unlabeled data.
  arXiv  Detail & Related papers  (2020-06-19T01:55:00Z)
• Revisiting Initialization of Neural Networks [72.24615341588846]
  We propose a rigorous estimation of the global curvature of weights across layers by approximating and controlling the norm of their Hessian matrix. Our experiments on Word2Vec and the MNIST/CIFAR image classification tasks confirm that tracking the Hessian norm is a useful diagnostic tool.
  arXiv  Detail & Related papers  (2020-04-20T18:12:56Z)
• Rectified Meta-Learning from Noisy Labels for Robust Image-based Plant Disease Diagnosis [64.82680813427054]
  Plant diseases serve as one of main threats to food security and crop production. One popular approach is to transform this problem as a leaf image classification task, which can be addressed by the powerful convolutional neural networks (CNNs) We propose a novel framework that incorporates rectified meta-learning module into common CNN paradigm to train a noise-robust deep network without using extra supervision information.
  arXiv  Detail & Related papers  (2020-03-17T09:51:30Z)
• Improving Generalization by Controlling Label-Noise Information in Neural Network Weights [33.85101318266319]
  In the presence of noisy or incorrect labels, neural networks have the undesirable tendency to memorize information about the noise. Standard regularization techniques such as dropout, weight decay or data augmentation sometimes help, but do not prevent this behavior. We show that for any training algorithm, low values of this term correspond to reduction in memorization of label-noise and better bounds.
  arXiv  Detail & Related papers  (2020-02-19T00:08:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.