Exponentiated Gradient Reweighting for Robust Training Under Label Noise and Beyond

• URL: http://arxiv.org/abs/2104.01493v1
• Date: Sat, 3 Apr 2021 22:54:49 GMT
• Title: Exponentiated Gradient Reweighting for Robust Training Under Label Noise and Beyond
• Authors: Negin Majidi, Ehsan Amid, Hossein Talebi, and Manfred K. Warmuth
• Abstract summary: We present a flexible approach to learning from noisy examples. Specifically, we treat each training example as an expert and maintain a distribution over all examples. Unlike other related methods, our approach handles a general class of loss functions and can be applied to a wide range of noise types and applications.
• Score: 21.594200327544968
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many learning tasks in machine learning can be viewed as taking a gradient step towards minimizing the average loss of a batch of examples in each training iteration. When noise is prevalent in the data, this uniform treatment of examples can lead to overfitting to noisy examples with larger loss values and result in poor generalization. Inspired by the expert setting in on-line learning, we present a flexible approach to learning from noisy examples. Specifically, we treat each training example as an expert and maintain a distribution over all examples. We alternate between updating the parameters of the model using gradient descent and updating the example weights using the exponentiated gradient update. Unlike other related methods, our approach handles a general class of loss functions and can be applied to a wide range of noise types and applications. We show the efficacy of our approach for multiple learning settings, namely noisy principal component analysis and a variety of noisy classification problems.

Related papers

• Learning with Noisy Foundation Models [95.50968225050012]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets. We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• Multiclass Learning from Noisy Labels for Non-decomposable Performance Measures [15.358504449550013]
  We design algorithms to learn from noisy labels for two broad classes of non-decomposable performance measures. In both cases, we develop noise-corrected versions of the algorithms under the widely studied class-conditional noise models. Our experiments demonstrate the effectiveness of our algorithms in handling label noise.
  arXiv  Detail & Related papers  (2024-02-01T23:03:53Z)
• Late Stopping: Avoiding Confidently Learning from Mislabeled Examples [61.00103151680946]
  We propose a new framework, Late Stopping, which leverages the intrinsic robust learning ability of DNNs through a prolonged training process. We empirically observe that mislabeled and clean examples exhibit differences in the number of epochs required for them to be consistently and correctly classified. Experimental results on benchmark-simulated and real-world noisy datasets demonstrate that the proposed method outperforms state-of-the-art counterparts.
  arXiv  Detail & Related papers  (2023-08-26T12:43:25Z)
• Intra-class Adaptive Augmentation with Neighbor Correction for Deep Metric Learning [99.14132861655223]
  We propose a novel intra-class adaptive augmentation (IAA) framework for deep metric learning. We reasonably estimate intra-class variations for every class and generate adaptive synthetic samples to support hard samples mining. Our method significantly improves and outperforms the state-of-the-art methods on retrieval performances by 3%-6%.
  arXiv  Detail & Related papers  (2022-11-29T14:52:38Z)
• The Dynamic of Consensus in Deep Networks and the Identification of Noisy Labels [5.28539620288341]
  noisy labels can't be distinguished from clean examples by the end of training. Recent research has dealt with this challenge by utilizing the fact that deep networks seem to memorize examples much earlier than noisy examples. We use this observation to develop a new method for noisy labels filtration.
  arXiv  Detail & Related papers  (2022-10-02T17:47:23Z)
• CMW-Net: Learning a Class-Aware Sample Weighting Mapping for Robust Deep Learning [55.733193075728096]
  Modern deep neural networks can easily overfit to biased training data containing corrupted labels or class imbalance. Sample re-weighting methods are popularly used to alleviate this data bias issue. We propose a meta-model capable of adaptively learning an explicit weighting scheme directly from data.
  arXiv  Detail & Related papers  (2022-02-11T13:49:51Z)
• Learning with Neighbor Consistency for Noisy Labels [69.83857578836769]
  We present a method for learning from noisy labels that leverages similarities between training examples in feature space. We evaluate our method on datasets evaluating both synthetic (CIFAR-10, CIFAR-100) and realistic (mini-WebVision, Clothing1M, mini-ImageNet-Red) noise.
  arXiv  Detail & Related papers  (2022-02-04T15:46:27Z)
• Perlin Noise Improve Adversarial Robustness [9.084544535198509]
  Adversarial examples are some special input that can perturb the output of a deep neural network. Most of the present methods for generating adversarial examples require gradient information. Procedural noise adversarial examples is a new way of adversarial example generation.
  arXiv  Detail & Related papers  (2021-12-26T15:58:28Z)
• When does loss-based prioritization fail? [18.982933391138268]
  We show that loss-based acceleration methods degrade in scenarios with noisy and corrupted data. Measures of example difficulty need to correctly separate out noise from other types of challenging examples.
  arXiv  Detail & Related papers  (2021-07-16T07:23:15Z)
• Graph Learning with Loss-Guided Training [16.815638149823744]
  We explore em loss-guided training in a new domain of node embedding methods pioneered by sc DeepWalk. Our empirical evaluation on a rich collection of datasets shows significant acceleration over the baseline static methods, both in terms of total training performed and overall computation.
  arXiv  Detail & Related papers  (2020-05-31T08:03:06Z)
• Robust and On-the-fly Dataset Denoising for Image Classification [72.10311040730815]
  On-the-fly Data Denoising (ODD) is robust to mislabeled examples, while introducing almost zero computational overhead compared to standard training. ODD is able to achieve state-of-the-art results on a wide range of datasets including real-world ones such as WebVision and Clothing1M.
  arXiv  Detail & Related papers  (2020-03-24T03:59:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.