Regularization in neural network optimization via trimmed stochastic gradient descent with noisy label

• URL: http://arxiv.org/abs/2012.11073v1
• Date: Mon, 21 Dec 2020 01:31:53 GMT
• Title: Regularization in neural network optimization via trimmed stochastic gradient descent with noisy label
• Authors: Kensuke Nakamura and Byung-Woo Hong
• Abstract summary: Regularization is essential for avoiding over-fitting to training data in neural network optimization. We propose a first-order optimization method (Label-Noised Trim-SGD) which combines the label noise with the example trimming. The proposed algorithm enables us to impose a large label noise and obtain a better regularization effect than the original methods.
• Score: 2.66512000865131
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Regularization is essential for avoiding over-fitting to training data in neural network optimization, leading to better generalization of the trained networks. The label noise provides a strong implicit regularization by replacing the target ground truth labels of training examples by uniform random labels. However, it may also cause undesirable misleading gradients due to the large loss associated with incorrect labels. We propose a first-order optimization method (Label-Noised Trim-SGD) which combines the label noise with the example trimming in order to remove the outliers. The proposed algorithm enables us to impose a large label noise and obtain a better regularization effect than the original methods. The quantitative analysis is performed by comparing the behavior of the label noise, the example trimming, and the proposed algorithm. We also present empirical results that demonstrate the effectiveness of our algorithm using the major benchmarks and the fundamental networks, where our method has successfully outperformed the state-of-the-art optimization methods.

Related papers

• Inaccurate Label Distribution Learning with Dependency Noise [52.08553913094809]
  We introduce the Dependent Noise-based Inaccurate Label Distribution Learning (DN-ILDL) framework to tackle the challenges posed by noise in label distribution learning. We show that DN-ILDL effectively addresses the ILDL problem and outperforms existing LDL methods.
  arXiv  Detail & Related papers  (2024-05-26T07:58:07Z)
• Extracting Clean and Balanced Subset for Noisy Long-tailed Classification [66.47809135771698]
  We develop a novel pseudo labeling method using class prototypes from the perspective of distribution matching. By setting a manually-specific probability measure, we can reduce the side-effects of noisy and long-tailed data simultaneously. Our method can extract this class-balanced subset with clean labels, which brings effective performance gains for long-tailed classification with label noise.
  arXiv  Detail & Related papers  (2024-04-10T07:34:37Z)
• ERASE: Error-Resilient Representation Learning on Graphs for Label Noise Tolerance [53.73316938815873]
  We propose a method called ERASE (Error-Resilient representation learning on graphs for lAbel noiSe tolerancE) to learn representations with error tolerance. ERASE combines prototype pseudo-labels with propagated denoised labels and updates representations with error resilience. Our method can outperform multiple baselines with clear margins in broad noise levels and enjoy great scalability.
  arXiv  Detail & Related papers  (2023-12-13T17:59:07Z)
• Robust Training under Label Noise by Over-parameterization [41.03008228953627]
  We propose a principled approach for robust training of over-parameterized deep networks in classification tasks where a proportion of training labels are corrupted. The main idea is yet very simple: label noise is sparse and incoherent with the network learned from clean data, so we model the noise and learn to separate it from the data. Remarkably, when trained using such a simple method in practice, we demonstrate state-of-the-art test accuracy against label noise on a variety of real datasets.
  arXiv  Detail & Related papers  (2022-02-28T18:50:10Z)
• Interpolation-based Contrastive Learning for Few-Label Semi-Supervised Learning [43.51182049644767]
  Semi-supervised learning (SSL) has long been proved to be an effective technique to construct powerful models with limited labels. Regularization-based methods which force the perturbed samples to have similar predictions with the original ones have attracted much attention. We propose a novel contrastive loss to guide the embedding of the learned network to change linearly between samples.
  arXiv  Detail & Related papers  (2022-02-24T06:00:05Z)
• Optimizing Diffusion Rate and Label Reliability in a Graph-Based Semi-supervised Classifier [2.4366811507669124]
  The Local and Global Consistency (LGC) algorithm is one of the most well-known graph-based semi-supervised (GSSL) classifiers. We discuss how removing the self-influence of a labeled instance may be beneficial, and how it relates to leave-one-out error. Within this framework, we propose methods to estimate label reliability and diffusion rate.
  arXiv  Detail & Related papers  (2022-01-10T16:58:52Z)
• Dash: Semi-Supervised Learning with Dynamic Thresholding [72.74339790209531]
  We propose a semi-supervised learning (SSL) approach that uses unlabeled examples to train models. Our proposed approach, Dash, enjoys its adaptivity in terms of unlabeled data selection.
  arXiv  Detail & Related papers  (2021-09-01T23:52:29Z)
• Tackling Instance-Dependent Label Noise via a Universal Probabilistic Model [80.91927573604438]
  This paper proposes a simple yet universal probabilistic model, which explicitly relates noisy labels to their instances. Experiments on datasets with both synthetic and real-world label noise verify that the proposed method yields significant improvements on robustness.
  arXiv  Detail & Related papers  (2021-01-14T05:43:51Z)
• Semi-Supervised Learning with Meta-Gradient [123.26748223837802]
  We propose a simple yet effective meta-learning algorithm in semi-supervised learning. We find that the proposed algorithm performs favorably against state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-07-08T08:48:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.