Learning from Data with Noisy Labels Using Temporal Self-Ensemble

• URL: http://arxiv.org/abs/2207.10354v1
• Date: Thu, 21 Jul 2022 08:16:31 GMT
• Title: Learning from Data with Noisy Labels Using Temporal Self-Ensemble
• Authors: Jun Ho Lee, Jae Soon Baik, Tae Hwan Hwang, and Jun Won Choi
• Abstract summary: Deep neural networks (DNNs) have an enormous capacity to memorize noisy labels. Current state-of-the-art methods present a co-training scheme that trains dual networks using samples associated with small losses. We propose a simple yet effective robust training scheme that operates by training only a single network.
• Score: 11.245833546360386
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: There are inevitably many mislabeled data in real-world datasets. Because deep neural networks (DNNs) have an enormous capacity to memorize noisy labels, a robust training scheme is required to prevent labeling errors from degrading the generalization performance of DNNs. Current state-of-the-art methods present a co-training scheme that trains dual networks using samples associated with small losses. In practice, however, training two networks simultaneously can burden computing resources. In this study, we propose a simple yet effective robust training scheme that operates by training only a single network. During training, the proposed method generates temporal self-ensemble by sampling intermediate network parameters from the weight trajectory formed by stochastic gradient descent optimization. The loss sum evaluated with these self-ensembles is used to identify incorrectly labeled samples. In parallel, our method generates multi-view predictions by transforming an input data into various forms and considers their agreement to identify incorrectly labeled samples. By combining the aforementioned metrics, we present the proposed {\it self-ensemble-based robust training} (SRT) method, which can filter the samples with noisy labels to reduce their influence on training. Experiments on widely-used public datasets demonstrate that the proposed method achieves a state-of-the-art performance in some categories without training the dual networks.

Related papers

• Rethinking Classifier Re-Training in Long-Tailed Recognition: A Simple Logits Retargeting Approach [102.0769560460338]
  We develop a simple logits approach (LORT) without the requirement of prior knowledge of the number of samples per class. Our method achieves state-of-the-art performance on various imbalanced datasets, including CIFAR100-LT, ImageNet-LT, and iNaturalist 2018.
  arXiv  Detail & Related papers  (2024-03-01T03:27:08Z)
• Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
  Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice. We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
  arXiv  Detail & Related papers  (2023-12-27T09:03:43Z)
• An unfolding method based on conditional Invertible Neural Networks (cINN) using iterative training [0.0]
  Generative networks like invertible neural networks(INN) enable a probabilistic unfolding. We introduce the iterative conditional INN(IcINN) for unfolding that adjusts for deviations between simulated training samples and data.
  arXiv  Detail & Related papers  (2022-12-16T19:00:05Z)
• CrossSplit: Mitigating Label Noise Memorization through Data Splitting [25.344386272010397]
  We propose a novel training procedure to mitigate the memorization of noisy labels, called CrossSplit. Experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet and mini-WebVision datasets demonstrate that our method can outperform the current state-of-the-art in a wide range of noise ratios.
  arXiv  Detail & Related papers  (2022-12-03T19:09:56Z)
• Distributed Adversarial Training to Robustify Deep Neural Networks at Scale [100.19539096465101]
  Current deep neural networks (DNNs) are vulnerable to adversarial attacks, where adversarial perturbations to the inputs can change or manipulate classification. To defend against such attacks, an effective approach, known as adversarial training (AT), has been shown to mitigate robust training. We propose a large-batch adversarial training framework implemented over multiple machines.
  arXiv  Detail & Related papers  (2022-06-13T15:39:43Z)
• Robust Training under Label Noise by Over-parameterization [41.03008228953627]
  We propose a principled approach for robust training of over-parameterized deep networks in classification tasks where a proportion of training labels are corrupted. The main idea is yet very simple: label noise is sparse and incoherent with the network learned from clean data, so we model the noise and learn to separate it from the data. Remarkably, when trained using such a simple method in practice, we demonstrate state-of-the-art test accuracy against label noise on a variety of real datasets.
  arXiv  Detail & Related papers  (2022-02-28T18:50:10Z)
• Synergistic Network Learning and Label Correction for Noise-robust Image Classification [28.27739181560233]
  Deep Neural Networks (DNNs) tend to overfit training label noise, resulting in poorer model performance in practice. We propose a robust label correction framework combining the ideas of small loss selection and noise correction. We demonstrate our method on both synthetic and real-world datasets with different noise types and rates.
  arXiv  Detail & Related papers  (2022-02-27T23:06:31Z)
• Delving into Sample Loss Curve to Embrace Noisy and Imbalanced Data [17.7825114228313]
  Corrupted labels and class imbalance are commonly encountered in practically collected training data. Existing approaches alleviate these issues by adopting a sample re-weighting strategy. However, biased samples with corrupted labels and of tailed classes commonly co-exist in training data.
  arXiv  Detail & Related papers  (2021-12-30T09:20:07Z)
• Dash: Semi-Supervised Learning with Dynamic Thresholding [72.74339790209531]
  We propose a semi-supervised learning (SSL) approach that uses unlabeled examples to train models. Our proposed approach, Dash, enjoys its adaptivity in terms of unlabeled data selection.
  arXiv  Detail & Related papers  (2021-09-01T23:52:29Z)
• FreeTickets: Accurate, Robust and Efficient Deep Ensemble by Training with Dynamic Sparsity [74.58777701536668]
  We introduce the FreeTickets concept, which can boost the performance of sparse convolutional neural networks over their dense network equivalents by a large margin. We propose two novel efficient ensemble methods with dynamic sparsity, which yield in one shot many diverse and accurate tickets "for free" during the sparse training process.
  arXiv  Detail & Related papers  (2021-06-28T10:48:20Z)
• DivideMix: Learning with Noisy Labels as Semi-supervised Learning [111.03364864022261]
  We propose DivideMix, a framework for learning with noisy labels. Experiments on multiple benchmark datasets demonstrate substantial improvements over state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-02-18T06:20:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.