Noisy Concurrent Training for Efficient Learning under Label Noise

• URL: http://arxiv.org/abs/2009.08325v1
• Date: Thu, 17 Sep 2020 14:22:17 GMT
• Title: Noisy Concurrent Training for Efficient Learning under Label Noise
• Authors: Fahad Sarfraz, Elahe Arani and Bahram Zonooz
• Abstract summary: Deep neural networks (DNNs) fail to learn effectively under label noise and have been shown to memorize random labels which affect their performance. We consider learning in isolation, using one-hot encoded labels as the sole source of supervision, and a lack of regularization to discourage memorization as the major shortcomings of the standard training procedure. We propose Noisy Concurrent Training (NCT) which leverages collaborative learning to use the consensus between two models as an additional source of supervision.
• Score: 13.041607703862724
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural networks (DNNs) fail to learn effectively under label noise and have been shown to memorize random labels which affect their generalization performance. We consider learning in isolation, using one-hot encoded labels as the sole source of supervision, and a lack of regularization to discourage memorization as the major shortcomings of the standard training procedure. Thus, we propose Noisy Concurrent Training (NCT) which leverages collaborative learning to use the consensus between two models as an additional source of supervision. Furthermore, inspired by trial-to-trial variability in the brain, we propose a counter-intuitive regularization technique, target variability, which entails randomly changing the labels of a percentage of training samples in each batch as a deterrent to memorization and over-generalization in DNNs. Target variability is applied independently to each model to keep them diverged and avoid the confirmation bias. As DNNs tend to prioritize learning simple patterns first before memorizing the noisy labels, we employ a dynamic learning scheme whereby as the training progresses, the two models increasingly rely more on their consensus. NCT also progressively increases the target variability to avoid memorization in later stages. We demonstrate the effectiveness of our approach on both synthetic and real-world noisy benchmark datasets.

Related papers

• Soften to Defend: Towards Adversarial Robustness via Self-Guided Label Refinement [5.865750284677784]
  Adversarial training (AT) is one of the most effective ways to obtain the robustness of deep neural networks against adversarial attacks. AT methods suffer from robust overfitting, i.e., a significant generalization gap between the training and testing curves. We propose a label refinement approach for AT, which self-refines a more accurate and informative label distribution from over-confident hard labels.
  arXiv  Detail & Related papers  (2024-03-14T04:48:31Z)
• Rethinking Classifier Re-Training in Long-Tailed Recognition: A Simple Logits Retargeting Approach [102.0769560460338]
  We develop a simple logits approach (LORT) without the requirement of prior knowledge of the number of samples per class. Our method achieves state-of-the-art performance on various imbalanced datasets, including CIFAR100-LT, ImageNet-LT, and iNaturalist 2018.
  arXiv  Detail & Related papers  (2024-03-01T03:27:08Z)
• Memory Consistency Guided Divide-and-Conquer Learning for Generalized Category Discovery [56.172872410834664]
  Generalized category discovery (GCD) aims at addressing a more realistic and challenging setting of semi-supervised learning. We propose a Memory Consistency guided Divide-and-conquer Learning framework (MCDL) Our method outperforms state-of-the-art models by a large margin on both seen and unseen classes of the generic image recognition.
  arXiv  Detail & Related papers  (2024-01-24T09:39:45Z)
• One-bit Supervision for Image Classification: Problem, Solution, and Beyond [114.95815360508395]
  This paper presents one-bit supervision, a novel setting of learning with fewer labels, for image classification. We propose a multi-stage training paradigm and incorporate negative label suppression into an off-the-shelf semi-supervised learning algorithm. In multiple benchmarks, the learning efficiency of the proposed approach surpasses that using full-bit, semi-supervised supervision.
  arXiv  Detail & Related papers  (2023-11-26T07:39:00Z)
• MILD: Modeling the Instance Learning Dynamics for Learning with Noisy Labels [19.650299232829546]
  We propose an iterative selection approach based on the Weibull mixture model to identify clean data. In particular, we measure the difficulty of memorization and memorize for each instance via the transition times between being misclassified and being memorized. Our strategy outperforms existing noisy-label learning methods.
  arXiv  Detail & Related papers  (2023-06-20T14:26:53Z)
• CNTN: Cyclic Noise-tolerant Network for Gait Recognition [12.571029673961315]
  Gait recognition aims to identify individuals by recognizing their walking patterns. Most of the previous gait recognition methods degenerate significantly due to two memorization effects, namely appearance memorization and label noise memorization. For the first time noisy gait recognition is studied, and a cyclic noise-tolerant network (CNTN) is proposed with a cyclic training algorithm.
  arXiv  Detail & Related papers  (2022-10-13T11:23:58Z)
• Improving Distantly-Supervised Named Entity Recognition with Self-Collaborative Denoising Learning [9.747173655999427]
  We propose a robust learning paradigm named Self-Collaborative Denoising Learning (SCDL) SCDL jointly trains two teacher-student networks in a mutually-beneficial manner to iteratively perform noisy label refinery. Extensive experimental results on five real-world datasets demonstrate that SCDL is superior to state-of-the-art DS-NER denoising methods.
  arXiv  Detail & Related papers  (2021-10-09T01:45:03Z)
• Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training [66.80558875393565]
  We study the problem of training named entity recognition (NER) models using only distantly-labeled data. We propose a noise-robust learning scheme comprised of a new loss function and a noisy label removal step. Our method achieves superior performance, outperforming existing distantly-supervised NER models by significant margins.
  arXiv  Detail & Related papers  (2021-09-10T17:19:56Z)
• Two-phase Pseudo Label Densification for Self-training based Domain Adaptation [93.03265290594278]
  We propose a novel Two-phase Pseudo Label Densification framework, referred to as TPLD. In the first phase, we use sliding window voting to propagate the confident predictions, utilizing intrinsic spatial-correlations in the images. In the second phase, we perform a confidence-based easy-hard classification. To ease the training process and avoid noisy predictions, we introduce the bootstrapping mechanism to the original self-training loss.
  arXiv  Detail & Related papers  (2020-12-09T02:35:25Z)
• Early-Learning Regularization Prevents Memorization of Noisy Labels [29.04549895470588]
  We propose a novel framework to perform classification via deep learning in the presence of noisy annotations. Deep neural networks have been observed to first fit the training data with clean labels during an "early learning" phase. We design a regularization term that steers the model towards these targets, implicitly preventing memorization of the false labels.
  arXiv  Detail & Related papers  (2020-06-30T23:46:33Z)
• DMT: Dynamic Mutual Training for Semi-Supervised Learning [69.17919491907296]
  Self-training methods usually rely on single model prediction confidence to filter low-confidence pseudo labels. We propose mutual training between two different models by a dynamically re-weighted loss function, called Dynamic Mutual Training. Our experiments show that DMT achieves state-of-the-art performance in both image classification and semantic segmentation.
  arXiv  Detail & Related papers  (2020-04-18T03:12:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.