Meta Self-Refinement for Robust Learning with Weak Supervision

• URL: http://arxiv.org/abs/2205.07290v2
• Date: Sun, 30 Apr 2023 13:43:19 GMT
• Title: Meta Self-Refinement for Robust Learning with Weak Supervision
• Authors: Dawei Zhu, Xiaoyu Shen, Michael A. Hedderich, Dietrich Klakow
• Abstract summary: We propose Meta Self-Refinement (MSR) to combat label noise from weak supervision. MSR is robust against label noise in all settings and outperforms state-of-the-art methods by up to 11.4% in accuracy and 9.26% in F1 score.
• Score: 29.80743717767389
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training deep neural networks (DNNs) under weak supervision has attracted increasing research attention as it can significantly reduce the annotation cost. However, labels from weak supervision can be noisy, and the high capacity of DNNs enables them to easily overfit the label noise, resulting in poor generalization. Recent methods leverage self-training to build noise-resistant models, in which a teacher trained under weak supervision is used to provide highly confident labels for teaching the students. Nevertheless, the teacher derived from such frameworks may have fitted a substantial amount of noise and therefore produce incorrect pseudo-labels with high confidence, leading to severe error propagation. In this work, we propose Meta Self-Refinement (MSR), a noise-resistant learning framework, to effectively combat label noise from weak supervision. Instead of relying on a fixed teacher trained with noisy labels, we encourage the teacher to refine its pseudo-labels. At each training step, MSR performs a meta gradient descent on the current mini-batch to maximize the student performance on a clean validation set. Extensive experimentation on eight NLP benchmarks demonstrates that MSR is robust against label noise in all settings and outperforms state-of-the-art methods by up to 11.4% in accuracy and 9.26% in F1 score.

Related papers

• Label Noise-Resistant Mean Teaching for Weakly Supervised Fake News Detection [93.6222609806278]
  We propose a novel label noise-resistant mean teaching approach (LNMT) for weakly supervised fake news detection. LNMT leverages unlabeled news and feedback comments of users to enlarge the amount of training data. LNMT establishes a mean teacher framework equipped with label propagation and label reliability estimation.
  arXiv  Detail & Related papers  (2022-06-10T16:01:58Z)
• Reliable Label Correction is a Good Booster When Learning with Extremely Noisy Labels [65.79898033530408]
  We introduce a novel framework, termed as LC-Booster, to explicitly tackle learning under extreme noise. LC-Booster incorporates label correction into the sample selection, so that more purified samples, through the reliable label correction, can be utilized for training. Experiments show that LC-Booster advances state-of-the-art results on several noisy-label benchmarks.
  arXiv  Detail & Related papers  (2022-04-30T07:19:03Z)
• Investigating Why Contrastive Learning Benefits Robustness Against Label Noise [6.855361451300868]
  Self-supervised contrastive learning has been shown to be very effective in preventing deep networks from overfitting noisy labels. We rigorously prove that the representation matrix learned by contrastive learning boosts robustness.
  arXiv  Detail & Related papers  (2022-01-29T05:19:26Z)
• Prototypical Classifier for Robust Class-Imbalanced Learning [64.96088324684683]
  We propose textitPrototypical, which does not require fitting additional parameters given the embedding network. Prototypical produces balanced and comparable predictions for all classes even though the training set is class-imbalanced. We test our method on CIFAR-10LT, CIFAR-100LT and Webvision datasets, observing that Prototypical obtains substaintial improvements compared with state of the arts.
  arXiv  Detail & Related papers  (2021-10-22T01:55:01Z)
• Training Classifiers that are Universally Robust to All Label Noise Levels [91.13870793906968]
  Deep neural networks are prone to overfitting in the presence of label noise. We propose a distillation-based framework that incorporates a new subcategory of Positive-Unlabeled learning. Our framework generally outperforms at medium to high noise levels.
  arXiv  Detail & Related papers  (2021-05-27T13:49:31Z)
• Boosting Semi-Supervised Face Recognition with Noise Robustness [54.342992887966616]
  This paper presents an effective solution to semi-supervised face recognition that is robust to the label noise aroused by the auto-labelling. We develop a semi-supervised face recognition solution, named Noise Robust Learning-Labelling (NRoLL), which is based on the robust training ability empowered by GN.
  arXiv  Detail & Related papers  (2021-05-10T14:43:11Z)
• Contrastive Learning Improves Model Robustness Under Label Noise [3.756550107432323]
  We show that by initializing supervised robust methods using representations learned through contrastive learning leads to significantly improved performance under label noise. Even the simplest method can outperform the state-of-the-art SSL method by more than 50% under high label noise when with contrastive learning.
  arXiv  Detail & Related papers  (2021-04-19T00:27:58Z)
• LongReMix: Robust Learning with High Confidence Samples in a Noisy Label Environment [33.376639002442914]
  We propose the new 2-stage noisy-label training algorithm LongReMix. We test LongReMix on the noisy-label benchmarks CIFAR-10, CIFAR-100, WebVision, Clothing1M, and Food101-N. Our approach achieves state-of-the-art performance in most datasets.
  arXiv  Detail & Related papers  (2021-03-06T18:48:40Z)
• Coresets for Robust Training of Neural Networks against Noisy Labels [78.03027938765746]
  We propose a novel approach with strong theoretical guarantees for robust training of deep networks trained with noisy labels. We select weighted subsets (coresets) of clean data points that provide an approximately low-rank Jacobian matrix. Our experiments corroborate our theory and demonstrate that deep networks trained on our subsets achieve a significantly superior performance compared to state-of-the art.
  arXiv  Detail & Related papers  (2020-11-15T04:58:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.