Jump-teaching: Ultra Efficient and Robust Learning with Noisy Label

• URL: http://arxiv.org/abs/2405.17137v4
• Date: Tue, 27 Aug 2024 04:02:58 GMT
• Title: Jump-teaching: Ultra Efficient and Robust Learning with Noisy Label
• Authors: Kangye Ji, Fei Cheng, Zeqing Wang, Bohu Huang,
• Abstract summary: We propose a novel technique to distinguish mislabeled samples during training. We employ only one network with the jump manner update to decouple the interplay and mine more semantic information from the loss. Our proposed approach achieves almost up to $2.53times$ speedup, $0.56times$ peak memory footprint, and superior robustness over state-of-the-art works with various noise settings.
• Score: 6.818488262543482
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Sample selection is the most straightforward technique to combat label noise, aiming to distinguish mislabeled samples during training and avoid the degradation of the robustness of the model. In the workflow, $\textit{selecting possibly clean data}$ and $\textit{model update}$ are iterative. However, their interplay and intrinsic characteristics hinder the robustness and efficiency of learning with noisy labels: 1) The model chooses clean data with selection bias, leading to the accumulated error in the model update. 2) Most selection strategies leverage partner networks or supplementary information to mitigate label corruption, albeit with increased computation resources and lower throughput speed. Therefore, we employ only one network with the jump manner update to decouple the interplay and mine more semantic information from the loss for a more precise selection. Specifically, the selection of clean data for each model update is based on one of the prior models, excluding the last iteration. The strategy of model update exhibits a jump behavior in the form. Moreover, we map the outputs of the network and labels into the same semantic feature space, respectively. In this space, a detailed and simple loss distribution is generated to distinguish clean samples more effectively. Our proposed approach achieves almost up to $2.53\times$ speedup, $0.46\times$ peak memory footprint, and superior robustness over state-of-the-art works with various noise settings.

Related papers

• Verifix: Post-Training Correction to Improve Label Noise Robustness with Verified Samples [9.91998873101083]
  Post-Training Correction adjusts model parameters after initial training to mitigate label noise. We introduce Verifix, a novel algorithm that leverages a small, verified dataset to correct the model weights using a single update. Experiments on the CIFAR dataset with 25% synthetic corruption show 7.36% generalization improvements on average.
  arXiv  Detail & Related papers  (2024-03-13T15:32:08Z)
• Combating Label Noise With A General Surrogate Model For Sample Selection [84.61367781175984]
  We propose to leverage the vision-language surrogate model CLIP to filter noisy samples automatically. We validate the effectiveness of our proposed method on both real-world and synthetic noisy datasets.
  arXiv  Detail & Related papers  (2023-10-16T14:43:27Z)
• Label-Retrieval-Augmented Diffusion Models for Learning from Noisy Labels [61.97359362447732]
  Learning from noisy labels is an important and long-standing problem in machine learning for real applications. In this paper, we reformulate the label-noise problem from a generative-model perspective. Our model achieves new state-of-the-art (SOTA) results on all the standard real-world benchmark datasets.
  arXiv  Detail & Related papers  (2023-05-31T03:01:36Z)
• Knockoffs-SPR: Clean Sample Selection in Learning with Noisy Labels [56.81761908354718]
  We propose a novel theoretically guaranteed clean sample selection framework for learning with noisy labels. Knockoffs-SPR can be regarded as a sample selection module for a standard supervised training pipeline. We further combine it with a semi-supervised algorithm to exploit the support of noisy data as unlabeled data.
  arXiv  Detail & Related papers  (2023-01-02T07:13:28Z)
• Learning to Detect Noisy Labels Using Model-Based Features [16.681748918518075]
  We propose Selection-Enhanced Noisy label Training (SENT) SENT does not rely on meta learning while having the flexibility of being data-driven. It improves performance over strong baselines under the settings of self-training and label corruption.
  arXiv  Detail & Related papers  (2022-12-28T10:12:13Z)
• UNICON: Combating Label Noise Through Uniform Selection and Contrastive Learning [89.56465237941013]
  We propose UNICON, a simple yet effective sample selection method which is robust to high label noise. We obtain an 11.4% improvement over the current state-of-the-art on CIFAR100 dataset with a 90% noise rate.
  arXiv  Detail & Related papers  (2022-03-28T07:36:36Z)
• L2B: Learning to Bootstrap Robust Models for Combating Label Noise [52.02335367411447]
  This paper introduces a simple and effective method, named Learning to Bootstrap (L2B) It enables models to bootstrap themselves using their own predictions without being adversely affected by erroneous pseudo-labels. It achieves this by dynamically adjusting the importance weight between real observed and generated labels, as well as between different samples through meta-learning.
  arXiv  Detail & Related papers  (2022-02-09T05:57:08Z)
• Cross-Model Pseudo-Labeling for Semi-Supervised Action Recognition [98.25592165484737]
  We propose a more effective pseudo-labeling scheme, called Cross-Model Pseudo-Labeling (CMPL) CMPL achieves $17.6%$ and $25.1%$ Top-1 accuracy on Kinetics-400 and UCF-101 using only the RGB modality and $1%$ labeled data, respectively.
  arXiv  Detail & Related papers  (2021-12-17T18:59:41Z)
• Robust Deep Learning from Crowds with Belief Propagation [6.643082745560235]
  A graphical model representing local dependencies between workers and tasks provides a principled way of reasoning over the true labels from the noisy answers. One needs a predictive model working on unseen data directly from crowdsourced datasets instead of the true labels in many cases. We propose a new data-generating process, where a neural network generates the true labels from task features.
  arXiv  Detail & Related papers  (2021-11-01T07:20:16Z)
• Learning from Noisy Labels for Entity-Centric Information Extraction [17.50856935207308]
  We propose a simple co-regularization framework for entity-centric information extraction. These models are jointly optimized with task-specific loss, and are regularized to generate similar predictions. In the end, we can take any of the trained models for inference.
  arXiv  Detail & Related papers  (2021-04-17T22:49:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.