Learning from Noisy Labels via Self-Taught On-the-Fly Meta Loss Rescaling

• URL: http://arxiv.org/abs/2412.12955v1
• Date: Tue, 17 Dec 2024 14:37:50 GMT
• Title: Learning from Noisy Labels via Self-Taught On-the-Fly Meta Loss Rescaling
• Authors: Michael Heck, Christian Geishauser, Nurul Lubis, Carel van Niekerk, Shutong Feng, Hsien-Chin Lin, Benjamin Matthias Ruppik, Renato Vukovic, Milica Gašić,
• Abstract summary: We propose unsupervised on-the-fly meta loss rescaling to reweight training samples. We are among the first to attempt on-the-fly training data reweighting on the challenging task of dialogue modeling. Our strategy is robust in the face of noisy and clean data, handles class imbalance, and prevents overfitting to noisy labels.
• Score: 6.861041888341339
• License:
• Abstract: Correct labels are indispensable for training effective machine learning models. However, creating high-quality labels is expensive, and even professionally labeled data contains errors and ambiguities. Filtering and denoising can be applied to curate labeled data prior to training, at the cost of additional processing and loss of information. An alternative is on-the-fly sample reweighting during the training process to decrease the negative impact of incorrect or ambiguous labels, but this typically requires clean seed data. In this work we propose unsupervised on-the-fly meta loss rescaling to reweight training samples. Crucially, we rely only on features provided by the model being trained, to learn a rescaling function in real time without knowledge of the true clean data distribution. We achieve this via a novel meta learning setup that samples validation data for the meta update directly from the noisy training corpus by employing the rescaling function being trained. Our proposed method consistently improves performance across various NLP tasks with minimal computational overhead. Further, we are among the first to attempt on-the-fly training data reweighting on the challenging task of dialogue modeling, where noisy and ambiguous labels are common. Our strategy is robust in the face of noisy and clean data, handles class imbalance, and prevents overfitting to noisy labels. Our self-taught loss rescaling improves as the model trains, showing the ability to keep learning from the model's own signals. As training progresses, the impact of correctly labeled data is scaled up, while the impact of wrongly labeled data is suppressed.

Related papers

• Early Stopping Against Label Noise Without Validation Data [54.27621957395026]
  We propose a novel early stopping method called Label Wave, which does not require validation data for selecting the desired model. We show both the effectiveness of the Label Wave method across various settings and its capability to enhance the performance of existing methods for learning with noisy labels.
  arXiv  Detail & Related papers  (2025-02-11T13:40:15Z)
• Learning in the Wild: Towards Leveraging Unlabeled Data for Effectively Tuning Pre-trained Code Models [38.7352992942213]
  We propose a novel approach named HINT to improve pre-trained code models with large-scale unlabeled datasets. HINT includes two main modules: HybrId pseudo-labeled data selection and Noise-tolerant Training. The experimental results show that HINT can better leverage those unlabeled data in a task-specific way.
  arXiv  Detail & Related papers  (2024-01-02T06:39:00Z)
• Soft Curriculum for Learning Conditional GANs with Noisy-Labeled and Uncurated Unlabeled Data [70.25049762295193]
  We introduce a novel conditional image generation framework that accepts noisy-labeled and uncurated data during training. We propose soft curriculum learning, which assigns instance-wise weights for adversarial training while assigning new labels for unlabeled data. Our experiments show that our approach outperforms existing semi-supervised and label-noise robust methods in terms of both quantitative and qualitative performance.
  arXiv  Detail & Related papers  (2023-07-17T08:31:59Z)
• Learning to Detect Noisy Labels Using Model-Based Features [16.681748918518075]
  We propose Selection-Enhanced Noisy label Training (SENT) SENT does not rely on meta learning while having the flexibility of being data-driven. It improves performance over strong baselines under the settings of self-training and label corruption.
  arXiv  Detail & Related papers  (2022-12-28T10:12:13Z)
• On-the-fly Denoising for Data Augmentation in Natural Language Understanding [101.46848743193358]
  We propose an on-the-fly denoising technique for data augmentation that learns from soft augmented labels provided by an organic teacher model trained on the cleaner original data. Our method can be applied to general augmentation techniques and consistently improve the performance on both text classification and question-answering tasks.
  arXiv  Detail & Related papers  (2022-12-20T18:58:33Z)
• Learning advisor networks for noisy image classification [22.77447144331876]
  We introduce the novel concept of advisor network to address the problem of noisy labels in image classification. We trained it with a meta-learning strategy so that it can adapt throughout the training of the main model. We tested our method on CIFAR10 and CIFAR100 with synthetic noise, and on Clothing1M which contains real-world noise, reporting state-of-the-art results.
  arXiv  Detail & Related papers  (2022-11-08T11:44:08Z)
• Pseudo-Label Noise Suppression Techniques for Semi-Supervised Semantic Segmentation [21.163070161951868]
  Semi-consuming learning (SSL) can reduce the need for large labelled datasets by incorporating unsupervised data into the training. Current SSL approaches use an initially supervised trained model to generate predictions for unlabelled images, called pseudo-labels. We use three mechanisms to control pseudo-label noise and errors.
  arXiv  Detail & Related papers  (2022-10-19T09:46:27Z)
• Semi-supervised Learning using Robust Loss [0.0]
  We suggest a semi-supervised training strategy for leveraging both manually labeled data and extra unlabeled data. In contrast to the existing approaches, we apply robust loss for the automated labeled data to compensate for the uneven data quality. We show that our proposed strategy improves the model performance by compensating for the uneven quality of labels in image classification.
  arXiv  Detail & Related papers  (2022-03-03T05:34:32Z)
• CaSP: Class-agnostic Semi-Supervised Pretraining for Detection and Segmentation [60.28924281991539]
  We propose a novel Class-agnostic Semi-supervised Pretraining (CaSP) framework to achieve a more favorable task-specificity balance. Using 3.6M unlabeled data, we achieve a remarkable performance gain of 4.7% over ImageNet-pretrained baseline on object detection.
  arXiv  Detail & Related papers  (2021-12-09T14:54:59Z)
• Instance Correction for Learning with Open-set Noisy Labels [145.06552420999986]
  We use the sample selection approach to handle open-set noisy labels. The discarded data are seen to be mislabeled and do not participate in training. We modify the instances of discarded data to make predictions for the discarded data consistent with given labels.
  arXiv  Detail & Related papers  (2021-06-01T13:05:55Z)
• Learning with Out-of-Distribution Data for Audio Classification [60.48251022280506]
  We show that detecting and relabelling certain OOD instances, rather than discarding them, can have a positive effect on learning. The proposed method is shown to improve the performance of convolutional neural networks by a significant margin.
  arXiv  Detail & Related papers  (2020-02-11T21:08:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.