Verifix: Post-Training Correction to Improve Label Noise Robustness with Verified Samples

• URL: http://arxiv.org/abs/2403.08618v1
• Date: Wed, 13 Mar 2024 15:32:08 GMT
• Title: Verifix: Post-Training Correction to Improve Label Noise Robustness with Verified Samples
• Authors: Sangamesh Kodge, Deepak Ravikumar, Gobinda Saha, Kaushik Roy
• Abstract summary: Post-Training Correction adjusts model parameters after initial training to mitigate label noise. We introduce Verifix, a novel algorithm that leverages a small, verified dataset to correct the model weights using a single update. Experiments on the CIFAR dataset with 25% synthetic corruption show 7.36% generalization improvements on average.
• Score: 9.91998873101083
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Label corruption, where training samples have incorrect labels, can significantly degrade the performance of machine learning models. This corruption often arises from non-expert labeling or adversarial attacks. Acquiring large, perfectly labeled datasets is costly, and retraining large models from scratch when a clean dataset becomes available is computationally expensive. To address this challenge, we propose Post-Training Correction, a new paradigm that adjusts model parameters after initial training to mitigate label noise, eliminating the need for retraining. We introduce Verifix, a novel Singular Value Decomposition (SVD) based algorithm that leverages a small, verified dataset to correct the model weights using a single update. Verifix uses SVD to estimate a Clean Activation Space and then projects the model's weights onto this space to suppress activations corresponding to corrupted data. We demonstrate Verifix's effectiveness on both synthetic and real-world label noise. Experiments on the CIFAR dataset with 25% synthetic corruption show 7.36% generalization improvements on average. Additionally, we observe generalization improvements of up to 2.63% on naturally corrupted datasets like WebVision1.0 and Clothing1M.

Related papers

• Data Diet: Can Trimming PET/CT Datasets Enhance Lesion Segmentation? [68.51625420337575]
  We describe our approach to compete in the autoPET3 datacentric track. We find that in the autoPETIII dataset, a model that is trained on the entire dataset exhibits undesirable characteristics. We counteract this by removing the easiest samples from the training dataset as measured by the model loss before retraining from scratch.
  arXiv  Detail & Related papers  (2024-09-20T14:47:58Z)
• Jump-teaching: Ultra Efficient and Robust Learning with Noisy Label [6.818488262543482]
  We propose a novel technique to distinguish mislabeled samples during training. We employ only one network with the jump manner update to decouple the interplay and mine more semantic information from the loss. Our proposed approach achieves almost up to $2.53times$ speedup, $0.56times$ peak memory footprint, and superior robustness over state-of-the-art works with various noise settings.
  arXiv  Detail & Related papers  (2024-05-27T12:54:09Z)
• Parameter-tuning-free data entry error unlearning with adaptive selective synaptic dampening [51.34904967046097]
  We introduce an extension to the selective synaptic dampening unlearning method that removes the need for parameter tuning. We demonstrate the performance of this extension, adaptive selective synaptic dampening (ASSD) on various ResNet18 and Vision Transformer unlearning tasks. The application of this approach is particularly compelling in industrial settings, such as supply chain management.
  arXiv  Detail & Related papers  (2024-02-06T14:04:31Z)
• Learning in the Wild: Towards Leveraging Unlabeled Data for Effectively Tuning Pre-trained Code Models [38.7352992942213]
  We propose a novel approach named HINT to improve pre-trained code models with large-scale unlabeled datasets. HINT includes two main modules: HybrId pseudo-labeled data selection and Noise-tolerant Training. The experimental results show that HINT can better leverage those unlabeled data in a task-specific way.
  arXiv  Detail & Related papers  (2024-01-02T06:39:00Z)
• Efficient Grammatical Error Correction Via Multi-Task Training and Optimized Training Schedule [55.08778142798106]
  We propose auxiliary tasks that exploit the alignment between the original and corrected sentences. We formulate each task as a sequence-to-sequence problem and perform multi-task training. We find that the order of datasets used for training and even individual instances within a dataset may have important effects on the final performance.
  arXiv  Detail & Related papers  (2023-11-20T14:50:12Z)
• Label-Retrieval-Augmented Diffusion Models for Learning from Noisy Labels [61.97359362447732]
  Learning from noisy labels is an important and long-standing problem in machine learning for real applications. In this paper, we reformulate the label-noise problem from a generative-model perspective. Our model achieves new state-of-the-art (SOTA) results on all the standard real-world benchmark datasets.
  arXiv  Detail & Related papers  (2023-05-31T03:01:36Z)
• Augment and Criticize: Exploring Informative Samples for Semi-Supervised Monocular 3D Object Detection [64.65563422852568]
  We improve the challenging monocular 3D object detection problem with a general semi-supervised framework. We introduce a novel, simple, yet effective Augment and Criticize' framework that explores abundant informative samples from unlabeled data. The two new detectors, dubbed 3DSeMo_DLE and 3DSeMo_FLEX, achieve state-of-the-art results with remarkable improvements for over 3.5% AP_3D/BEV (Easy) on KITTI.
  arXiv  Detail & Related papers  (2023-03-20T16:28:15Z)
• Knockoffs-SPR: Clean Sample Selection in Learning with Noisy Labels [56.81761908354718]
  We propose a novel theoretically guaranteed clean sample selection framework for learning with noisy labels. Knockoffs-SPR can be regarded as a sample selection module for a standard supervised training pipeline. We further combine it with a semi-supervised algorithm to exploit the support of noisy data as unlabeled data.
  arXiv  Detail & Related papers  (2023-01-02T07:13:28Z)
• CTRL: Clustering Training Losses for Label Error Detection [4.49681473359251]
  In supervised machine learning, use of correct labels is extremely important to ensure high accuracy. We propose a novel framework, calledClustering TRaining Losses for label error detection. It detects label errors in two steps based on the observation that models learn clean and noisy labels in different ways.
  arXiv  Detail & Related papers  (2022-08-17T18:09:19Z)
• Self-paced Resistance Learning against Overfitting on Noisy Labels [25.916498598323667]
  Deep neural networks might first memorize the probably correct-label data and then corrupt-label samples. We propose a novel yet simple self-paced resistance framework to resist corrupted labels.
  arXiv  Detail & Related papers  (2021-05-07T04:17:20Z)
• Improving Generalization of Deep Fault Detection Models in the Presence of Mislabeled Data [1.3535770763481902]
  We propose a novel two-step framework for robust training with label noise. In the first step, we identify outliers (including the mislabeled samples) based on the update in the hypothesis space. In the second step, we propose different approaches to modifying the training data based on the identified outliers and a data augmentation technique.
  arXiv  Detail & Related papers  (2020-09-30T12:33:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.