Robustness to Label Noise Depends on the Shape of the Noise Distribution
in Feature Space
- URL: http://arxiv.org/abs/2206.01106v1
- Date: Thu, 2 Jun 2022 15:41:59 GMT
- Title: Robustness to Label Noise Depends on the Shape of the Noise Distribution
in Feature Space
- Authors: Diane Oyen, Michal Kucer, Nick Hengartner, Har Simrat Singh
- Abstract summary: We show that both the scale and the shape of the noise distribution influence the posterior likelihood.
We show that when the noise distribution targets decision boundaries, classification robustness can drop off even at a small scale of noise.
- Score: 6.748225062396441
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Machine learning classifiers have been demonstrated, both empirically and
theoretically, to be robust to label noise under certain conditions -- notably
the typical assumption is that label noise is independent of the features given
the class label. We provide a theoretical framework that generalizes beyond
this typical assumption by modeling label noise as a distribution over feature
space. We show that both the scale and the shape of the noise distribution
influence the posterior likelihood; and the shape of the noise distribution has
a stronger impact on classification performance if the noise is concentrated in
feature space where the decision boundary can be moved. For the special case of
uniform label noise (independent of features and the class label), we show that
the Bayes optimal classifier for $c$ classes is robust to label noise until the
ratio of noisy samples goes above $\frac{c-1}{c}$ (e.g. 90% for 10 classes),
which we call the tipping point. However, for the special case of
class-dependent label noise (independent of features given the class label),
the tipping point can be as low as 50%. Most importantly, we show that when the
noise distribution targets decision boundaries (label noise is directly
dependent on feature space), classification robustness can drop off even at a
small scale of noise. Even when evaluating recent label-noise mitigation
methods we see reduced accuracy when label noise is dependent on features.
These findings explain why machine learning often handles label noise well if
the noise distribution is uniform in feature-space; yet it also points to the
difficulty of overcoming label noise when it is concentrated in a region of
feature space where a decision boundary can move.
Related papers
- Extracting Clean and Balanced Subset for Noisy Long-tailed Classification [66.47809135771698]
We develop a novel pseudo labeling method using class prototypes from the perspective of distribution matching.
By setting a manually-specific probability measure, we can reduce the side-effects of noisy and long-tailed data simultaneously.
Our method can extract this class-balanced subset with clean labels, which brings effective performance gains for long-tailed classification with label noise.
arXiv Detail & Related papers (2024-04-10T07:34:37Z) - Robust Online Classification: From Estimation to Denoising [14.535583931446807]
We study online classification of features into labels with general hypothesis classes.
Predictions are made using observed noisy labels and noiseless features.
The performance is measured via minimax risk when comparing against true labels.
arXiv Detail & Related papers (2023-09-04T16:17:39Z) - Handling Realistic Label Noise in BERT Text Classification [1.0515439489916731]
Real label noise is not random; rather, it is often correlated with input features or other annotator-specific factors.
We show that the presence of these types of noise significantly degrades BERT classification performance.
arXiv Detail & Related papers (2023-05-23T18:30:31Z) - Class Prototype-based Cleaner for Label Noise Learning [73.007001454085]
Semi-supervised learning methods are current SOTA solutions to the noisy-label learning problem.
We propose a simple yet effective solution, named textbfClass textbfPrototype-based label noise textbfCleaner.
arXiv Detail & Related papers (2022-12-21T04:56:41Z) - Instance-dependent Label-noise Learning under a Structural Causal Model [92.76400590283448]
Label noise will degenerate the performance of deep learning algorithms.
By leveraging a structural causal model, we propose a novel generative approach for instance-dependent label-noise learning.
arXiv Detail & Related papers (2021-09-07T10:42:54Z) - Learning with Feature-Dependent Label Noise: A Progressive Approach [19.425199841491246]
We propose a new family of feature-dependent label noise, which is much more general than commonly used i.i.d. label noise.
We provide theoretical guarantees showing that for a wide variety of (unknown) noise patterns, a classifier trained with this strategy converges to be consistent with the Bayes classifier.
arXiv Detail & Related papers (2021-03-13T17:34:22Z) - A Second-Order Approach to Learning with Instance-Dependent Label Noise [58.555527517928596]
The presence of label noise often misleads the training of deep neural networks.
We show that the errors in human-annotated labels are more likely to be dependent on the difficulty levels of tasks.
arXiv Detail & Related papers (2020-12-22T06:36:58Z) - Extended T: Learning with Mixed Closed-set and Open-set Noisy Labels [86.5943044285146]
The label noise transition matrix $T$ reflects the probabilities that true labels flip into noisy ones.
In this paper, we focus on learning under the mixed closed-set and open-set label noise.
Our method can better model the mixed label noise, following its more robust performance than the prior state-of-the-art label-noise learning methods.
arXiv Detail & Related papers (2020-12-02T02:42:45Z) - NoiseRank: Unsupervised Label Noise Reduction with Dependence Models [11.08987870095179]
We propose NoiseRank, for unsupervised label noise reduction using Markov Random Fields (MRF)
We construct a dependence model to estimate the posterior probability of an instance being incorrectly labeled given the dataset, and rank instances based on their estimated probabilities.
NoiseRank improves state-of-the-art classification on Food101-N (20% noise) and is effective on high noise Clothing-1M (40% noise)
arXiv Detail & Related papers (2020-03-15T01:10:25Z) - Multi-Class Classification from Noisy-Similarity-Labeled Data [98.13491369929798]
We propose a method for learning from only noisy-similarity-labeled data.
We use a noise transition matrix to bridge the class-posterior probability between clean and noisy data.
We build a novel learning system which can assign noise-free class labels for instances.
arXiv Detail & Related papers (2020-02-16T05:10:21Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.