AlleNoise -- large-scale text classification benchmark dataset with real-world label noise

• URL: http://arxiv.org/abs/2407.10992v1
• Date: Mon, 24 Jun 2024 09:29:14 GMT
• Title: AlleNoise -- large-scale text classification benchmark dataset with real-world label noise
• Authors: Alicja Rączkowska, Aleksandra Osowska-Kurczab, Jacek Szczerbiński, Kalina Jasinska-Kobus, Klaudia Nazarko,
• Abstract summary: We present AlleNoise, a new curated text classification benchmark dataset with real-world instance-dependent label noise. The noise distribution comes from actual users of a major e-commerce marketplace, so it realistically reflects the semantics of human mistakes. We demonstrate that a representative selection of established methods for learning with noisy labels is inadequate to handle such real-world noise.
• Score: 40.11095094521714
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Label noise remains a challenge for training robust classification models. Most methods for mitigating label noise have been benchmarked using primarily datasets with synthetic noise. While the need for datasets with realistic noise distribution has partially been addressed by web-scraped benchmarks such as WebVision and Clothing1M, those benchmarks are restricted to the computer vision domain. With the growing importance of Transformer-based models, it is crucial to establish text classification benchmarks for learning with noisy labels. In this paper, we present AlleNoise, a new curated text classification benchmark dataset with real-world instance-dependent label noise, containing over 500,000 examples across approximately 5,600 classes, complemented with a meaningful, hierarchical taxonomy of categories. The noise distribution comes from actual users of a major e-commerce marketplace, so it realistically reflects the semantics of human mistakes. In addition to the noisy labels, we provide human-verified clean labels, which help to get a deeper insight into the noise distribution, unlike web-scraped datasets typically used in the field. We demonstrate that a representative selection of established methods for learning with noisy labels is inadequate to handle such real-world noise. In addition, we show evidence that these algorithms do not alleviate excessive memorization. As such, with AlleNoise, we set the bar high for the development of label noise methods that can handle real-world label noise in text classification tasks. The code and dataset are available for download at https://github.com/allegro/AlleNoise.

Related papers

• NoisyAG-News: A Benchmark for Addressing Instance-Dependent Noise in Text Classification [7.464154519547575]
  Existing research on learning with noisy labels predominantly focuses on synthetic noise patterns. We constructed a benchmark dataset to better understand label noise in real-world text classification settings. Our findings reveal that while pre-trained models are resilient to synthetic noise, they struggle against instance-dependent noise.
  arXiv  Detail & Related papers  (2024-07-09T06:18:40Z)
• Generating the Ground Truth: Synthetic Data for Soft Label and Label Noise Research [0.0]
  We introduce SYNLABEL, a framework designed to create noiseless datasets informed by real-world data. We demonstrate its ability to precisely quantify label noise and its improvement over existing methodologies.
  arXiv  Detail & Related papers  (2023-09-08T13:31:06Z)
• Rethinking the Value of Labels for Instance-Dependent Label Noise Learning [43.481591776038144]
  noisy labels in real-world applications often depend on both the true label and the features. In this work, we tackle instance-dependent label noise with a novel deep generative model that avoids explicitly modeling the noise transition matrix. Our algorithm leverages casual representation learning and simultaneously identifies the high-level content and style latent factors from the data.
  arXiv  Detail & Related papers  (2023-05-10T15:29:07Z)
• Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations [54.400167806154535]
  Existing research on learning with noisy labels mainly focuses on synthetic label noise. This work presents two new benchmark datasets (CIFAR-10N, CIFAR-100N) We show that real-world noisy labels follow an instance-dependent pattern rather than the classically adopted class-dependent ones.
  arXiv  Detail & Related papers  (2021-10-22T22:42:11Z)
• Training Classifiers that are Universally Robust to All Label Noise Levels [91.13870793906968]
  Deep neural networks are prone to overfitting in the presence of label noise. We propose a distillation-based framework that incorporates a new subcategory of Positive-Unlabeled learning. Our framework generally outperforms at medium to high noise levels.
  arXiv  Detail & Related papers  (2021-05-27T13:49:31Z)
• Noisy Labels Can Induce Good Representations [53.47668632785373]
  We study how architecture affects learning with noisy labels. We show that training with noisy labels can induce useful hidden representations, even when the model generalizes poorly. This finding leads to a simple method to improve models trained on noisy labels.
  arXiv  Detail & Related papers  (2020-12-23T18:58:05Z)
• A Second-Order Approach to Learning with Instance-Dependent Label Noise [58.555527517928596]
  The presence of label noise often misleads the training of deep neural networks. We show that the errors in human-annotated labels are more likely to be dependent on the difficulty levels of tasks.
  arXiv  Detail & Related papers  (2020-12-22T06:36:58Z)
• Label Noise Types and Their Effects on Deep Learning [0.0]
  In this work, we provide a detailed analysis of the effects of different kinds of label noise on learning. We propose a generic framework to generate feature-dependent label noise, which we show to be the most challenging case for learning. For the ease of other researchers to test their algorithms with noisy labels, we share corrupted labels for the most commonly used benchmark datasets.
  arXiv  Detail & Related papers  (2020-03-23T18:03:39Z)
• NoiseRank: Unsupervised Label Noise Reduction with Dependence Models [11.08987870095179]
  We propose NoiseRank, for unsupervised label noise reduction using Markov Random Fields (MRF) We construct a dependence model to estimate the posterior probability of an instance being incorrectly labeled given the dataset, and rank instances based on their estimated probabilities. NoiseRank improves state-of-the-art classification on Food101-N (20% noise) and is effective on high noise Clothing-1M (40% noise)
  arXiv  Detail & Related papers  (2020-03-15T01:10:25Z)
• Multi-Class Classification from Noisy-Similarity-Labeled Data [98.13491369929798]
  We propose a method for learning from only noisy-similarity-labeled data. We use a noise transition matrix to bridge the class-posterior probability between clean and noisy data. We build a novel learning system which can assign noise-free class labels for instances.
  arXiv  Detail & Related papers  (2020-02-16T05:10:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.