NoisyAG-News: A Benchmark for Addressing Instance-Dependent Noise in Text Classification

• URL: http://arxiv.org/abs/2407.06579v1
• Date: Tue, 9 Jul 2024 06:18:40 GMT
• Title: NoisyAG-News: A Benchmark for Addressing Instance-Dependent Noise in Text Classification
• Authors: Hongfei Huang, Tingting Liang, Xixi Sun, Zikang Jin, Yuyu Yin,
• Abstract summary: Existing research on learning with noisy labels predominantly focuses on synthetic noise patterns. We constructed a benchmark dataset to better understand label noise in real-world text classification settings. Our findings reveal that while pre-trained models are resilient to synthetic noise, they struggle against instance-dependent noise.
• Score: 7.464154519547575
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Existing research on learning with noisy labels predominantly focuses on synthetic label noise. Although synthetic noise possesses well-defined structural properties, it often fails to accurately replicate real-world noise patterns. In recent years, there has been a concerted effort to construct generalizable and controllable instance-dependent noise datasets for image classification, significantly advancing the development of noise-robust learning in this area. However, studies on noisy label learning for text classification remain scarce. To better understand label noise in real-world text classification settings, we constructed the benchmark dataset NoisyAG-News through manual annotation. Initially, we analyzed the annotated data to gather observations about real-world noise. We qualitatively and quantitatively demonstrated that real-world noisy labels adhere to instance-dependent patterns. Subsequently, we conducted comprehensive learning experiments on NoisyAG-News and its corresponding synthetic noise datasets using pre-trained language models and noise-handling techniques. Our findings reveal that while pre-trained models are resilient to synthetic noise, they struggle against instance-dependent noise, with samples of varying confusion levels showing inconsistent performance during training and testing. These real-world noise patterns pose new, significant challenges, prompting a reevaluation of noisy label handling methods. We hope that NoisyAG-News will facilitate the development and evaluation of future solutions for learning with noisy labels.

Related papers

• NoiseBench: Benchmarking the Impact of Real Label Noise on Named Entity Recognition [3.726602636064681]
  We present an analysis that shows that real noise is significantly more challenging than simulated noise. We show that current state-of-the-art models for noise-robust learning fall far short of their theoretically achievable upper bound.
  arXiv  Detail & Related papers  (2024-05-13T10:20:31Z)
• NoisywikiHow: A Benchmark for Learning with Real-world Noisy Labels in Natural Language Processing [26.678589684142548]
  Large-scale datasets in the real world inevitably involve label noise. Deep models can gradually overfit noisy labels and thus degrade generalization performance. To mitigate the effects of label noise, learning with noisy labels (LNL) methods are designed to achieve better generalization performance.
  arXiv  Detail & Related papers  (2023-05-18T05:01:04Z)
• Latent Class-Conditional Noise Model [54.56899309997246]
  We introduce a Latent Class-Conditional Noise model (LCCN) to parameterize the noise transition under a Bayesian framework. We then deduce a dynamic label regression method for LCCN, whose Gibbs sampler allows us efficiently infer the latent true labels. Our approach safeguards the stable update of the noise transition, which avoids previous arbitrarily tuning from a mini-batch of samples.
  arXiv  Detail & Related papers  (2023-02-19T15:24:37Z)
• Is BERT Robust to Label Noise? A Study on Learning with Noisy Labels in Text Classification [23.554544399110508]
  Wrong labels in training data occur when human annotators make mistakes or when the data is generated via weak or distant supervision. It has been shown that complex noise-handling techniques are required to prevent models from fitting this label noise. We show in this work that, for text classification tasks with modern NLP models like BERT, over a variety of noise types, existing noisehandling methods do not always improve its performance, and may even deteriorate it.
  arXiv  Detail & Related papers  (2022-04-20T10:24:19Z)
• Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations [54.400167806154535]
  Existing research on learning with noisy labels mainly focuses on synthetic label noise. This work presents two new benchmark datasets (CIFAR-10N, CIFAR-100N) We show that real-world noisy labels follow an instance-dependent pattern rather than the classically adopted class-dependent ones.
  arXiv  Detail & Related papers  (2021-10-22T22:42:11Z)
• Learning to Aggregate and Refine Noisy Labels for Visual Sentiment Analysis [69.48582264712854]
  We propose a robust learning method to perform robust visual sentiment analysis. Our method relies on an external memory to aggregate and filter noisy labels during training. We establish a benchmark for visual sentiment analysis with label noise using publicly available datasets.
  arXiv  Detail & Related papers  (2021-09-15T18:18:28Z)
• Training Classifiers that are Universally Robust to All Label Noise Levels [91.13870793906968]
  Deep neural networks are prone to overfitting in the presence of label noise. We propose a distillation-based framework that incorporates a new subcategory of Positive-Unlabeled learning. Our framework generally outperforms at medium to high noise levels.
  arXiv  Detail & Related papers  (2021-05-27T13:49:31Z)
• Analysing the Noise Model Error for Realistic Noisy Label Data [14.766574408868806]
  We study the quality of estimated noise models from the theoretical side by deriving the expected error of the noise model. We also publish NoisyNER, a new noisy label dataset from the NLP domain.
  arXiv  Detail & Related papers  (2021-01-24T17:45:15Z)
• Tackling Instance-Dependent Label Noise via a Universal Probabilistic Model [80.91927573604438]
  This paper proposes a simple yet universal probabilistic model, which explicitly relates noisy labels to their instances. Experiments on datasets with both synthetic and real-world label noise verify that the proposed method yields significant improvements on robustness.
  arXiv  Detail & Related papers  (2021-01-14T05:43:51Z)
• A Second-Order Approach to Learning with Instance-Dependent Label Noise [58.555527517928596]
  The presence of label noise often misleads the training of deep neural networks. We show that the errors in human-annotated labels are more likely to be dependent on the difficulty levels of tasks.
  arXiv  Detail & Related papers  (2020-12-22T06:36:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.