NoiseBench: Benchmarking the Impact of Real Label Noise on Named Entity Recognition

• URL: http://arxiv.org/abs/2405.07609v2
• Date: Mon, 14 Oct 2024 10:19:37 GMT
• Title: NoiseBench: Benchmarking the Impact of Real Label Noise on Named Entity Recognition
• Authors: Elena Merdjanovska, Ansar Aynetdinov, Alan Akbik,
• Abstract summary: We present an analysis that shows that real noise is significantly more challenging than simulated noise. We show that current state-of-the-art models for noise-robust learning fall far short of their theoretically achievable upper bound.
• Score: 3.726602636064681
• License:
• Abstract: Available training data for named entity recognition (NER) often contains a significant percentage of incorrect labels for entity types and entity boundaries. Such label noise poses challenges for supervised learning and may significantly deteriorate model quality. To address this, prior work proposed various noise-robust learning approaches capable of learning from data with partially incorrect labels. These approaches are typically evaluated using simulated noise where the labels in a clean dataset are automatically corrupted. However, as we show in this paper, this leads to unrealistic noise that is far easier to handle than real noise caused by human error or semi-automatic annotation. To enable the study of the impact of various types of real noise, we introduce NoiseBench, an NER benchmark consisting of clean training data corrupted with 6 types of real noise, including expert errors, crowdsourcing errors, automatic annotation errors and LLM errors. We present an analysis that shows that real noise is significantly more challenging than simulated noise, and show that current state-of-the-art models for noise-robust learning fall far short of their theoretically achievable upper bound. We release NoiseBench to the research community.

Related papers

• NoisyAG-News: A Benchmark for Addressing Instance-Dependent Noise in Text Classification [7.464154519547575]
  Existing research on learning with noisy labels predominantly focuses on synthetic noise patterns. We constructed a benchmark dataset to better understand label noise in real-world text classification settings. Our findings reveal that while pre-trained models are resilient to synthetic noise, they struggle against instance-dependent noise.
  arXiv  Detail & Related papers  (2024-07-09T06:18:40Z)
• SoftPatch: Unsupervised Anomaly Detection with Noisy Data [67.38948127630644]
  This paper considers label-level noise in image sensory anomaly detection for the first time. We propose a memory-based unsupervised AD method, SoftPatch, which efficiently denoises the data at the patch level. Compared with existing methods, SoftPatch maintains a strong modeling ability of normal data and alleviates the overconfidence problem in coreset.
  arXiv  Detail & Related papers  (2024-03-21T08:49:34Z)
• NoisywikiHow: A Benchmark for Learning with Real-world Noisy Labels in Natural Language Processing [26.678589684142548]
  Large-scale datasets in the real world inevitably involve label noise. Deep models can gradually overfit noisy labels and thus degrade generalization performance. To mitigate the effects of label noise, learning with noisy labels (LNL) methods are designed to achieve better generalization performance.
  arXiv  Detail & Related papers  (2023-05-18T05:01:04Z)
• Improving the Robustness of Summarization Models by Detecting and Removing Input Noise [50.27105057899601]
  We present a large empirical study quantifying the sometimes severe loss in performance from different types of input noise for a range of datasets and model sizes. We propose a light-weight method for detecting and removing such noise in the input during model inference without requiring any training, auxiliary models, or even prior knowledge of the type of noise.
  arXiv  Detail & Related papers  (2022-12-20T00:33:11Z)
• Noise tolerance of learning to rank under class-conditional label noise [1.14219428942199]
  We describe a class of noise-tolerant LtR losses for which empirical risk minimization is a consistent procedure. We also develop noise-tolerant analogs of commonly used loss functions.
  arXiv  Detail & Related papers  (2022-08-03T15:04:48Z)
• Is BERT Robust to Label Noise? A Study on Learning with Noisy Labels in Text Classification [23.554544399110508]
  Wrong labels in training data occur when human annotators make mistakes or when the data is generated via weak or distant supervision. It has been shown that complex noise-handling techniques are required to prevent models from fitting this label noise. We show in this work that, for text classification tasks with modern NLP models like BERT, over a variety of noise types, existing noisehandling methods do not always improve its performance, and may even deteriorate it.
  arXiv  Detail & Related papers  (2022-04-20T10:24:19Z)
• Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations [54.400167806154535]
  Existing research on learning with noisy labels mainly focuses on synthetic label noise. This work presents two new benchmark datasets (CIFAR-10N, CIFAR-100N) We show that real-world noisy labels follow an instance-dependent pattern rather than the classically adopted class-dependent ones.
  arXiv  Detail & Related papers  (2021-10-22T22:42:11Z)
• Training Classifiers that are Universally Robust to All Label Noise Levels [91.13870793906968]
  Deep neural networks are prone to overfitting in the presence of label noise. We propose a distillation-based framework that incorporates a new subcategory of Positive-Unlabeled learning. Our framework generally outperforms at medium to high noise levels.
  arXiv  Detail & Related papers  (2021-05-27T13:49:31Z)
• Analysing the Noise Model Error for Realistic Noisy Label Data [14.766574408868806]
  We study the quality of estimated noise models from the theoretical side by deriving the expected error of the noise model. We also publish NoisyNER, a new noisy label dataset from the NLP domain.
  arXiv  Detail & Related papers  (2021-01-24T17:45:15Z)
• Tackling Instance-Dependent Label Noise via a Universal Probabilistic Model [80.91927573604438]
  This paper proposes a simple yet universal probabilistic model, which explicitly relates noisy labels to their instances. Experiments on datasets with both synthetic and real-world label noise verify that the proposed method yields significant improvements on robustness.
  arXiv  Detail & Related papers  (2021-01-14T05:43:51Z)
• Towards Noise-resistant Object Detection with Noisy Annotations [119.63458519946691]
  Training deep object detectors requires significant amount of human-annotated images with accurate object labels and bounding box coordinates. Noisy annotations are much more easily accessible, but they could be detrimental for learning. We address the challenging problem of training object detectors with noisy annotations, where the noise contains a mixture of label noise and bounding box noise.
  arXiv  Detail & Related papers  (2020-03-03T01:32:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.