CleanCoNLL: A Nearly Noise-Free Named Entity Recognition Dataset

• URL: http://arxiv.org/abs/2310.16225v1
• Date: Tue, 24 Oct 2023 22:34:43 GMT
• Title: CleanCoNLL: A Nearly Noise-Free Named Entity Recognition Dataset
• Abstract summary: We present a comprehensive relabeling effort assisted by automatic consistency checking that corrects 7.0% of all labels in the English CoNLL-03. Our experimental evaluation finds not only that state-of-the-art approaches reach significantly higher F1-scores (97.1%) on our data, but crucially that the share of correct predictions falsely counted as errors due to annotation noise drops from 47% to 6%. This indicates that our resource is well suited to analyze the remaining errors made by state-of-the-art models, and that the theoretical upper bound even on high resource, coarse-grained NER is not yet
• Score: 3.3162484539136416
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The CoNLL-03 corpus is arguably the most well-known and utilized benchmark dataset for named entity recognition (NER). However, prior works found significant numbers of annotation errors, incompleteness, and inconsistencies in the data. This poses challenges to objectively comparing NER approaches and analyzing their errors, as current state-of-the-art models achieve F1-scores that are comparable to or even exceed the estimated noise level in CoNLL-03. To address this issue, we present a comprehensive relabeling effort assisted by automatic consistency checking that corrects 7.0% of all labels in the English CoNLL-03. Our effort adds a layer of entity linking annotation both for better explainability of NER labels and as additional safeguard of annotation quality. Our experimental evaluation finds not only that state-of-the-art approaches reach significantly higher F1-scores (97.1%) on our data, but crucially that the share of correct predictions falsely counted as errors due to annotation noise drops from 47% to 6%. This indicates that our resource is well suited to analyze the remaining errors made by state-of-the-art models, and that the theoretical upper bound even on high resource, coarse-grained NER is not yet reached. To facilitate such analysis, we make CleanCoNLL publicly available to the research community.

Related papers

• ER-KANs: Efficient and Robust Kolmogorov-Arnold Networks for Data-Scarce Scientific Machine Learning [0.0]
  We show that ChebyKAN's test MSE increases by a factor of 10.6x when training data is corrupted with sigma=0.1 noise.<n> ER-KAN combines three design choices targeting the noisy, data-scarce setting.
  arXiv  Detail & Related papers  (2026-08-14T16:15:37Z)
• Hard Cases, Bad Labels: Testing Error Exposure and Error Location in Uncertainty Sampling Under Bounded Label Noise [0.0]
  Active learning can reduce labeling cost by selecting informative examples, but the most uncertain examples may also be the hardest to label correctly.<n>This study tests whether uncertainty sampling fails because it acquires more corrupted labels or because errors concentrated in difficult regions are especially harmful.
  arXiv  Detail & Related papers  (2026-08-05T03:01:19Z)
• HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam [63.84155758655084]
  Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models.<n>We introduce HLE-Verified, a verified and revised version of HLE with a transparent verification protocol and fine-grained error taxonomy.<n>We evaluate seven state-of-the-art language models on HLE and HLE-Verified, observing an average absolute accuracy gain of 7--10 percentage points.
  arXiv  Detail & Related papers  (2026-02-15T02:50:15Z)
• Label Convergence: Defining an Upper Performance Bound in Object Recognition through Contradictory Annotations [0.0]
  We introduce the notion of "label convergence" to describe the highest achievable performance under the constraint of contradictory test annotations. We approximate that label convergence is between 62.63-67.52 mAP@[0.5:0.95:0.05] for LVIS with 95% confidence, attributing these bounds to the presence of real annotation errors. With current state-of-the-art (SOTA) models at the upper end of the label convergence interval for the well-studied LVIS dataset, we conclude that model capacity is sufficient to solve current object detection problems.
  arXiv  Detail & Related papers  (2024-09-14T10:59:25Z)
• CLEANANERCorp: Identifying and Correcting Incorrect Labels in the ANERcorp Dataset [0.0]
  We dive deep into one of the widely adopted Arabic NER benchmark datasets (ANERcorp) We find a significant number of annotation errors, missing labels, and inconsistencies. We propose a cleaner version of the dataset named CLEANANERCorp.
  arXiv  Detail & Related papers  (2024-08-22T12:59:05Z)
• Contrastive Error Attribution for Finetuned Language Models [35.80256755393739]
  noisy and misannotated data is a core cause of hallucinations and unfaithful outputs in Natural Language Generation (NLG) tasks. We introduce a framework to identify and remove low-quality training instances that lead to undesirable outputs. We show that existing approaches for error tracing, such as gradient-based influence measures, do not perform reliably for detecting faithfulness errors.
  arXiv  Detail & Related papers  (2022-12-21T02:28:07Z)
• Neighborhood Collective Estimation for Noisy Label Identification and Correction [92.20697827784426]
  Learning with noisy labels (LNL) aims at designing strategies to improve model performance and generalization by mitigating the effects of model overfitting to noisy labels. Recent advances employ the predicted label distributions of individual samples to perform noise verification and noisy label correction, easily giving rise to confirmation bias. We propose Neighborhood Collective Estimation, in which the predictive reliability of a candidate sample is re-estimated by contrasting it against its feature-space nearest neighbors.
  arXiv  Detail & Related papers  (2022-08-05T14:47:22Z)
• Sparse Conditional Hidden Markov Model for Weakly Supervised Named Entity Recognition [68.68300358332156]
  We propose the sparse conditional hidden Markov model (Sparse-CHMM) to evaluate noisy labeling functions. Sparse-CHMM is optimized through unsupervised learning with a three-stage training pipeline. It achieves a 3.01 average F1 score improvement on five comprehensive datasets.
  arXiv  Detail & Related papers  (2022-05-27T20:47:30Z)
• Named Entity Recognition with Small Strongly Labeled and Large Weakly Labeled Data [37.980010197914105]
  Weak supervision has shown promising results in many natural language processing tasks, such as Named Entity Recognition (NER) We propose a new multi-stage computational framework -- NEEDLE with three essential ingredients: weak label completion, noise-aware loss function, and final fine-tuning over the strongly labeled data. We demonstrate that NEEDLE can effectively suppress the noise of the weak labels and outperforms existing methods.
  arXiv  Detail & Related papers  (2021-06-16T17:18:14Z)
• Boosting Semi-Supervised Face Recognition with Noise Robustness [54.342992887966616]
  This paper presents an effective solution to semi-supervised face recognition that is robust to the label noise aroused by the auto-labelling. We develop a semi-supervised face recognition solution, named Noise Robust Learning-Labelling (NRoLL), which is based on the robust training ability empowered by GN.
  arXiv  Detail & Related papers  (2021-05-10T14:43:11Z)
• Re-TACRED: Addressing Shortcomings of the TACRED Dataset [5.820381428297218]
  TACRED is one of the largest and most widely used sentence-level relation extraction datasets. Proposed models that are evaluated using this dataset consistently set new state-of-the-art performance. However, they still exhibit large error rates despite leveraging external knowledge and unsupervised pretraining on large text corpora.
  arXiv  Detail & Related papers  (2021-04-16T22:55:11Z)
• Exploiting Sample Uncertainty for Domain Adaptive Person Re-Identification [137.9939571408506]
  We estimate and exploit the credibility of the assigned pseudo-label of each sample to alleviate the influence of noisy labels. Our uncertainty-guided optimization brings significant improvement and achieves the state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2020-12-16T04:09:04Z)
• Improving Face Recognition by Clustering Unlabeled Faces in the Wild [77.48677160252198]
  We propose a novel identity separation method based on extreme value theory. It greatly reduces the problems caused by overlapping-identity label noise. Experiments on both controlled and real settings demonstrate our method's consistent improvements.
  arXiv  Detail & Related papers  (2020-07-14T12:26:50Z)
• TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [80.38130122127882]
  TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE) In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled.
  arXiv  Detail & Related papers  (2020-04-30T15:07:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.