CLEANANERCorp: Identifying and Correcting Incorrect Labels in the ANERcorp Dataset

• URL: http://arxiv.org/abs/2408.12362v1
• Date: Thu, 22 Aug 2024 12:59:05 GMT
• Title: CLEANANERCorp: Identifying and Correcting Incorrect Labels in the ANERcorp Dataset
• Abstract summary: We dive deep into one of the widely adopted Arabic NER benchmark datasets (ANERcorp) We find a significant number of annotation errors, missing labels, and inconsistencies. We propose a cleaner version of the dataset named CLEANANERCorp.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Label errors are a common issue in machine learning datasets, particularly for tasks such as Named Entity Recognition. Such label errors might hurt model training, affect evaluation results, and lead to an inaccurate assessment of model performance. In this study, we dived deep into one of the widely adopted Arabic NER benchmark datasets (ANERcorp) and found a significant number of annotation errors, missing labels, and inconsistencies. Therefore, in this study, we conducted empirical research to understand these errors, correct them and propose a cleaner version of the dataset named CLEANANERCorp. CLEANANERCorp will serve the research community as a more accurate and consistent benchmark.

Related papers

• From Label Error Detection to Correction: A Modular Framework and Benchmark for Object Detection Datasets [4.864032555684836]
  We introduce a semi-automated framework for label-error correction called REC$checkmark$D (Rechecked)<n>We show that current label error detection methods, when combined with our correction framework, can recover hundreds of errors in the time it would take a human to annotate bounding boxes from scratch.<n>This validated set will be released as a new real-world benchmark for label error detection and correction.
  arXiv  Detail & Related papers  (2025-08-06T10:03:05Z)
• Class-wise Autoencoders Measure Classification Difficulty And Detect Label Mistakes [22.45812577928658]
  We introduce a new framework for analyzing classification datasets based on the ratios of reconstruction errors between autoencoders trained on individual classes. This analysis framework enables efficient characterization of datasets on the sample, class, and entire dataset levels.
  arXiv  Detail & Related papers  (2024-12-03T17:29:00Z)
• Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performance [21.926934384262594]
  Large language models (LLMs) offer new opportunities to enhance the annotation process. We compare expert, crowd-sourced, and our LLM-based annotations in terms of agreement, label quality, and efficiency. Our findings reveal a substantial number of label errors, which, when corrected, induce a significant upward shift in reported model performance.
  arXiv  Detail & Related papers  (2024-10-24T16:27:03Z)
• SINDER: Repairing the Singular Defects of DINOv2 [61.98878352956125]
  Vision Transformer models trained on large-scale datasets often exhibit artifacts in the patch token they extract. We propose a novel fine-tuning smooth regularization that rectifies structural deficiencies using only a small dataset.
  arXiv  Detail & Related papers  (2024-07-23T20:34:23Z)
• Parameter-tuning-free data entry error unlearning with adaptive selective synaptic dampening [51.34904967046097]
  We introduce an extension to the selective synaptic dampening unlearning method that removes the need for parameter tuning. We demonstrate the performance of this extension, adaptive selective synaptic dampening (ASSD) on various ResNet18 and Vision Transformer unlearning tasks. The application of this approach is particularly compelling in industrial settings, such as supply chain management.
  arXiv  Detail & Related papers  (2024-02-06T14:04:31Z)
• CleanCoNLL: A Nearly Noise-Free Named Entity Recognition Dataset [3.3162484539136416]
  We present a comprehensive relabeling effort assisted by automatic consistency checking that corrects 7.0% of all labels in the English CoNLL-03. Our experimental evaluation finds not only that state-of-the-art approaches reach significantly higher F1-scores (97.1%) on our data, but crucially that the share of correct predictions falsely counted as errors due to annotation noise drops from 47% to 6%. This indicates that our resource is well suited to analyze the remaining errors made by state-of-the-art models, and that the theoretical upper bound even on high resource, coarse-grained NER is not yet
  arXiv  Detail & Related papers  (2023-10-24T22:34:43Z)
• Estimating label quality and errors in semantic segmentation data via any model [19.84626033109009]
  We study methods to score label quality, such that the images with the lowest scores are least likely to be correctly labeled. This helps prioritize what data to review in order to ensure a high-quality training/evaluation dataset.
  arXiv  Detail & Related papers  (2023-07-11T07:29:09Z)
• AQuA: A Benchmarking Tool for Label Quality Assessment [16.83510474053401]
  Recent studies have found datasets widely used to train and evaluate machine learning models to have pervasive labeling errors. We propose a benchmarking environment AQuA to rigorously evaluate methods that enable machine learning in the presence of label noise.
  arXiv  Detail & Related papers  (2023-06-15T19:42:11Z)
• Automated Labeling of German Chest X-Ray Radiology Reports using Deep Learning [50.591267188664666]
  We propose a deep learning-based CheXpert label prediction model, pre-trained on reports labeled by a rule-based German CheXpert model. Our results demonstrate the effectiveness of our approach, which significantly outperformed the rule-based model on all three tasks.
  arXiv  Detail & Related papers  (2023-06-09T16:08:35Z)
• A Confidence-based Partial Label Learning Model for Crowd-Annotated Named Entity Recognition [74.79785063365289]
  Existing models for named entity recognition (NER) are mainly based on large-scale labeled datasets. We propose a Confidence-based Partial Label Learning (CPLL) method to integrate the prior confidence (given by annotators) and posterior confidences (learned by models) for crowd-annotated NER.
  arXiv  Detail & Related papers  (2023-05-21T15:31:23Z)
• Investigation of Training Label Error Impact on RNN-T [8.470599402385302]
  We analyze impacts of different training label errors to RNN-T based ASR models. We suggest to design data pipelines for RNN-T with higher priority on reducing deletion label errors.
  arXiv  Detail & Related papers  (2021-12-01T08:57:39Z)
• Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training [66.80558875393565]
  We study the problem of training named entity recognition (NER) models using only distantly-labeled data. We propose a noise-robust learning scheme comprised of a new loss function and a noisy label removal step. Our method achieves superior performance, outperforming existing distantly-supervised NER models by significant margins.
  arXiv  Detail & Related papers  (2021-09-10T17:19:56Z)
• Temporal Calibrated Regularization for Robust Noisy Label Learning [60.90967240168525]
  Deep neural networks (DNNs) exhibit great success on many tasks with the help of large-scale well annotated datasets. However, labeling large-scale data can be very costly and error-prone so that it is difficult to guarantee the annotation quality. We propose a Temporal Calibrated Regularization (TCR) in which we utilize the original labels and the predictions in the previous epoch together.
  arXiv  Detail & Related papers  (2020-07-01T04:48:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.