Label Errors in the Tobacco3482 Dataset

• URL: http://arxiv.org/abs/2412.13140v1
• Date: Tue, 17 Dec 2024 18:06:28 GMT
• Title: Label Errors in the Tobacco3482 Dataset
• Abstract summary: Tobacco2 is a widely used document classification benchmark dataset.<n>We find that 11.7% of the dataset is improperly annotated and should either have an unknown label or a corrected label.<n>We then analyze the mistakes of a top-performing model and find that 35% of the model's mistakes can be directly attributed to these label issues.
• Score: 5.9848836847249185
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tobacco3482 is a widely used document classification benchmark dataset. However, our manual inspection of the entire dataset uncovers widespread ontological issues, especially large amounts of annotation label problems in the dataset. We establish data label guidelines and find that 11.7% of the dataset is improperly annotated and should either have an unknown label or a corrected label, and 16.7% of samples in the dataset have multiple valid labels. We then analyze the mistakes of a top-performing model and find that 35% of the model's mistakes can be directly attributed to these label issues, highlighting the inherent problems with using a noisily labeled dataset as a benchmark. Supplementary material, including dataset annotations and code, is available at https://github.com/gordon-lim/tobacco3482-mistakes/.

Related papers

• LabelMate: An LLM-Driven Framework for Refined Issue Report Labeling [0.3347876022070875]
  Software users often submit issue reports to a product's issue tracking system to report defects, suggest enhancements, or raise other product-related concerns.<n>Many issue reports remain unlabeled due to the substantial manual effort required to design an appropriate label taxonomy.<n>We propose LabelMate, a novel framework that derives a comprehensive, project-specific label set from historical issue reports.
  arXiv  Detail & Related papers  (2026-09-03T16:31:42Z)
• Learning to Detect Label Errors by Making Them: A Method for Segmentation and Object Detection Datasets [4.553529552666485]
  We present a unified method for detecting label errors in object detection, semantic segmentation, and instance segmentation datasets.<n>In our experiments, we compare the label error detection performance of our method with various baselines and state-of-the-art approaches of each task's domain.<n>We release 459 real label errors identified in the Cityscapes dataset and provide a benchmark for real label error detection in Cityscapes.
  arXiv  Detail & Related papers  (2025-08-25T11:59:56Z)
• FlatMatch: Bridging Labeled Data and Unlabeled Data with Cross-Sharpness for Semi-Supervised Learning [73.13448439554497]
  Semi-Supervised Learning (SSL) has been an effective way to leverage abundant unlabeled data with extremely scarce labeled data. Most SSL methods are commonly based on instance-wise consistency between different data transformations. We propose FlatMatch which minimizes a cross-sharpness measure to ensure consistent learning performance between the two datasets.
  arXiv  Detail & Related papers  (2023-10-25T06:57:59Z)
• Bridging the Gap between Model Explanations in Partially Annotated Multi-label Classification [85.76130799062379]
  We study how false negative labels affect the model's explanation. We propose to boost the attribution scores of the model trained with partial labels to make its explanation resemble that of the model trained with full labels.
  arXiv  Detail & Related papers  (2023-04-04T14:00:59Z)
• ScarceNet: Animal Pose Estimation with Scarce Annotations [74.48263583706712]
  ScarceNet is a pseudo label-based approach to generate artificial labels for the unlabeled images. We evaluate our approach on the challenging AP-10K dataset, where our approach outperforms existing semi-supervised approaches by a large margin.
  arXiv  Detail & Related papers  (2023-03-27T09:15:53Z)
• Identifying Label Errors in Object Detection Datasets by Loss Inspection [4.442111891959355]
  We introduce a benchmark for label error detection methods on object detection datasets. We simulate four different types of randomly introduced label errors on train and test sets of well-labeled object detection datasets.
  arXiv  Detail & Related papers  (2023-03-13T10:54:52Z)
• Identifying Incorrect Annotations in Multi-Label Classification Data [14.94741409713251]
  We consider algorithms for finding mislabeled examples in multi-label classification datasets. We propose an extension of the Confident Learning framework to this setting, as well as a label quality score that ranks examples with label errors much higher than those which are correctly labeled.
  arXiv  Detail & Related papers  (2022-11-25T05:03:56Z)
• Detecting Label Errors in Token Classification Data [22.539748563923123]
  We consider the task of finding sentences that contain label errors in token classification datasets. We study 11 different straightforward methods that score tokens/sentences based on the predicted class probabilities. We identify a simple and effective method that consistently detects those sentences containing label errors when applied with different token classification models.
  arXiv  Detail & Related papers  (2022-10-08T05:14:22Z)
• Learning with Noisy Labels by Targeted Relabeling [52.0329205268734]
  Crowdsourcing platforms are often used to collect datasets for training deep neural networks. We propose an approach which reserves a fraction of annotations to explicitly relabel highly probable labeling errors.
  arXiv  Detail & Related papers  (2021-10-15T20:37:29Z)
• Label-Assemble: Leveraging Multiple Datasets with Partial Labels [68.46767639240564]
  "Label-Assemble" aims to unleash the full potential of partial labels from an assembly of public datasets. We discovered that learning from negative examples facilitates both computer-aided disease diagnosis and detection.
  arXiv  Detail & Related papers  (2021-09-25T02:48:17Z)
• Instance Correction for Learning with Open-set Noisy Labels [145.06552420999986]
  We use the sample selection approach to handle open-set noisy labels. The discarded data are seen to be mislabeled and do not participate in training. We modify the instances of discarded data to make predictions for the discarded data consistent with given labels.
  arXiv  Detail & Related papers  (2021-06-01T13:05:55Z)
• Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks [12.992191397900806]
  We identify label errors in the test sets of 10 of the most commonly-used computer vision, natural language, and audio datasets. We estimate an average of 3.4% errors across the 10 datasets, where for example 2916 label errors comprise 6% of the ImageNet validation set.
  arXiv  Detail & Related papers  (2021-03-26T21:54:36Z)
• A Study on the Autoregressive and non-Autoregressive Multi-label Learning [77.11075863067131]
  We propose a self-attention based variational encoder-model to extract the label-label and label-feature dependencies jointly. Our model can therefore be used to predict all labels in parallel while still including both label-label and label-feature dependencies.
  arXiv  Detail & Related papers  (2020-12-03T05:41:44Z)
• openXDATA: A Tool for Multi-Target Data Generation and Missing Label Completion [23.14045574165086]
  A common problem in machine learning is to deal with datasets with disjoint label spaces and missing labels. In this work, we introduce the openXdata tool that completes the missing labels in partially labelled or unlabelled datasets. We show the ability to estimate both categories and continuous labels for all of the datasets, at rates that approached the ground truth values.
  arXiv  Detail & Related papers  (2020-07-27T22:05:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.