dopanim: A Dataset of Doppelganger Animals with Noisy Annotations from Multiple Humans

• URL: http://arxiv.org/abs/2407.20950v1
• Date: Tue, 30 Jul 2024 16:27:51 GMT
• Title: dopanim: A Dataset of Doppelganger Animals with Noisy Annotations from Multiple Humans
• Authors: Marek Herde, Denis Huseljic, Lukas Rauch, Bernhard Sick,
• Abstract summary: We introduce a novel benchmark dataset, dopanim, consisting of about 15,750 animal images of 15 classes with ground truth labels. For approximately 10,500 of these images, 20 humans provided over 52,000 annotations with an accuracy of circa 67%. We benchmark well-known multi-annotator learning approaches using seven variants of this dataset and outline further evaluation use cases such as learning beyond hard class labels and active learning.
• Score: 1.99197168821625
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human annotators typically provide annotated data for training machine learning models, such as neural networks. Yet, human annotations are subject to noise, impairing generalization performances. Methodological research on approaches counteracting noisy annotations requires corresponding datasets for a meaningful empirical evaluation. Consequently, we introduce a novel benchmark dataset, dopanim, consisting of about 15,750 animal images of 15 classes with ground truth labels. For approximately 10,500 of these images, 20 humans provided over 52,000 annotations with an accuracy of circa 67%. Its key attributes include (1) the challenging task of classifying doppelganger animals, (2) human-estimated likelihoods as annotations, and (3) annotator metadata. We benchmark well-known multi-annotator learning approaches using seven variants of this dataset and outline further evaluation use cases such as learning beyond hard class labels and active learning. Our dataset and a comprehensive codebase are publicly available to emulate the data collection process and to reproduce all empirical results.

Related papers

• The Re-Label Method For Data-Centric Machine Learning [0.24475591916185496]
  In industry deep learning application, our manually labeled data has a certain number of noisy data. We present a simple method to find the noisy data and re-label the noisy data by human, given the model predictions as references in human labeling.
  arXiv  Detail & Related papers  (2023-02-09T01:09:57Z)
• Improving Named Entity Recognition in Telephone Conversations via Effective Active Learning with Human in the Loop [2.1004132913758267]
  We present an active learning framework that leverages human in the loop learning to identify data samples from the annotated dataset for re-annotation. By re-annotating only about 6% training instances out of the whole dataset, the F1 score for a certain entity type can be significantly improved by about 25%.
  arXiv  Detail & Related papers  (2022-11-02T17:55:04Z)
• MABe22: A Multi-Species Multi-Task Benchmark for Learned Representations of Behavior [28.878568752724235]
  We introduce MABe22, a benchmark to assess the quality of learned behavior representations. This dataset is collected from a variety of biology experiments. We test self-supervised video and trajectory representation learning methods to demonstrate the use of our benchmark.
  arXiv  Detail & Related papers  (2022-07-21T15:51:30Z)
• Learning with Neighbor Consistency for Noisy Labels [69.83857578836769]
  We present a method for learning from noisy labels that leverages similarities between training examples in feature space. We evaluate our method on datasets evaluating both synthetic (CIFAR-10, CIFAR-100) and realistic (mini-WebVision, Clothing1M, mini-ImageNet-Red) noise.
  arXiv  Detail & Related papers  (2022-02-04T15:46:27Z)
• NoisyActions2M: A Multimedia Dataset for Video Understanding from Noisy Labels [33.659146748289444]
  We create a benchmark dataset consisting of around 2 million videos with associated user-generated annotations and other meta information. We show how a network pretrained on the proposed dataset can help against video corruption and label noise in downstream datasets.
  arXiv  Detail & Related papers  (2021-10-13T16:12:18Z)
• Combining Feature and Instance Attribution to Detect Artifacts [62.63504976810927]
  We propose methods to facilitate identification of training data artifacts. We show that this proposed training-feature attribution approach can be used to uncover artifacts in training data. We execute a small user study to evaluate whether these methods are useful to NLP researchers in practice.
  arXiv  Detail & Related papers  (2021-07-01T09:26:13Z)
• Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets [90.61266099147053]
  We investigate efficient annotation strategies for collecting multi-class classification labels for a large collection of images. We propose modifications and best practices aimed at minimizing human labeling effort. Simulated experiments on a 125k image subset of the ImageNet100 show that it can be annotated to 80% top-1 accuracy with 0.35 annotations per image on average.
  arXiv  Detail & Related papers  (2021-04-26T16:29:32Z)
• Visual Distant Supervision for Scene Graph Generation [66.10579690929623]
  Scene graph models usually require supervised learning on large quantities of labeled data with intensive human annotation. We propose visual distant supervision, a novel paradigm of visual relation learning, which can train scene graph models without any human-labeled data. Comprehensive experimental results show that our distantly supervised model outperforms strong weakly supervised and semi-supervised baselines.
  arXiv  Detail & Related papers  (2021-03-29T06:35:24Z)
• Semi-Automatic Data Annotation guided by Feature Space Projection [117.9296191012968]
  We present a semi-automatic data annotation approach based on suitable feature space projection and semi-supervised label estimation. We validate our method on the popular MNIST dataset and on images of human intestinal parasites with and without fecal impurities. Our results demonstrate the added-value of visual analytics tools that combine complementary abilities of humans and machines for more effective machine learning.
  arXiv  Detail & Related papers  (2020-07-27T17:03:50Z)
• Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation [57.68890534164427]
  In this work, we ask if we may leverage semi-supervised learning in unlabeled video sequences and extra images to improve the performance on urban scene segmentation. We simply predict pseudo-labels for the unlabeled data and train subsequent models with both human-annotated and pseudo-labeled data. Our Naive-Student model, trained with such simple yet effective iterative semi-supervised learning, attains state-of-the-art results at all three Cityscapes benchmarks.
  arXiv  Detail & Related papers  (2020-05-20T18:00:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.