Bugs in the Data: How ImageNet Misrepresents Biodiversity
- URL: http://arxiv.org/abs/2208.11695v1
- Date: Wed, 24 Aug 2022 17:55:48 GMT
- Title: Bugs in the Data: How ImageNet Misrepresents Biodiversity
- Authors: Alexandra Sasha Luccioni and David Rolnick
- Abstract summary: We analyze the 13,450 images from 269 classes that represent wild animals in the ImageNet-1k validation set.
We find that many of the classes are ill-defined or overlapping, and that 12% of the images are incorrectly labeled.
We also find that both the wildlife-related labels and images included in ImageNet-1k present significant geographical and cultural biases.
- Score: 98.98950914663813
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: ImageNet-1k is a dataset often used for benchmarking machine learning (ML)
models and evaluating tasks such as image recognition and object detection.
Wild animals make up 27% of ImageNet-1k but, unlike classes representing people
and objects, these data have not been closely scrutinized. In the current
paper, we analyze the 13,450 images from 269 classes that represent wild
animals in the ImageNet-1k validation set, with the participation of expert
ecologists. We find that many of the classes are ill-defined or overlapping,
and that 12% of the images are incorrectly labeled, with some classes having
>90% of images incorrect. We also find that both the wildlife-related labels
and images included in ImageNet-1k present significant geographical and
cultural biases, as well as ambiguities such as artificial animals, multiple
species in the same image, or the presence of humans. Our findings highlight
serious issues with the extensive use of this dataset for evaluating ML
systems, the use of such algorithms in wildlife-related tasks, and more broadly
the ways in which ML datasets are commonly created and curated.
Related papers
- SPOTS-10: Animal Pattern Benchmark Dataset for Machine Learning Algorithms [0.08158530638728499]
SPOTS-10 is an extensive collection of grayscale images showcasing diverse patterns in ten animal species.
This dataset is a resource for evaluating machine learning algorithms in situ.
The training set comprises 40,000 images, while the test set contains 10,000 images.
arXiv Detail & Related papers (2024-10-28T14:00:02Z) - Diverse, Difficult, and Odd Instances (D2O): A New Test Set for Object
Classification [47.64219291655723]
We introduce a new test set, called D2O, which is sufficiently different from existing test sets.
Our dataset contains 8,060 images spread across 36 categories, out of which 29 appear in ImageNet.
The best Top-1 accuracy on our dataset is around 60% which is much lower than 91% best Top-1 accuracy on ImageNet.
arXiv Detail & Related papers (2023-01-29T19:58:32Z) - DEArt: Dataset of European Art [0.17188280334580194]
We propose DEArt, an object detection and pose classification dataset meant to be a reference for paintings between the XIIth and the XVIIIth centuries.
It contains more than 15000 images, about 80% non-iconic, aligned with manual annotations for the bounding boxes identifying all instances of 69 classes as well as 12 possible poses for boxes identifying human-like objects.
Our results show that object detectors for the cultural heritage domain can achieve a level of precision comparable to state-of-art models for generic images via transfer learning.
arXiv Detail & Related papers (2022-11-02T16:05:35Z) - VizWiz-FewShot: Locating Objects in Images Taken by People With Visual
Impairments [74.72656607288185]
We introduce a few-shot localization dataset originating from photographers who authentically were trying to learn about the visual content in the images they took.
It includes nearly 10,000 segmentations of 100 categories in over 4,500 images that were taken by people with visual impairments.
Compared to existing few-shot object detection and instance segmentation datasets, our dataset is the first to locate holes in objects.
arXiv Detail & Related papers (2022-07-24T20:44:51Z) - Oracle-MNIST: a Realistic Image Dataset for Benchmarking Machine
Learning Algorithms [57.29464116557734]
We introduce the Oracle-MNIST dataset, comprising of 28$times $28 grayscale images of 30,222 ancient characters.
The training set totally consists of 27,222 images, and the test set contains 300 images per class.
arXiv Detail & Related papers (2022-05-19T09:57:45Z) - Correlated Input-Dependent Label Noise in Large-Scale Image
Classification [4.979361059762468]
We take a principled probabilistic approach to modelling input-dependent, also known as heteroscedastic, label noise in datasets.
We demonstrate that the learned covariance structure captures known sources of label noise between semantically similar and co-occurring classes.
We set a new state-of-the-art result on WebVision 1.0 with 76.6% top-1 accuracy.
arXiv Detail & Related papers (2021-05-19T17:30:59Z) - Towards Good Practices for Efficiently Annotating Large-Scale Image
Classification Datasets [90.61266099147053]
We investigate efficient annotation strategies for collecting multi-class classification labels for a large collection of images.
We propose modifications and best practices aimed at minimizing human labeling effort.
Simulated experiments on a 125k image subset of the ImageNet100 show that it can be annotated to 80% top-1 accuracy with 0.35 annotations per image on average.
arXiv Detail & Related papers (2021-04-26T16:29:32Z) - How many images do I need? Understanding how sample size per class
affects deep learning model performance metrics for balanced designs in
autonomous wildlife monitoring [0.0]
We explore in depth the issues of deep learning model performance for progressively increasing per class (species) sample sizes.
We provide ecologists with an approximation formula to estimate how many images per animal species they need for certain accuracy level a priori.
arXiv Detail & Related papers (2020-10-16T06:28:35Z) - MLRSNet: A Multi-label High Spatial Resolution Remote Sensing Dataset
for Semantic Scene Understanding [6.880271407391406]
We construct a multi-label high spatial resolution remote sensing dataset named MLRSNet for semantic scene understanding with deep learning.
MLRSNet contains 109,161 samples within 46 scene categories, and each image has at least one of 60 predefined labels.
The experimental results demonstrate that MLRSNet is a significant benchmark for future research.
arXiv Detail & Related papers (2020-10-01T08:03:47Z) - Background Splitting: Finding Rare Classes in a Sea of Background [55.03789745276442]
We focus on the real-world problem of training accurate deep models for image classification of a small number of rare categories.
In these scenarios, almost all images belong to the background category in the dataset (>95% of the dataset is background)
We demonstrate that both standard fine-tuning approaches and state-of-the-art approaches for training on imbalanced datasets do not produce accurate deep models in the presence of this extreme imbalance.
arXiv Detail & Related papers (2020-08-28T23:05:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.