Flaws of ImageNet, Computer Vision's Favourite Dataset

• URL: http://arxiv.org/abs/2412.00076v1
• Date: Tue, 26 Nov 2024 16:47:59 GMT
• Title: Flaws of ImageNet, Computer Vision's Favourite Dataset
• Authors: Nikita Kisel, Illia Volkov, Katerina Hanzelkova, Klara Janouskova, Jiri Matas,
• Abstract summary: ImageNet-1k dataset has become a gold standard for evaluating model performance. In this blog post, we analyze the issues in the ImageNet-1k dataset.
• Score: 18.700895407332787
• License:
• Abstract: Since its release, ImageNet-1k dataset has become a gold standard for evaluating model performance. It has served as the foundation for numerous other datasets and training tasks in computer vision. As models have improved in accuracy, issues related to label correctness have become increasingly apparent. In this blog post, we analyze the issues in the ImageNet-1k dataset, including incorrect labels, overlapping or ambiguous class definitions, training-evaluation domain shifts, and image duplicates. The solutions for some problems are straightforward. For others, we hope to start a broader conversation about refining this influential dataset to better serve future research.

Related papers

• Toffee: Efficient Million-Scale Dataset Construction for Subject-Driven Text-to-Image Generation [58.09421301921607]
  We construct the first large-scale dataset for subject-driven image editing and generation. Our dataset is 5 times the size of previous largest dataset, yet our cost is tens of thousands of GPU hours lower.
  arXiv  Detail & Related papers  (2024-06-13T16:40:39Z)
• ImageNot: A contrast with ImageNet preserves model rankings [16.169858780154893]
  We introduce ImageNot, a dataset designed to match the scale of ImageNet while differing drastically in other aspects. Key model architectures developed for ImageNet over the years rank identically when trained and evaluated on ImageNot to how they rank on ImageNet.
  arXiv  Detail & Related papers  (2024-04-02T17:13:04Z)
• Evaluating Data Attribution for Text-to-Image Models [62.844382063780365]
  We evaluate attribution through "customization" methods, which tune an existing large-scale model toward a given exemplar object or style. Our key insight is that this allows us to efficiently create synthetic images that are computationally influenced by the exemplar by construction. By taking into account the inherent uncertainty of the problem, we can assign soft attribution scores over a set of training images.
  arXiv  Detail & Related papers  (2023-06-15T17:59:51Z)
• Diverse, Difficult, and Odd Instances (D2O): A New Test Set for Object Classification [47.64219291655723]
  We introduce a new test set, called D2O, which is sufficiently different from existing test sets. Our dataset contains 8,060 images spread across 36 categories, out of which 29 appear in ImageNet. The best Top-1 accuracy on our dataset is around 60% which is much lower than 91% best Top-1 accuracy on ImageNet.
  arXiv  Detail & Related papers  (2023-01-29T19:58:32Z)
• Does progress on ImageNet transfer to real-world datasets? [28.918770106968843]
  We evaluate ImageNet pre-trained models with varying accuracy on six practical image classification datasets. On multiple datasets, models with higher ImageNet accuracy do not consistently yield performance improvements. We hope that future benchmarks will include more diverse datasets to encourage a more comprehensive approach to improving learning algorithms.
  arXiv  Detail & Related papers  (2023-01-11T18:55:53Z)
• ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotations [36.348968311668564]
  We introduce ImageNet-X, a set of sixteen human annotations of factors such as pose, background, or lighting. We investigate 2,200 current recognition models and study the types of mistakes as a function of model's architecture. We find models have consistent failure modes across ImageNet-X categories.
  arXiv  Detail & Related papers  (2022-11-03T14:56:32Z)
• BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations [89.42397034542189]
  We synthesize a large labeled dataset via a generative adversarial network (GAN) We take image samples from the class-conditional generative model BigGAN trained on ImageNet, and manually annotate 5 images per class, for all 1k classes. We create a new ImageNet benchmark by labeling an additional set of 8k real images and evaluate segmentation performance in a variety of settings.
  arXiv  Detail & Related papers  (2022-01-12T20:28:34Z)
• Automated Cleanup of the ImageNet Dataset by Model Consensus, Explainability and Confident Learning [0.0]
  ImageNet was the backbone of various convolutional neural networks (CNNs) trained on ILSVRC12Net. This paper describes automated applications based on model consensus, explainability and confident learning to correct labeling mistakes. The ImageNet-Clean improves the model performance by 2-2.4 % for SqueezeNet and EfficientNet-B0 models.
  arXiv  Detail & Related papers  (2021-03-30T13:16:35Z)
• Contemplating real-world object classification [53.10151901863263]
  We reanalyze the ObjectNet dataset recently proposed by Barbu et al. containing objects in daily life situations. We find that applying deep models to the isolated objects, rather than the entire scene as is done in the original paper, results in around 20-30% performance improvement.
  arXiv  Detail & Related papers  (2021-03-08T23:29:59Z)
• From ImageNet to Image Classification: Contextualizing Progress on Benchmarks [99.19183528305598]
  We study how specific design choices in the ImageNet creation process impact the fidelity of the resulting dataset. Our analysis pinpoints how a noisy data collection pipeline can lead to a systematic misalignment between the resulting benchmark and the real-world task it serves as a proxy for.
  arXiv  Detail & Related papers  (2020-05-22T17:39:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.