Does progress on ImageNet transfer to real-world datasets?

• URL: http://arxiv.org/abs/2301.04644v1
• Date: Wed, 11 Jan 2023 18:55:53 GMT
• Title: Does progress on ImageNet transfer to real-world datasets?
• Authors: Alex Fang and Simon Kornblith and Ludwig Schmidt
• Abstract summary: We evaluate ImageNet pre-trained models with varying accuracy on six practical image classification datasets. On multiple datasets, models with higher ImageNet accuracy do not consistently yield performance improvements. We hope that future benchmarks will include more diverse datasets to encourage a more comprehensive approach to improving learning algorithms.
• Score: 28.918770106968843
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Does progress on ImageNet transfer to real-world datasets? We investigate this question by evaluating ImageNet pre-trained models with varying accuracy (57% - 83%) on six practical image classification datasets. In particular, we study datasets collected with the goal of solving real-world tasks (e.g., classifying images from camera traps or satellites), as opposed to web-scraped benchmarks collected for comparing models. On multiple datasets, models with higher ImageNet accuracy do not consistently yield performance improvements. For certain tasks, interventions such as data augmentation improve performance even when architectures do not. We hope that future benchmarks will include more diverse datasets to encourage a more comprehensive approach to improving learning algorithms.

Related papers

• Rethinking Image Super-Resolution from Training Data Perspectives [54.28824316574355]
  We investigate the understudied effect of the training data used for image super-resolution (SR) With this, we propose an automated image evaluation pipeline. We find that datasets with (i) low compression artifacts, (ii) high within-image diversity as judged by the number of different objects, and (iii) a large number of images from ImageNet or PASS all positively affect SR performance.
  arXiv  Detail & Related papers  (2024-09-01T16:25:04Z)
• ImageNot: A contrast with ImageNet preserves model rankings [16.169858780154893]
  We introduce ImageNot, a dataset designed to match the scale of ImageNet while differing drastically in other aspects. Key model architectures developed for ImageNet over the years rank identically when trained and evaluated on ImageNot to how they rank on ImageNet.
  arXiv  Detail & Related papers  (2024-04-02T17:13:04Z)
• Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement [68.44100784364987]
  We propose a strategy to improve a dataset once such that the accuracy of any model architecture trained on the reinforced dataset is improved at no additional training cost for users. We create a reinforced version of the ImageNet training dataset, called ImageNet+, as well as reinforced datasets CIFAR-100+, Flowers-102+, and Food-101+. Models trained with ImageNet+ are more accurate, robust, and calibrated, and transfer well to downstream tasks.
  arXiv  Detail & Related papers  (2023-03-15T23:10:17Z)
• BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations [89.42397034542189]
  We synthesize a large labeled dataset via a generative adversarial network (GAN) We take image samples from the class-conditional generative model BigGAN trained on ImageNet, and manually annotate 5 images per class, for all 1k classes. We create a new ImageNet benchmark by labeling an additional set of 8k real images and evaluate segmentation performance in a variety of settings.
  arXiv  Detail & Related papers  (2022-01-12T20:28:34Z)
• Spoken ObjectNet: A Bias-Controlled Spoken Caption Dataset [14.44921491933053]
  Spoken ObjectNet is designed to remove some of these biases and provide a way to better evaluate how effectively models will perform in real-world scenarios. This dataset expands upon ObjectNet, which is a bias-controlled image dataset that features similar image classes to those present in ImageNet. Results show that models trained on other datasets and then evaluated on Spoken ObjectNet tend to perform poorly due to biases in other datasets that the models have learned.
  arXiv  Detail & Related papers  (2021-10-14T17:38:20Z)
• Contrastive Learning with Stronger Augmentations [63.42057690741711]
  We propose a general framework called Contrastive Learning with Stronger Augmentations(A) to complement current contrastive learning approaches. Here, the distribution divergence between the weakly and strongly augmented images over the representation bank is adopted to supervise the retrieval of strongly augmented queries. Experiments showed the information from the strongly augmented images can significantly boost the performance.
  arXiv  Detail & Related papers  (2021-04-15T18:40:04Z)
• Is it Enough to Optimize CNN Architectures on ImageNet? [0.0]
  We train 500 CNN architectures on ImageNet and 8 other image classification datasets. The relationship between architecture and performance varies wildly, depending on the datasets. We identify two dataset-specific performance indicators: the cumulative width across layers as well as the total depth of the network.
  arXiv  Detail & Related papers  (2021-03-16T14:42:01Z)
• Contemplating real-world object classification [53.10151901863263]
  We reanalyze the ObjectNet dataset recently proposed by Barbu et al. containing objects in daily life situations. We find that applying deep models to the isolated objects, rather than the entire scene as is done in the original paper, results in around 20-30% performance improvement.
  arXiv  Detail & Related papers  (2021-03-08T23:29:59Z)
• From ImageNet to Image Classification: Contextualizing Progress on Benchmarks [99.19183528305598]
  We study how specific design choices in the ImageNet creation process impact the fidelity of the resulting dataset. Our analysis pinpoints how a noisy data collection pipeline can lead to a systematic misalignment between the resulting benchmark and the real-world task it serves as a proxy for.
  arXiv  Detail & Related papers  (2020-05-22T17:39:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.