ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Object

• URL: http://arxiv.org/abs/2403.18775v1
• Date: Wed, 27 Mar 2024 17:23:39 GMT
• Title: ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Object
• Authors: Chenshuang Zhang, Fei Pan, Junmo Kim, In So Kweon, Chengzhi Mao,
• Abstract summary: We introduce generative model as a data source for hard images that benchmark deep models' robustness. We are able to generate images with more diversified backgrounds, textures, and materials than any prior work, where we term this benchmark as ImageNet-D. Our work suggests that diffusion models can be an effective source to test vision models.
• Score: 78.58860252442045
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We establish rigorous benchmarks for visual perception robustness. Synthetic images such as ImageNet-C, ImageNet-9, and Stylized ImageNet provide specific type of evaluation over synthetic corruptions, backgrounds, and textures, yet those robustness benchmarks are restricted in specified variations and have low synthetic quality. In this work, we introduce generative model as a data source for synthesizing hard images that benchmark deep models' robustness. Leveraging diffusion models, we are able to generate images with more diversified backgrounds, textures, and materials than any prior work, where we term this benchmark as ImageNet-D. Experimental results show that ImageNet-D results in a significant accuracy drop to a range of vision models, from the standard ResNet visual classifier to the latest foundation models like CLIP and MiniGPT-4, significantly reducing their accuracy by up to 60\%. Our work suggests that diffusion models can be an effective source to test vision models. The code and dataset are available at https://github.com/chenshuang-zhang/imagenet_d.

Related papers

• ImagiNet: A Multi-Content Dataset for Generalizable Synthetic Image Detection via Contrastive Learning [0.0]
  Generative models produce images with a level of authenticity nearly indistinguishable from real photos and artwork. The difficulty of identifying synthetic images leaves online media platforms vulnerable to impersonation and misinformation attempts. We introduce ImagiNet, a high-resolution and balanced dataset for synthetic image detection.
  arXiv  Detail & Related papers  (2024-07-29T13:57:24Z)
• ImageNet-E: Benchmarking Neural Network Robustness via Attribute Editing [45.14977000707886]
  Higher accuracy on ImageNet usually leads to better robustness against different corruptions. We create a toolkit for object editing with controls of backgrounds, sizes, positions, and directions. We evaluate the performance of current deep learning models, including both convolutional neural networks and vision transformers.
  arXiv  Detail & Related papers  (2023-03-30T02:02:32Z)
• Fake it till you make it: Learning transferable representations from synthetic ImageNet clones [30.264601433216246]
  We show that ImageNet clones can close a large part of the gap between models produced by synthetic images and models trained with real images. We show that models trained on synthetic images exhibit strong generalization properties and perform on par with models trained on real data for transfer.
  arXiv  Detail & Related papers  (2022-12-16T11:44:01Z)
• Lafite2: Few-shot Text-to-Image Generation [132.14211027057766]
  We propose a novel method for pre-training text-to-image generation model on image-only datasets. It considers a retrieval-then-optimization procedure to synthesize pseudo text features. It can be beneficial to a wide range of settings, including the few-shot, semi-supervised and fully-supervised learning.
  arXiv  Detail & Related papers  (2022-10-25T16:22:23Z)
• Is synthetic data from generative models ready for image recognition? [69.42645602062024]
  We study whether and how synthetic images generated from state-of-the-art text-to-image generation models can be used for image recognition tasks. We showcase the powerfulness and shortcomings of synthetic data from existing generative models, and propose strategies for better applying synthetic data for recognition tasks.
  arXiv  Detail & Related papers  (2022-10-14T06:54:24Z)
• Towards Robustness of Neural Networks [0.0]
  We introduce ImageNet-A/O and ImageNet-R as well as a synthetic environment and testing suite we called CAOS. All of the datasets were created for testing robustness and measuring progress in robustness. We build off of simple baselines in the form of Maximum Logit, and Typicality Score as well as create a novel data augmentation method in the form of DeepAugment.
  arXiv  Detail & Related papers  (2021-12-30T19:41:10Z)
• Contemplating real-world object classification [53.10151901863263]
  We reanalyze the ObjectNet dataset recently proposed by Barbu et al. containing objects in daily life situations. We find that applying deep models to the isolated objects, rather than the entire scene as is done in the original paper, results in around 20-30% performance improvement.
  arXiv  Detail & Related papers  (2021-03-08T23:29:59Z)
• Shape-Texture Debiased Neural Network Training [50.6178024087048]
  Convolutional Neural Networks are often biased towards either texture or shape, depending on the training dataset. We develop an algorithm for shape-texture debiased learning. Experiments show that our method successfully improves model performance on several image recognition benchmarks.
  arXiv  Detail & Related papers  (2020-10-12T19:16:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.