The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization

• URL: http://arxiv.org/abs/2006.16241v3
• Date: Sat, 24 Jul 2021 04:28:58 GMT
• Title: The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
• Authors: Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, Justin Gilmer
• Abstract summary: We introduce four new real-world distribution shift datasets consisting of changes in image style, image blurriness, geographic location, camera operation, and more. We find that using larger models and artificial data augmentations can improve robustness on real-world distribution shifts, contrary to claims in prior work. We also introduce a new data augmentation method which advances the state-of-the-art and outperforms models pretrained with 1000 times more labeled data.
• Score: 64.61630743818024
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce four new real-world distribution shift datasets consisting of changes in image style, image blurriness, geographic location, camera operation, and more. With our new datasets, we take stock of previously proposed methods for improving out-of-distribution robustness and put them to the test. We find that using larger models and artificial data augmentations can improve robustness on real-world distribution shifts, contrary to claims in prior work. We find improvements in artificial robustness benchmarks can transfer to real-world distribution shifts, contrary to claims in prior work. Motivated by our observation that data augmentations can help with real-world distribution shifts, we also introduce a new data augmentation method which advances the state-of-the-art and outperforms models pretrained with 1000 times more labeled data. Overall we find that some methods consistently help with distribution shifts in texture and local image statistics, but these methods do not help with some other distribution shifts like geographic changes. Our results show that future research must study multiple distribution shifts simultaneously, as we demonstrate that no evaluated method consistently improves robustness.

Related papers

• Control+Shift: Generating Controllable Distribution Shifts [1.3060023718506917]
  We propose a new method for generating realistic datasets with distribution shifts using any decoder-based generative model. Our approach systematically creates datasets with varying intensities of distribution shifts, facilitating a comprehensive analysis of model performance degradation.
  arXiv  Detail & Related papers  (2024-09-12T11:07:53Z)
• Distribution-Aware Data Expansion with Diffusion Models [55.979857976023695]
  We propose DistDiff, a training-free data expansion framework based on the distribution-aware diffusion model. DistDiff consistently enhances accuracy across a diverse range of datasets compared to models trained solely on original data.
  arXiv  Detail & Related papers  (2024-03-11T14:07:53Z)
• Consistency Regularization for Generalizable Source-free Domain Adaptation [62.654883736925456]
  Source-free domain adaptation (SFDA) aims to adapt a well-trained source model to an unlabelled target domain without accessing the source dataset. Existing SFDA methods ONLY assess their adapted models on the target training set, neglecting the data from unseen but identically distributed testing sets. We propose a consistency regularization framework to develop a more generalizable SFDA method.
  arXiv  Detail & Related papers  (2023-08-03T07:45:53Z)
• Non-adversarial Robustness of Deep Learning Methods for Computer Vision [0.0]
  Non-adversarial robustness, also known as natural robustness, is a property of deep learning models. We present a brief overview of the most recent techniques for improving the robustness of computer vision methods.
  arXiv  Detail & Related papers  (2023-05-24T10:21:31Z)
• An Empirical Study on Distribution Shift Robustness From the Perspective of Pre-Training and Data Augmentation [91.62129090006745]
  This paper studies the distribution shift problem from the perspective of pre-training and data augmentation. We provide the first comprehensive empirical study focusing on pre-training and data augmentation.
  arXiv  Detail & Related papers  (2022-05-25T13:04:53Z)
• A Fine-Grained Analysis on Distribution Shift [24.084676204709723]
  We introduce a framework that enables fine-grained analysis of various distribution shifts. We evaluate 19 distinct methods grouped into five categories across both synthetic and real-world datasets. Our framework can be easily extended to include new methods, shifts, and datasets.
  arXiv  Detail & Related papers  (2021-10-21T17:57:08Z)
• WILDS: A Benchmark of in-the-Wild Distribution Shifts [157.53410583509924]
  Distribution shifts can substantially degrade the accuracy of machine learning systems deployed in the wild. We present WILDS, a curated collection of 8 benchmark datasets that reflect a diverse range of distribution shifts. We show that standard training results in substantially lower out-of-distribution than in-distribution performance.
  arXiv  Detail & Related papers  (2020-12-14T11:14:56Z)
• On Robustness and Transferability of Convolutional Neural Networks [147.71743081671508]
  Modern deep convolutional networks (CNNs) are often criticized for not generalizing under distributional shifts. We study the interplay between out-of-distribution and transfer performance of modern image classification CNNs for the first time. We find that increasing both the training set and model sizes significantly improve the distributional shift robustness.
  arXiv  Detail & Related papers  (2020-07-16T18:39:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.