Measuring Robustness to Natural Distribution Shifts in Image Classification

• URL: http://arxiv.org/abs/2007.00644v2
• Date: Mon, 14 Sep 2020 09:55:13 GMT
• Title: Measuring Robustness to Natural Distribution Shifts in Image Classification
• Authors: Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, Ludwig Schmidt
• Abstract summary: We study how robust current ImageNet models are to distribution shifts arising from natural variations in datasets. We find that there is often little to no transfer of robustness from current synthetic to natural distribution shift. Our results indicate that distribution shifts arising in real data are currently an open research problem.
• Score: 67.96056447092428
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study how robust current ImageNet models are to distribution shifts arising from natural variations in datasets. Most research on robustness focuses on synthetic image perturbations (noise, simulated weather artifacts, adversarial examples, etc.), which leaves open how robustness on synthetic distribution shift relates to distribution shift arising in real data. Informed by an evaluation of 204 ImageNet models in 213 different test conditions, we find that there is often little to no transfer of robustness from current synthetic to natural distribution shift. Moreover, most current techniques provide no robustness to the natural distribution shifts in our testbed. The main exception is training on larger and more diverse datasets, which in multiple cases increases robustness, but is still far from closing the performance gaps. Our results indicate that distribution shifts arising in real data are currently an open research problem. We provide our testbed and data as a resource for future work at https://modestyachts.github.io/imagenet-testbed/ .

Related papers

• Temporal Test-Time Adaptation with State-Space Models [4.248760709042802]
  Adapting a model on test samples can help mitigate this drop in performance. Most test-time adaptation methods have focused on synthetic corruption shifts. We propose STAD, a probabilistic state-space model that adapts a deployed model to temporal distribution shifts.
  arXiv  Detail & Related papers  (2024-07-17T11:18:49Z)
• Certifying Model Accuracy under Distribution Shifts [151.67113334248464]
  We present provable robustness guarantees on the accuracy of a model under bounded Wasserstein shifts of the data distribution. We show that a simple procedure that randomizes the input of the model within a transformation space is provably robust to distributional shifts under the transformation.
  arXiv  Detail & Related papers  (2022-01-28T22:03:50Z)
• Robustness via Uncertainty-aware Cycle Consistency [44.34422859532988]
  Unpaired image-to-image translation refers to learning inter-image-domain mapping without corresponding image pairs. Existing methods learn deterministic mappings without explicitly modelling the robustness to outliers or predictive uncertainty. We propose a novel probabilistic method based on Uncertainty-aware Generalized Adaptive Cycle Consistency (UGAC)
  arXiv  Detail & Related papers  (2021-10-24T15:33:21Z)
• Predicting with Confidence on Unseen Distributions [90.68414180153897]
  We connect domain adaptation and predictive uncertainty literature to predict model accuracy on challenging unseen distributions. We find that the difference of confidences (DoC) of a classifier's predictions successfully estimates the classifier's performance change over a variety of shifts. We specifically investigate the distinction between synthetic and natural distribution shifts and observe that despite its simplicity DoC consistently outperforms other quantifications of distributional difference.
  arXiv  Detail & Related papers  (2021-07-07T15:50:18Z)
• Encoding Robustness to Image Style via Adversarial Feature Perturbations [72.81911076841408]
  We adapt adversarial training by directly perturbing feature statistics, rather than image pixels, to produce robust models. Our proposed method, Adversarial Batch Normalization (AdvBN), is a single network layer that generates worst-case feature perturbations during training.
  arXiv  Detail & Related papers  (2020-09-18T17:52:34Z)
• The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization [64.61630743818024]
  We introduce four new real-world distribution shift datasets consisting of changes in image style, image blurriness, geographic location, camera operation, and more. We find that using larger models and artificial data augmentations can improve robustness on real-world distribution shifts, contrary to claims in prior work. We also introduce a new data augmentation method which advances the state-of-the-art and outperforms models pretrained with 1000 times more labeled data.
  arXiv  Detail & Related papers  (2020-06-29T17:59:10Z)
• Understanding the Intrinsic Robustness of Image Distributions using Conditional Generative Models [87.00072607024026]
  We study the intrinsic robustness of two common image benchmarks under $ell$ perturbations. We show the existence of a large gap between the robustness limits implied by our theory and the adversarial robustness achieved by current state-of-the-art robust models.
  arXiv  Detail & Related papers  (2020-03-01T01:45:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.