Noise or Signal: The Role of Image Backgrounds in Object Recognition

• URL: http://arxiv.org/abs/2006.09994v1
• Date: Wed, 17 Jun 2020 16:54:43 GMT
• Title: Noise or Signal: The Role of Image Backgrounds in Object Recognition
• Authors: Kai Xiao and Logan Engstrom and Andrew Ilyas and Aleksander Madry
• Abstract summary: We create a toolkit for disentangling foreground and background signal on ImageNet images. We find that (a) models can achieve non-trivial accuracy by relying on the background alone, (b) models often misclassify images even in the presence of correctly classified foregrounds.
• Score: 93.55720207356603
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We assess the tendency of state-of-the-art object recognition models to depend on signals from image backgrounds. We create a toolkit for disentangling foreground and background signal on ImageNet images, and find that (a) models can achieve non-trivial accuracy by relying on the background alone, (b) models often misclassify images even in the presence of correctly classified foregrounds--up to 87.5% of the time with adversarially chosen backgrounds, and (c) more accurate models tend to depend on backgrounds less. Our analysis of backgrounds brings us closer to understanding which correlations machine learning models use, and how they determine models' out of distribution performance.

Related papers

• Reinforcing Pre-trained Models Using Counterfactual Images [54.26310919385808]
  This paper proposes a novel framework to reinforce classification models using language-guided generated counterfactual images. We identify model weaknesses by testing the model using the counterfactual image dataset. We employ the counterfactual images as an augmented dataset to fine-tune and reinforce the classification model.
  arXiv  Detail & Related papers  (2024-06-19T08:07:14Z)
• Invariant Learning via Diffusion Dreamed Distribution Shifts [121.71383835729848]
  We propose a dataset called Diffusion Dreamed Distribution Shifts (D3S) D3S consists of synthetic images generated through StableDiffusion using text prompts and image guides obtained by pasting a sample foreground image onto a background template image. Due to the incredible photorealism of the diffusion model, our images are much closer to natural images than previous synthetic datasets.
  arXiv  Detail & Related papers  (2022-11-18T17:07:43Z)
• Optimizing Relevance Maps of Vision Transformers Improves Robustness [91.61353418331244]
  It has been observed that visual classification models often rely mostly on the image background, neglecting the foreground, which hurts their robustness to distribution changes. We propose to monitor the model's relevancy signal and manipulate it such that the model is focused on the foreground object. This is done as a finetuning step, involving relatively few samples consisting of pairs of images and their associated foreground masks.
  arXiv  Detail & Related papers  (2022-06-02T17:24:48Z)
• A Comprehensive Study of Image Classification Model Sensitivity to Foregrounds, Backgrounds, and Visual Attributes [58.633364000258645]
  We call this dataset RIVAL10 consisting of roughly $26k$ instances over $10$ classes. We evaluate the sensitivity of a broad set of models to noise corruptions in foregrounds, backgrounds and attributes. In our analysis, we consider diverse state-of-the-art architectures (ResNets, Transformers) and training procedures (CLIP, SimCLR, DeiT, Adversarial Training)
  arXiv  Detail & Related papers  (2022-01-26T06:31:28Z)
• Myope Models -- Are face presentation attack detection models short-sighted? [3.4376560669160394]
  Presentation attacks are recurrent threats to biometric systems, where impostors attempt to bypass these systems. This work presents a comparative study of face presentation attack detection (PAD) models with and without crops. The results show that the performance is consistently better when the background is present in the images.
  arXiv  Detail & Related papers  (2021-11-22T11:28:44Z)
• Representation, Analysis of Bayesian Refinement Approximation Network: A Survey [4.60479555961894]
  In this paper, we focus on using a modified U-Net model to approximate the result of the Bayesian refinement method. In our modified U-Net model, the result of background subtraction from other models will be combined with the source image as input for learning the statistical distribution.
  arXiv  Detail & Related papers  (2021-03-27T12:55:09Z)
• Rethinking Natural Adversarial Examples for Classification Models [43.87819913022369]
  ImageNet-A is a famous dataset of natural adversarial examples. We validated the hypothesis by reducing the background influence in ImageNet-A examples with object detection techniques. Experiments showed that the object detection models with various classification models as backbones obtained much higher accuracy than their corresponding classification models.
  arXiv  Detail & Related papers  (2021-02-23T14:46:48Z)
• An application of a pseudo-parabolic modeling to texture image recognition [0.0]
  We present a novel methodology for texture image recognition using a partial differential equation modeling. We employ the pseudo-parabolic Buckley-Leverett equation to provide a dynamics to the digital image representation and collect local descriptors from those images evolving in time.
  arXiv  Detail & Related papers  (2021-02-09T18:08:42Z)
• Stereopagnosia: Fooling Stereo Networks with Adversarial Perturbations [71.00754846434744]
  We show that imperceptible additive perturbations can significantly alter the disparity map. We show that, when used for adversarial data augmentation, our perturbations result in trained models that are more robust.
  arXiv  Detail & Related papers  (2020-09-21T19:20:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.