Causal ImageNet: How to discover spurious features in Deep Learning?

• URL: http://arxiv.org/abs/2110.04301v1
• Date: Fri, 8 Oct 2021 15:55:05 GMT
• Title: Causal ImageNet: How to discover spurious features in Deep Learning?
• Authors: Sahil Singla, Soheil Feizi
• Abstract summary: A key reason for the lack of reliability of deep neural networks in the real world is their heavy reliance on it spurious input features that are causally unrelated to the true label. In this work, we introduce a it scalable framework to discover a subset of spurious and causal visual attributes used in inferences of a general model. We show that these neural feature annotations it generalize extremely well to many more images it without any human supervision.
• Score: 44.053067014796596
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A key reason for the lack of reliability of deep neural networks in the real world is their heavy reliance on {\it spurious} input features that are causally unrelated to the true label. Focusing on image classifications, we define causal attributes as the set of visual features that are always a part of the object while spurious attributes are the ones that are likely to {\it co-occur} with the object but not a part of it (e.g., attribute ``fingers" for class ``band aid"). Traditional methods for discovering spurious features either require extensive human annotations (thus, not scalable), or are useful on specific models. In this work, we introduce a {\it scalable} framework to discover a subset of spurious and causal visual attributes used in inferences of a general model and localize them on a large number of images with minimal human supervision. Our methodology is based on this key idea: to identify spurious or causal \textit{visual attributes} used in model predictions, we identify spurious or causal \textit{neural features} (penultimate layer neurons of a robust model) via limited human supervision (e.g., using top 5 activating images per feature). We then show that these neural feature annotations {\it generalize} extremely well to many more images {\it without} any human supervision. We use the activation maps for these neural features as the soft masks to highlight spurious or causal visual attributes. Using this methodology, we introduce the {\it Causal Imagenet} dataset containing causal and spurious masks for a large set of samples from Imagenet. We assess the performance of several popular Imagenet models and show that they rely heavily on various spurious features in their predictions.

Related papers

• Fine-Tuning Text-To-Image Diffusion Models for Class-Wise Spurious Feature Generation [14.016699770436961]
  We propose a method for generating spurious features by leveraging large-scale text-to-image diffusion models. Experiment results show that our method can generate spurious images that are consistently spurious across different classifiers.
  arXiv  Detail & Related papers  (2024-02-13T03:55:09Z)
• Unlocking Feature Visualization for Deeper Networks with MAgnitude Constrained Optimization [17.93878159391899]
  We describe MACO, a simple approach to generate interpretable images. Our approach yields significantly better results (both qualitatively and quantitatively) and unlocks efficient and interpretable feature visualizations for large state-of-the-art neural networks. We validate our method on a novel benchmark for comparing feature visualization methods, and release its visualizations for all classes of the ImageNet dataset.
  arXiv  Detail & Related papers  (2023-06-11T23:33:59Z)
• Attribute-Guided Multi-Level Attention Network for Fine-Grained Fashion Retrieval [27.751399400911932]
  We introduce an attribute-guided multi-level attention network (AG-MAN) for fine-grained fashion retrieval. Specifically, we first enhance the pre-trained feature extractor to capture multi-level image embedding. Then, we propose a classification scheme where images with the same attribute, albeit with different values, are categorized into the same class.
  arXiv  Detail & Related papers  (2022-12-27T05:28:38Z)
• Spurious Features Everywhere -- Large-Scale Detection of Harmful Spurious Features in ImageNet [36.48282338829549]
  In this paper, we develop a framework that allows us to systematically identify spurious features in large datasets like ImageNet. We validate our results by showing that presence of the harmful spurious feature of a class alone is sufficient to trigger the prediction of that class. We introduce SpuFix as a simple mitigation method to reduce the dependence of any ImageNet classifier on previously identified harmful spurious features.
  arXiv  Detail & Related papers  (2022-12-09T14:23:25Z)
• DisPositioNet: Disentangled Pose and Identity in Semantic Image Manipulation [83.51882381294357]
  DisPositioNet is a model that learns a disentangled representation for each object for the task of image manipulation using scene graphs. Our framework enables the disentanglement of the variational latent embeddings as well as the feature representation in the graph.
  arXiv  Detail & Related papers  (2022-11-10T11:47:37Z)
• Core Risk Minimization using Salient ImageNet [53.616101711801484]
  We introduce the Salient Imagenet dataset with more than 1 million soft masks localizing core and spurious features for all 1000 Imagenet classes. Using this dataset, we first evaluate the reliance of several Imagenet pretrained models (42 total) on spurious features. Next, we introduce a new learning paradigm called Core Risk Minimization (CoRM) whose objective ensures that the model predicts a class using its core features.
  arXiv  Detail & Related papers  (2022-03-28T01:53:34Z)
• Where is the Model Looking At?--Concentrate and Explain the Network Attention [21.037241523836553]
  We propose an Explainable Attribute-based Multi-task (EAT) framework to concentrate the model attention on the discriminative image area. We generate attribute-based textual explanations for the network and ground the attributes on the image to show visual explanations. Results indicate that the EAT framework can give multi-modal explanations that interpret the network decision.
  arXiv  Detail & Related papers  (2020-09-29T08:36:18Z)
• Saliency-driven Class Impressions for Feature Visualization of Deep Neural Networks [55.11806035788036]
  It is advantageous to visualize the features considered to be essential for classification. Existing visualization methods develop high confidence images consisting of both background and foreground features. In this work, we propose a saliency-driven approach to visualize discriminative features that are considered most important for a given task.
  arXiv  Detail & Related papers  (2020-07-31T06:11:06Z)
• CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning [78.3857991931479]
  We present GROLLA, an evaluation framework for Grounded Language Learning with Attributes. We also propose a new dataset CompGuessWhat?! as an instance of this framework for evaluating the quality of learned neural representations.
  arXiv  Detail & Related papers  (2020-06-03T11:21:42Z)
• Focus Longer to See Better:Recursively Refined Attention for Fine-Grained Image Classification [148.4492675737644]
  Deep Neural Network has shown great strides in the coarse-grained image classification task. In this paper, we try to focus on these marginal differences to extract more representative features. Our network repetitively focuses on parts of images to spot small discriminative parts among the classes.
  arXiv  Detail & Related papers  (2020-05-22T03:14:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.