Toward Understanding the Disagreement Problem in Neural Network Feature Attribution

• URL: http://arxiv.org/abs/2404.11330v1
• Date: Wed, 17 Apr 2024 12:45:59 GMT
• Title: Toward Understanding the Disagreement Problem in Neural Network Feature Attribution
• Authors: Niklas Koenen, Marvin N. Wright,
• Abstract summary: neural networks have demonstrated their remarkable ability to discern intricate patterns and relationships from raw data. Understanding the inner workings of these black box models remains challenging, yet crucial for high-stake decisions. Our work addresses this confusion by investigating the explanations' fundamental and distributional behavior.
• Score: 0.8057006406834466
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: In recent years, neural networks have demonstrated their remarkable ability to discern intricate patterns and relationships from raw data. However, understanding the inner workings of these black box models remains challenging, yet crucial for high-stake decisions. Among the prominent approaches for explaining these black boxes are feature attribution methods, which assign relevance or contribution scores to each input variable for a model prediction. Despite the plethora of proposed techniques, ranging from gradient-based to backpropagation-based methods, a significant debate persists about which method to use. Various evaluation metrics have been proposed to assess the trustworthiness or robustness of their results. However, current research highlights disagreement among state-of-the-art methods in their explanations. Our work addresses this confusion by investigating the explanations' fundamental and distributional behavior. Additionally, through a comprehensive simulation study, we illustrate the impact of common scaling and encoding techniques on the explanation quality, assess their efficacy across different effect sizes, and demonstrate the origin of inconsistency in rank-based evaluation metrics.

Related papers

• Understanding Disparities in Post Hoc Machine Learning Explanation [2.965442487094603]
  Previous work has highlighted that existing post-hoc explanation methods exhibit disparities in explanation fidelity (across 'race' and 'gender' as sensitive attributes) We specifically assess challenges to explanation disparities that originate from properties of the data. Results indicate that disparities in model explanations can also depend on data and model properties.
  arXiv  Detail & Related papers  (2024-01-25T22:09:28Z)
• Benchmarking Bayesian Causal Discovery Methods for Downstream Treatment Effect Estimation [137.3520153445413]
  A notable gap exists in the evaluation of causal discovery methods, where insufficient emphasis is placed on downstream inference. We evaluate seven established baseline causal discovery methods including a newly proposed method based on GFlowNets. The results of our study demonstrate that some of the algorithms studied are able to effectively capture a wide range of useful and diverse ATE modes.
  arXiv  Detail & Related papers  (2023-07-11T02:58:10Z)
• Better Understanding Differences in Attribution Methods via Systematic Evaluations [57.35035463793008]
  Post-hoc attribution methods have been proposed to identify image regions most influential to the models' decisions. We propose three novel evaluation schemes to more reliably measure the faithfulness of those methods. We use these evaluation schemes to study strengths and shortcomings of some widely used attribution methods over a wide range of models.
  arXiv  Detail & Related papers  (2023-03-21T14:24:58Z)
• EvalAttAI: A Holistic Approach to Evaluating Attribution Maps in Robust and Non-Robust Models [0.3425341633647624]
  This paper focuses on evaluating methods of attribution mapping to find whether robust neural networks are more explainable. We propose a new explainability faithfulness metric (called EvalAttAI) that addresses the limitations of prior metrics.
  arXiv  Detail & Related papers  (2023-03-15T18:33:22Z)
• On The Coherence of Quantitative Evaluation of Visual Explanations [0.7212939068975619]
  Evaluation methods have been proposed to assess the "goodness" of visual explanations. We study a subset of the ImageNet-1k validation set where we evaluate a number of different commonly-used explanation methods. Results of our study suggest that there is a lack of coherency on the grading provided by some of the considered evaluation methods.
  arXiv  Detail & Related papers  (2023-02-14T13:41:57Z)
• Neural Causal Models for Counterfactual Identification and Estimation [62.30444687707919]
  We study the evaluation of counterfactual statements through neural models. First, we show that neural causal models (NCMs) are expressive enough. Second, we develop an algorithm for simultaneously identifying and estimating counterfactual distributions.
  arXiv  Detail & Related papers  (2022-09-30T18:29:09Z)
• Towards Better Understanding Attribution Methods [77.1487219861185]
  Post-hoc attribution methods have been proposed to identify image regions most influential to the models' decisions. We propose three novel evaluation schemes to more reliably measure the faithfulness of those methods. We also propose a post-processing smoothing step that significantly improves the performance of some attribution methods.
  arXiv  Detail & Related papers  (2022-05-20T20:50:17Z)
• Discriminative Attribution from Counterfactuals [64.94009515033984]
  We present a method for neural network interpretability by combining feature attribution with counterfactual explanations. We show that this method can be used to quantitatively evaluate the performance of feature attribution methods in an objective manner.
  arXiv  Detail & Related papers  (2021-09-28T00:53:34Z)
• On the Objective Evaluation of Post Hoc Explainers [10.981508361941335]
  Modern trends in machine learning research have led to algorithms that are increasingly intricate to the degree that they are considered to be black boxes. In an effort to reduce the opacity of decisions, methods have been proposed to construe the inner workings of such models in a human-comprehensible manner. We propose a framework for the evaluation of post hoc explainers on ground truth that is directly derived from the additive structure of a model.
  arXiv  Detail & Related papers  (2021-06-15T19:06:51Z)
• A Diagnostic Study of Explainability Techniques for Text Classification [52.879658637466605]
  We develop a list of diagnostic properties for evaluating existing explainability techniques. We compare the saliency scores assigned by the explainability techniques with human annotations of salient input regions to find relations between a model's performance and the agreement of its rationales with human ones.
  arXiv  Detail & Related papers  (2020-09-25T12:01:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.