Human Interpretation of Saliency-based Explanation Over Text

• URL: http://arxiv.org/abs/2201.11569v1
• Date: Thu, 27 Jan 2022 15:20:32 GMT
• Title: Human Interpretation of Saliency-based Explanation Over Text
• Authors: Hendrik Schuff, Alon Jacovi, Heike Adel, Yoav Goldberg and Ngoc Thang Vu
• Abstract summary: We study saliency-based explanations over textual data. We find that people often mis-interpret the explanations. We propose a method to adjust saliencies based on model estimates of over- and under-perception.
• Score: 65.29015910991261
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While a lot of research in explainable AI focuses on producing effective explanations, less work is devoted to the question of how people understand and interpret the explanation. In this work, we focus on this question through a study of saliency-based explanations over textual data. Feature-attribution explanations of text models aim to communicate which parts of the input text were more influential than others towards the model decision. Many current explanation methods, such as gradient-based or Shapley value-based methods, provide measures of importance which are well-understood mathematically. But how does a person receiving the explanation (the explainee) comprehend it? And does their understanding match what the explanation attempted to communicate? We empirically investigate the effect of various factors of the input, the feature-attribution explanation, and visualization procedure, on laypeople's interpretation of the explanation. We query crowdworkers for their interpretation on tasks in English and German, and fit a GAMM model to their responses considering the factors of interest. We find that people often mis-interpret the explanations: superficial and unrelated factors, such as word length, influence the explainees' importance assignment despite the explanation communicating importance directly. We then show that some of this distortion can be attenuated: we propose a method to adjust saliencies based on model estimates of over- and under-perception, and explore bar charts as an alternative to heatmap saliency visualization. We find that both approaches can attenuate the distorting effect of specific factors, leading to better-calibrated understanding of the explanation.

Related papers

• Dissenting Explanations: Leveraging Disagreement to Reduce Model Overreliance [4.962171160815189]
  We introduce the notion of dissenting explanations: conflicting predictions with accompanying explanations. We first explore the advantage of dissenting explanations in the setting of model multiplicity. We demonstrate that dissenting explanations reduce overreliance on model predictions, without reducing overall accuracy.
  arXiv  Detail & Related papers  (2023-07-14T21:27:00Z)
• Explanation Selection Using Unlabeled Data for Chain-of-Thought Prompting [80.9896041501715]
  Explanations that have not been "tuned" for a task, such as off-the-shelf explanations written by nonexperts, may lead to mediocre performance. This paper tackles the problem of how to optimize explanation-infused prompts in a blackbox fashion.
  arXiv  Detail & Related papers  (2023-02-09T18:02:34Z)
• Complementary Explanations for Effective In-Context Learning [77.83124315634386]
  Large language models (LLMs) have exhibited remarkable capabilities in learning from explanations in prompts. This work aims to better understand the mechanisms by which explanations are used for in-context learning.
  arXiv  Detail & Related papers  (2022-11-25T04:40:47Z)
• REX: Reasoning-aware and Grounded Explanation [30.392986232906107]
  We develop a new type of multi-modal explanations that explain the decisions by traversing the reasoning process and grounding keywords in the images. Second, we identify the critical need to tightly couple important components across the visual and textual modalities for explaining the decisions. Third, we propose a novel explanation generation method that explicitly models the pairwise correspondence between words and regions of interest.
  arXiv  Detail & Related papers  (2022-03-11T17:28:42Z)
• Causal Explanations and XAI [8.909115457491522]
  An important goal of Explainable Artificial Intelligence (XAI) is to compensate for mismatches by offering explanations. I take a step further by formally defining the causal notions of sufficient explanations and counterfactual explanations. I also touch upon the significance of this work for fairness in AI by showing how actual causation can be used to improve the idea of path-specific counterfactual fairness.
  arXiv  Detail & Related papers  (2022-01-31T12:32:10Z)
• Contrastive Explanations for Model Interpretability [77.92370750072831]
  We propose a methodology to produce contrastive explanations for classification models. Our method is based on projecting model representation to a latent space. Our findings shed light on the ability of label-contrastive explanations to provide a more accurate and finer-grained interpretability of a model's decision.
  arXiv  Detail & Related papers  (2021-03-02T00:36:45Z)
• Explainers in the Wild: Making Surrogate Explainers Robust to Distortions through Perception [77.34726150561087]
  We propose a methodology to evaluate the effect of distortions in explanations by embedding perceptual distances. We generate explanations for images in the Imagenet-C dataset and demonstrate how using a perceptual distances in the surrogate explainer creates more coherent explanations for the distorted and reference images.
  arXiv  Detail & Related papers  (2021-02-22T12:38:53Z)
• The Struggles of Feature-Based Explanations: Shapley Values vs. Minimal Sufficient Subsets [61.66584140190247]
  We show that feature-based explanations pose problems even for explaining trivial models. We show that two popular classes of explainers, Shapley explainers and minimal sufficient subsets explainers, target fundamentally different types of ground-truth explanations.
  arXiv  Detail & Related papers  (2020-09-23T09:45:23Z)
• The Explanation Game: Towards Prediction Explainability through Sparse Communication [6.497816402045099]
  We provide a unified perspective of explainability as a problem between an explainer and a layperson. We use this framework to compare several prior approaches for extracting explanations. We propose new embedded methods for explainability, through the use of selective, sparse attention.
  arXiv  Detail & Related papers  (2020-04-28T22:27:19Z)
• Order Matters: Generating Progressive Explanations for Planning Tasks in Human-Robot Teaming [11.35869940310993]
  We aim to investigate effects during explanation generation when an explanation is broken into multiple parts that are communicated sequentially. We first evaluate our approach on a scavenger-hunt domain to demonstrate its effectively capturing the humans' preferences. Results confirmed our hypothesis that the process of understanding an explanation was a dynamic process.
  arXiv  Detail & Related papers  (2020-04-16T00:17:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.