Evaluating Explanation Methods for Neural Machine Translation

• URL: http://arxiv.org/abs/2005.01672v1
• Date: Mon, 4 May 2020 17:26:25 GMT
• Title: Evaluating Explanation Methods for Neural Machine Translation
• Authors: Jierui Li, Lemao Liu, Huayang Li, Guanlin Li, Guoping Huang, Shuming Shi
• Abstract summary: We propose a principled metric based on fidelity in regard to the predictive behavior of the NMT model. On six standard translation tasks, we quantitatively evaluate several explanation methods in terms of the proposed metric.
• Score: 44.836653992441455
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently many efforts have been devoted to interpreting the black-box NMT models, but little progress has been made on metrics to evaluate explanation methods. Word Alignment Error Rate can be used as such a metric that matches human understanding, however, it can not measure explanation methods on those target words that are not aligned to any source word. This paper thereby makes an initial attempt to evaluate explanation methods from an alternative viewpoint. To this end, it proposes a principled metric based on fidelity in regard to the predictive behavior of the NMT model. As the exact computation for this metric is intractable, we employ an efficient approach as its approximation. On six standard translation tasks, we quantitatively evaluate several explanation methods in terms of the proposed metric and we reveal some valuable findings for these explanation methods in our experiments.

Related papers

• Rethinking Distance Metrics for Counterfactual Explainability [53.436414009687]
  We investigate a framing for counterfactual generation methods that considers counterfactuals not as independent draws from a region around the reference, but as jointly sampled with the reference from the underlying data distribution. We derive a distance metric, tailored for counterfactual similarity that can be applied to a broad range of settings.
  arXiv  Detail & Related papers  (2024-10-18T15:06:50Z)
• Counterfactuals of Counterfactuals: a back-translation-inspired approach to analyse counterfactual editors [3.4253416336476246]
  We focus on the analysis of counterfactual, contrastive explanations. We propose a new back translation-inspired evaluation methodology. We show that by iteratively feeding the counterfactual to the explainer we can obtain valuable insights into the behaviour of both the predictor and the explainer models.
  arXiv  Detail & Related papers  (2023-05-26T16:04:28Z)
• Abductive Commonsense Reasoning Exploiting Mutually Exclusive Explanations [118.0818807474809]
  Abductive reasoning aims to find plausible explanations for an event. Existing approaches for abductive reasoning in natural language processing often rely on manually generated annotations for supervision. This work proposes an approach for abductive commonsense reasoning that exploits the fact that only a subset of explanations is correct for a given context.
  arXiv  Detail & Related papers  (2023-05-24T01:35:10Z)
• Robustness of Explanation Methods for NLP Models [5.191443390565865]
  Explanation methods have emerged as an important tool to highlight the features responsible for the predictions of neural networks. There is mounting evidence that many explanation methods are rather unreliable and susceptible to malicious manipulations. We provide initial insights and results towards devising a successful adversarial attack against text explanations.
  arXiv  Detail & Related papers  (2022-06-24T13:34:07Z)
• Principled Paraphrase Generation with Parallel Corpora [52.78059089341062]
  We formalize the implicit similarity function induced by round-trip Machine Translation. We show that it is susceptible to non-paraphrase pairs sharing a single ambiguous translation. We design an alternative similarity metric that mitigates this issue.
  arXiv  Detail & Related papers  (2022-05-24T17:22:42Z)
• The Solvability of Interpretability Evaluation Metrics [7.3709604810699085]
  Feature attribution methods are often evaluated on metrics such as comprehensiveness and sufficiency. In this paper, we highlight an intriguing property of these metrics: their solvability. We present a series of investigations showing that this beam search explainer is generally comparable or favorable to current choices.
  arXiv  Detail & Related papers  (2022-05-18T02:52:03Z)
• A Unified Study of Machine Learning Explanation Evaluation Metrics [16.4602888153369]
  Many existing metrics for explanations are introduced by researchers as by-products of their proposed explanation techniques to demonstrate the advantages of their methods. We claim that the lack of acknowledged and justified metrics results in chaos in benchmarking these explanation methods. We propose guidelines in dealing with the problems in evaluating machine learning explanation and encourage researchers to carefully deal with these problems when developing explanation techniques and metrics.
  arXiv  Detail & Related papers  (2022-03-27T10:12:06Z)
• On Sample Based Explanation Methods for NLP:Efficiency, Faithfulness, and Semantic Evaluation [23.72825603188359]
  We can improve the interpretability of explanations by allowing arbitrary text sequences as the explanation unit. We propose a semantic-based evaluation metric that can better align with humans' judgment of explanations.
  arXiv  Detail & Related papers  (2021-06-09T00:49:56Z)
• Evaluating Explanations: How much do explanations from the teacher aid students? [103.05037537415811]
  We formalize the value of explanations using a student-teacher paradigm that measures the extent to which explanations improve student models in learning. Unlike many prior proposals to evaluate explanations, our approach cannot be easily gamed, enabling principled, scalable, and automatic evaluation of attributions.
  arXiv  Detail & Related papers  (2020-12-01T23:40:21Z)
• Approximate MMAP by Marginal Search [78.50747042819503]
  We present a strategy for marginal MAP queries in graphical models. The proposed confidence measure is properly detecting instances for which the algorithm is accurate. For sufficiently high confidence levels, the algorithm gives the exact solution or an approximation whose Hamming distance from the exact one is small.
  arXiv  Detail & Related papers  (2020-02-12T07:41:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.