NILE : Natural Language Inference with Faithful Natural Language Explanations

• URL: http://arxiv.org/abs/2005.12116v1
• Date: Mon, 25 May 2020 13:56:03 GMT
• Title: NILE : Natural Language Inference with Faithful Natural Language Explanations
• Authors: Sawan Kumar and Partha Talukdar
• Abstract summary: We propose Natural-language Inference over Label-specific Explanations (NILE) NILE is a novel NLI method which utilizes auto-generated label-specific explanations to produce labels along with its faithful explanation. We discuss the faithfulness of NILE's explanations in terms of sensitivity of the decisions to the corresponding explanations.
• Score: 10.074153632701952
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The recent growth in the popularity and success of deep learning models on NLP classification tasks has accompanied the need for generating some form of natural language explanation of the predicted labels. Such generated natural language (NL) explanations are expected to be faithful, i.e., they should correlate well with the model's internal decision making. In this work, we focus on the task of natural language inference (NLI) and address the following question: can we build NLI systems which produce labels with high accuracy, while also generating faithful explanations of its decisions? We propose Natural-language Inference over Label-specific Explanations (NILE), a novel NLI method which utilizes auto-generated label-specific NL explanations to produce labels along with its faithful explanation. We demonstrate NILE's effectiveness over previously reported methods through automated and human evaluation of the produced labels and explanations. Our evaluation of NILE also supports the claim that accurate systems capable of providing testable explanations of their decisions can be designed. We discuss the faithfulness of NILE's explanations in terms of sensitivity of the decisions to the corresponding explanations. We argue that explicit evaluation of faithfulness, in addition to label and explanation accuracy, is an important step in evaluating model's explanations. Further, we demonstrate that task-specific probes are necessary to establish such sensitivity.

Related papers

• TAGExplainer: Narrating Graph Explanations for Text-Attributed Graph Learning Models [14.367754016281934]
  This paper presents TAGExplainer, the first method designed to generate natural language explanations for TAG learning. To address the lack of annotated ground truth explanations in real-world scenarios, we propose first generating pseudo-labels that capture the model's decisions from saliency-based explanations. The high-quality pseudo-labels are finally utilized to train an end-to-end explanation generator model.
  arXiv  Detail & Related papers  (2024-10-20T03:55:46Z)
• Ecologically Valid Explanations for Label Variation in NLI [27.324994764803808]
  We build LiveNLI, an English dataset of 1,415 ecologically valid explanations (annotators explain the NLI labels they chose) for 122 MNLI items. LiveNLI explanations confirm that people can systematically vary on their interpretation and highlight within-label variation. This suggests that explanations are crucial for navigating label interpretations in general.
  arXiv  Detail & Related papers  (2023-10-20T22:52:19Z)
• Situated Natural Language Explanations [54.083715161895036]
  Natural language explanations (NLEs) are among the most accessible tools for explaining decisions to humans. Existing NLE research perspectives do not take the audience into account. Situated NLE provides a perspective and facilitates further research on the generation and evaluation of explanations.
  arXiv  Detail & Related papers  (2023-08-27T14:14:28Z)
• Faithfulness Tests for Natural Language Explanations [87.01093277918599]
  Explanations of neural models aim to reveal a model's decision-making process for its predictions. Recent work shows that current methods giving explanations such as saliency maps or counterfactuals can be misleading. This work explores the challenging question of evaluating the faithfulness of natural language explanations.
  arXiv  Detail & Related papers  (2023-05-29T11:40:37Z)
• Understanding and Predicting Human Label Variation in Natural Language Inference through Explanation [18.161206115232066]
  We create the first ecologically valid explanation dataset with diverse reasoning, LiveNLI. LiveNLI contains annotators' highlights and free-text explanations for the label(s) of their choice for 122 English Natural Language Inference items. We used its explanations for chain-of-thought prompting, and found there is still room for improvement in GPT-3's ability to predict label distribution with in-context learning.
  arXiv  Detail & Related papers  (2023-04-24T20:45:09Z)
• Benchmarking Faithfulness: Towards Accurate Natural Language Explanations in Vision-Language Tasks [0.0]
  Natural language explanations (NLEs) promise to enable the communication of a model's decision-making in an easily intelligible way. While current models successfully generate convincing explanations, it is an open question how well the NLEs actually represent the reasoning process of the models. We propose three faithfulness metrics: Attribution-Similarity, NLE-Sufficiency, and NLE-Comprehensiveness.
  arXiv  Detail & Related papers  (2023-04-03T08:24:10Z)
• Language Models as Inductive Reasoners [125.99461874008703]
  We propose a new paradigm (task) for inductive reasoning, which is to induce natural language rules from natural language facts. We create a dataset termed DEER containing 1.2k rule-fact pairs for the task, where rules and facts are written in natural language. We provide the first and comprehensive analysis of how well pretrained language models can induce natural language rules from natural language facts.
  arXiv  Detail & Related papers  (2022-12-21T11:12:14Z)
• The Unreliability of Explanations in Few-Shot In-Context Learning [50.77996380021221]
  We focus on two NLP tasks that involve reasoning over text, namely question answering and natural language inference. We show that explanations judged as good by humans--those that are logically consistent with the input--usually indicate more accurate predictions. We present a framework for calibrating model predictions based on the reliability of the explanations.
  arXiv  Detail & Related papers  (2022-05-06T17:57:58Z)
• Interpreting Language Models with Contrastive Explanations [99.7035899290924]
  Language models must consider various features to predict a token, such as its part of speech, number, tense, or semantics. Existing explanation methods conflate evidence for all these features into a single explanation, which is less interpretable for human understanding. We show that contrastive explanations are quantifiably better than non-contrastive explanations in verifying major grammatical phenomena.
  arXiv  Detail & Related papers  (2022-02-21T18:32:24Z)
• Does External Knowledge Help Explainable Natural Language Inference? Automatic Evaluation vs. Human Ratings [35.2513653224183]
  Natural language inference (NLI) requires models to learn and apply commonsense knowledge. We investigate whether external knowledge can also improve their explanation capabilities. We conduct the largest and most fine-grained explainable NLI crowdsourcing study to date.
  arXiv  Detail & Related papers  (2021-09-16T09:56:20Z)
• Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language? [86.60613602337246]
  We introduce a leakage-adjusted simulatability (LAS) metric for evaluating NL explanations. LAS measures how well explanations help an observer predict a model's output, while controlling for how explanations can directly leak the output. We frame explanation generation as a multi-agent game and optimize explanations for simulatability while penalizing label leakage.
  arXiv  Detail & Related papers  (2020-10-08T16:59:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.