LIREx: Augmenting Language Inference with Relevant Explanation

• URL: http://arxiv.org/abs/2012.09157v1
• Date: Wed, 16 Dec 2020 18:49:29 GMT
• Title: LIREx: Augmenting Language Inference with Relevant Explanation
• Authors: Xinyan Zhao, V.G.Vinod Vydiswaran
• Abstract summary: Natural language explanations (NLEs) are a form of data annotation in which annotators identify rationales when assigning labels to data instances. NLEs have been shown to capture human reasoning better, but not as beneficial for natural language inference. We propose a novel framework, LIREx, that incorporates both a rationale-enabled explanation generator and an instance selector to select only relevant NLEs.
• Score: 1.4780878458667916
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Natural language explanations (NLEs) are a special form of data annotation in which annotators identify rationales (most significant text tokens) when assigning labels to data instances, and write out explanations for the labels in natural language based on the rationales. NLEs have been shown to capture human reasoning better, but not as beneficial for natural language inference (NLI). In this paper, we analyze two primary flaws in the way NLEs are currently used to train explanation generators for language inference tasks. We find that the explanation generators do not take into account the variability inherent in human explanation of labels, and that the current explanation generation models generate spurious explanations. To overcome these limitations, we propose a novel framework, LIREx, that incorporates both a rationale-enabled explanation generator and an instance selector to select only relevant, plausible NLEs to augment NLI models. When evaluated on the standardized SNLI data set, LIREx achieved an accuracy of 91.87%, an improvement of 0.32 over the baseline and matching the best-reported performance on the data set. It also achieves significantly better performance than previous studies when transferred to the out-of-domain MultiNLI data set. Qualitative analysis shows that LIREx generates flexible, faithful, and relevant NLEs that allow the model to be more robust to spurious explanations. The code is available at https://github.com/zhaoxy92/LIREx.

Related papers

• ExaRanker-Open: Synthetic Explanation for IR using Open-Source LLMs [60.81649785463651]
  We introduce ExaRanker-Open, where we adapt and explore the use of open-source language models to generate explanations. Our findings reveal that incorporating explanations consistently enhances neural rankers, with benefits escalating as the LLM size increases.
  arXiv  Detail & Related papers  (2024-02-09T11:23:14Z)
• FOLIO: Natural Language Reasoning with First-Order Logic [147.50480350846726]
  We present FOLIO, a human-annotated, logically complex and diverse dataset for reasoning in natural language (NL) FOLIO consists of 1,430 examples (unique conclusions), each paired with one of 487 sets of premises used to deductively reason for the validity of each conclusion. For both NL reasoning and NL-FOL translation, we benchmark multiple state-of-the-art language models.
  arXiv  Detail & Related papers  (2022-09-02T06:50:11Z)
• Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters [35.103851212995046]
  Natural Language Inference (NLI) has been extensively studied by the NLP community as a framework for estimating the semantic relation between sentence pairs. We explore the direct zero-shot applicability of NLI models to real applications, beyond the sentence-pair setting they were trained on. We develop new aggregation methods to allow operating over full documents, reaching state-of-the-art performance on the ContractNLI dataset.
  arXiv  Detail & Related papers  (2022-04-15T12:56:39Z)
• Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language? [86.60613602337246]
  We introduce a leakage-adjusted simulatability (LAS) metric for evaluating NL explanations. LAS measures how well explanations help an observer predict a model's output, while controlling for how explanations can directly leak the output. We frame explanation generation as a multi-agent game and optimize explanations for simulatability while penalizing label leakage.
  arXiv  Detail & Related papers  (2020-10-08T16:59:07Z)
• Discriminatively-Tuned Generative Classifiers for Robust Natural Language Inference [59.62779187457773]
  We propose a generative classifier for natural language inference (NLI) We compare it to five baselines, including discriminative models and large-scale pretrained language representation models like BERT. Experiments show that GenNLI outperforms both discriminative and pretrained baselines across several challenging NLI experimental settings.
  arXiv  Detail & Related papers  (2020-10-08T04:44:00Z)
• Reading Comprehension as Natural Language Inference: A Semantic Analysis [15.624486319943015]
  We explore the utility of Natural language Inference (NLI) for Question Answering (QA) We transform the one of the largest available MRC dataset (RACE) to an NLI form, and compare the performances of a state-of-the-art model (RoBERTa) on both forms. We highlight clear categories for which the model is able to perform better when the data is presented in a coherent entailment form, and a structured question-answer concatenation form.
  arXiv  Detail & Related papers  (2020-10-04T22:50:59Z)
• Mining Knowledge for Natural Language Inference from Wikipedia Categories [53.26072815839198]
  We introduce WikiNLI: a resource for improving model performance on NLI and LE tasks. It contains 428,899 pairs of phrases constructed from naturally annotated category hierarchies in Wikipedia. We show that we can improve strong baselines such as BERT and RoBERTa by pretraining them on WikiNLI and transferring the models on downstream tasks.
  arXiv  Detail & Related papers  (2020-10-03T00:45:01Z)
• NILE : Natural Language Inference with Faithful Natural Language Explanations [10.074153632701952]
  We propose Natural-language Inference over Label-specific Explanations (NILE) NILE is a novel NLI method which utilizes auto-generated label-specific explanations to produce labels along with its faithful explanation. We discuss the faithfulness of NILE's explanations in terms of sensitivity of the decisions to the corresponding explanations.
  arXiv  Detail & Related papers  (2020-05-25T13:56:03Z)
• e-SNLI-VE: Corrected Visual-Textual Entailment with Natural Language Explanations [87.71914254873857]
  We present a data collection effort to correct the class with the highest error rate in SNLI-VE. Thirdly, we introduce e-SNLI-VE, which appends human-written natural language explanations to SNLI-VE. We train models that learn from these explanations at training time, and output such explanations at testing time.
  arXiv  Detail & Related papers  (2020-04-07T23:12:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.