Does External Knowledge Help Explainable Natural Language Inference? Automatic Evaluation vs. Human Ratings

• URL: http://arxiv.org/abs/2109.07833v1
• Date: Thu, 16 Sep 2021 09:56:20 GMT
• Title: Does External Knowledge Help Explainable Natural Language Inference? Automatic Evaluation vs. Human Ratings
• Authors: Hendrik Schuff, Hsiu-Yu Yang, Heike Adel, Ngoc Thang Vu
• Abstract summary: Natural language inference (NLI) requires models to learn and apply commonsense knowledge. We investigate whether external knowledge can also improve their explanation capabilities. We conduct the largest and most fine-grained explainable NLI crowdsourcing study to date.
• Score: 35.2513653224183
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Natural language inference (NLI) requires models to learn and apply commonsense knowledge. These reasoning abilities are particularly important for explainable NLI systems that generate a natural language explanation in addition to their label prediction. The integration of external knowledge has been shown to improve NLI systems, here we investigate whether it can also improve their explanation capabilities. For this, we investigate different sources of external knowledge and evaluate the performance of our models on in-domain data as well as on special transfer datasets that are designed to assess fine-grained reasoning capabilities. We find that different sources of knowledge have a different effect on reasoning abilities, for example, implicit knowledge stored in language models can hinder reasoning on numbers and negations. Finally, we conduct the largest and most fine-grained explainable NLI crowdsourcing study to date. It reveals that even large differences in automatic performance scores do neither reflect in human ratings of label, explanation, commonsense nor grammar correctness.

Related papers

• Does Knowledge Localization Hold True? Surprising Differences Between Entity and Relation Perspectives in Language Models [20.157061521694096]
  This study investigates the differences between entity and relational knowledge through knowledge editing. To further elucidate the differences between entity and relational knowledge, we employ causal analysis to investigate how relational knowledge is stored in pre-trained models. This insight highlights the multifaceted nature of knowledge storage in language models, underscoring the complexity of manipulating specific types of knowledge within these models.
  arXiv  Detail & Related papers  (2024-09-01T05:09:11Z)
• Large Language Models Are Cross-Lingual Knowledge-Free Reasoners [43.99097308487008]
  We decompose the process of reasoning tasks into two separated components: knowledge retrieval and knowledge-free reasoning. We show that the knowledge-free reasoning capability can be nearly perfectly transferred across various source-target language directions. We hypothesize that knowledge-free reasoning shares similar neurons in different languages for reasoning, while knowledge is stored separately in different languages.
  arXiv  Detail & Related papers  (2024-06-24T14:03:04Z)
• Large Language Models are Limited in Out-of-Context Knowledge Reasoning [65.72847298578071]
  Large Language Models (LLMs) possess extensive knowledge and strong capabilities in performing in-context reasoning. This paper focuses on a significant aspect of out-of-context reasoning: Out-of-Context Knowledge Reasoning (OCKR), which is to combine multiple knowledge to infer new knowledge.
  arXiv  Detail & Related papers  (2024-06-11T15:58:59Z)
• The KITMUS Test: Evaluating Knowledge Integration from Multiple Sources in Natural Language Understanding Systems [87.3207729953778]
  We evaluate state-of-the-art coreference resolution models on our dataset. Several models struggle to reason on-the-fly over knowledge observed both at pretrain time and at inference time. Still, even the best performing models seem to have difficulties with reliably integrating knowledge presented only at inference time.
  arXiv  Detail & Related papers  (2022-12-15T23:26:54Z)
• Transparency Helps Reveal When Language Models Learn Meaning [71.96920839263457]
  Our systematic experiments with synthetic data reveal that, with languages where all expressions have context-independent denotations, both autoregressive and masked language models learn to emulate semantic relations between expressions. Turning to natural language, our experiments with a specific phenomenon -- referential opacity -- add to the growing body of evidence that current language models do not well-represent natural language semantics.
  arXiv  Detail & Related papers  (2022-10-14T02:35:19Z)
• Generated Knowledge Prompting for Commonsense Reasoning [53.88983683513114]
  We propose generating knowledge statements directly from a language model with a generic prompt format. This approach improves performance of both off-the-shelf and finetuned language models on four commonsense reasoning tasks. Notably, we find that a model's predictions can improve when using its own generated knowledge.
  arXiv  Detail & Related papers  (2021-10-15T21:58:03Z)
• Does Knowledge Help General NLU? An Empirical Study [13.305282275999781]
  We investigate the contribution of external knowledge by measuring the end-to-end performance of language models. We find that the introduction of knowledge can significantly improve the results on certain tasks while having no adverse effects on other tasks.
  arXiv  Detail & Related papers  (2021-09-01T18:17:36Z)
• ExBERT: An External Knowledge Enhanced BERT for Natural Language Inference [5.188712126001397]
  We introduce a new model for Natural Language Inference (NLI) called External Knowledge Enhanced BERT (ExBERT) ExBERT takes full advantage of contextual word representations obtained from BERT and employs them to retrieve relevant external knowledge from knowledge graphs. Our model adaptively incorporates the external knowledge context required for reasoning over the inputs.
  arXiv  Detail & Related papers  (2021-08-03T15:56:49Z)
• NILE : Natural Language Inference with Faithful Natural Language Explanations [10.074153632701952]
  We propose Natural-language Inference over Label-specific Explanations (NILE) NILE is a novel NLI method which utilizes auto-generated label-specific explanations to produce labels along with its faithful explanation. We discuss the faithfulness of NILE's explanations in terms of sensitivity of the decisions to the corresponding explanations.
  arXiv  Detail & Related papers  (2020-05-25T13:56:03Z)
• Unsupervised Commonsense Question Answering with Self-Talk [71.63983121558843]
  We propose an unsupervised framework based on self-talk as a novel alternative to commonsense tasks. Inspired by inquiry-based discovery learning, our approach inquires language models with a number of information seeking questions. Empirical results demonstrate that the self-talk procedure substantially improves the performance of zero-shot language model baselines.
  arXiv  Detail & Related papers  (2020-04-11T20:43:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.