ExBERT: An External Knowledge Enhanced BERT for Natural Language Inference

• URL: http://arxiv.org/abs/2108.01589v1
• Date: Tue, 3 Aug 2021 15:56:49 GMT
• Title: ExBERT: An External Knowledge Enhanced BERT for Natural Language Inference
• Authors: Amit Gajbhiye, Noura Al Moubayed, Steven Bradley
• Abstract summary: We introduce a new model for Natural Language Inference (NLI) called External Knowledge Enhanced BERT (ExBERT) ExBERT takes full advantage of contextual word representations obtained from BERT and employs them to retrieve relevant external knowledge from knowledge graphs. Our model adaptively incorporates the external knowledge context required for reasoning over the inputs.
• Score: 5.188712126001397
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Neural language representation models such as BERT, pre-trained on large-scale unstructured corpora lack explicit grounding to real-world commonsense knowledge and are often unable to remember facts required for reasoning and inference. Natural Language Inference (NLI) is a challenging reasoning task that relies on common human understanding of language and real-world commonsense knowledge. We introduce a new model for NLI called External Knowledge Enhanced BERT (ExBERT), to enrich the contextual representation with real-world commonsense knowledge from external knowledge sources and enhance BERT's language understanding and reasoning capabilities. ExBERT takes full advantage of contextual word representations obtained from BERT and employs them to retrieve relevant external knowledge from knowledge graphs and to encode the retrieved external knowledge. Our model adaptively incorporates the external knowledge context required for reasoning over the inputs. Extensive experiments on the challenging SciTail and SNLI benchmarks demonstrate the effectiveness of ExBERT: in comparison to the previous state-of-the-art, we obtain an accuracy of 95.9% on SciTail and 91.5% on SNLI.

Related papers

• Large Language Models are Limited in Out-of-Context Knowledge Reasoning [65.72847298578071]
  Large Language Models (LLMs) possess extensive knowledge and strong capabilities in performing in-context reasoning. This paper focuses on a significant aspect of out-of-context reasoning: Out-of-Context Knowledge Reasoning (OCKR), which is to combine multiple knowledge to infer new knowledge.
  arXiv  Detail & Related papers  (2024-06-11T15:58:59Z)
• Knowledge Rumination for Pre-trained Language Models [77.55888291165462]
  We propose a new paradigm dubbed Knowledge Rumination to help the pre-trained language model utilize related latent knowledge without retrieving it from the external corpus. We apply the proposed knowledge rumination to various language models, including RoBERTa, DeBERTa, and GPT-3.
  arXiv  Detail & Related papers  (2023-05-15T15:47:09Z)
• UNTER: A Unified Knowledge Interface for Enhancing Pre-trained Language Models [100.4659557650775]
  We propose a UNified knowledge inTERface, UNTER, to provide a unified perspective to exploit both structured knowledge and unstructured knowledge. With both forms of knowledge injected, UNTER gains continuous improvements on a series of knowledge-driven NLP tasks.
  arXiv  Detail & Related papers  (2023-05-02T17:33:28Z)
• LM-CORE: Language Models with Contextually Relevant External Knowledge [13.451001884972033]
  We argue that storing large amounts of knowledge in the model parameters is sub-optimal given the ever-growing amounts of knowledge and resource requirements. We present LM-CORE -- a general framework to achieve this -- that allows textitdecoupling of the language model training from the external knowledge source. Experimental results show that LM-CORE, having access to external knowledge, achieves significant and robust outperformance over state-of-the-art knowledge-enhanced language models on knowledge probing tasks.
  arXiv  Detail & Related papers  (2022-08-12T18:59:37Z)
• Knowledge Graph Fusion for Language Model Fine-tuning [0.0]
  We investigate the benefits of knowledge incorporation into the fine-tuning stages of BERT. An existing K-BERT model, which enriches sentences with triplets from a Knowledge Graph, is adapted for the English language. Changes made to K-BERT for accommodating the English language also extend to other word-based languages.
  arXiv  Detail & Related papers  (2022-06-21T08:06:22Z)
• Generated Knowledge Prompting for Commonsense Reasoning [53.88983683513114]
  We propose generating knowledge statements directly from a language model with a generic prompt format. This approach improves performance of both off-the-shelf and finetuned language models on four commonsense reasoning tasks. Notably, we find that a model's predictions can improve when using its own generated knowledge.
  arXiv  Detail & Related papers  (2021-10-15T21:58:03Z)
• Does External Knowledge Help Explainable Natural Language Inference? Automatic Evaluation vs. Human Ratings [35.2513653224183]
  Natural language inference (NLI) requires models to learn and apply commonsense knowledge. We investigate whether external knowledge can also improve their explanation capabilities. We conduct the largest and most fine-grained explainable NLI crowdsourcing study to date.
  arXiv  Detail & Related papers  (2021-09-16T09:56:20Z)
• Does Knowledge Help General NLU? An Empirical Study [13.305282275999781]
  We investigate the contribution of external knowledge by measuring the end-to-end performance of language models. We find that the introduction of knowledge can significantly improve the results on certain tasks while having no adverse effects on other tasks.
  arXiv  Detail & Related papers  (2021-09-01T18:17:36Z)
• CoLAKE: Contextualized Language and Knowledge Embedding [81.90416952762803]
  We propose the Contextualized Language and Knowledge Embedding (CoLAKE) CoLAKE jointly learns contextualized representation for both language and knowledge with the extended objective. We conduct experiments on knowledge-driven tasks, knowledge probing tasks, and language understanding tasks.
  arXiv  Detail & Related papers  (2020-10-01T11:39:32Z)
• Unsupervised Commonsense Question Answering with Self-Talk [71.63983121558843]
  We propose an unsupervised framework based on self-talk as a novel alternative to commonsense tasks. Inspired by inquiry-based discovery learning, our approach inquires language models with a number of information seeking questions. Empirical results demonstrate that the self-talk procedure substantially improves the performance of zero-shot language model baselines.
  arXiv  Detail & Related papers  (2020-04-11T20:43:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.