Does Knowledge Help General NLU? An Empirical Study

• URL: http://arxiv.org/abs/2109.00563v1
• Date: Wed, 1 Sep 2021 18:17:36 GMT
• Title: Does Knowledge Help General NLU? An Empirical Study
• Authors: Ruochen Xu, Yuwei Fang, Chenguang Zhu, Michael Zeng
• Abstract summary: We investigate the contribution of external knowledge by measuring the end-to-end performance of language models. We find that the introduction of knowledge can significantly improve the results on certain tasks while having no adverse effects on other tasks.
• Score: 13.305282275999781
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: It is often observed in knowledge-centric tasks (e.g., common sense question and answering, relation classification) that the integration of external knowledge such as entity representation into language models can help provide useful information to boost the performance. However, it is still unclear whether this benefit can extend to general natural language understanding (NLU) tasks. In this work, we empirically investigated the contribution of external knowledge by measuring the end-to-end performance of language models with various knowledge integration methods. We find that the introduction of knowledge can significantly improve the results on certain tasks while having no adverse effects on other tasks. We then employ mutual information to reflect the difference brought by knowledge and a neural interpretation model to reveal how a language model utilizes external knowledge. Our study provides valuable insights and guidance for practitioners to equip NLP models with knowledge.

Related papers

• Does Knowledge Localization Hold True? Surprising Differences Between Entity and Relation Perspectives in Language Models [20.157061521694096]
  This study investigates the differences between entity and relational knowledge through knowledge editing. To further elucidate the differences between entity and relational knowledge, we employ causal analysis to investigate how relational knowledge is stored in pre-trained models. This insight highlights the multifaceted nature of knowledge storage in language models, underscoring the complexity of manipulating specific types of knowledge within these models.
  arXiv  Detail & Related papers  (2024-09-01T05:09:11Z)
• Chain-of-Knowledge: Integrating Knowledge Reasoning into Large Language Models by Learning from Knowledge Graphs [55.317267269115845]
  Chain-of-Knowledge (CoK) is a comprehensive framework for knowledge reasoning. CoK includes methodologies for both dataset construction and model learning. We conduct extensive experiments with KnowReason.
  arXiv  Detail & Related papers  (2024-06-30T10:49:32Z)
• Large Language Models are Limited in Out-of-Context Knowledge Reasoning [65.72847298578071]
  Large Language Models (LLMs) possess extensive knowledge and strong capabilities in performing in-context reasoning. This paper focuses on a significant aspect of out-of-context reasoning: Out-of-Context Knowledge Reasoning (OCKR), which is to combine multiple knowledge to infer new knowledge.
  arXiv  Detail & Related papers  (2024-06-11T15:58:59Z)
• Towards Knowledge-Grounded Natural Language Understanding and Generation [1.450405446885067]
  This thesis investigates how natural language understanding and generation with transformer models can benefit from grounding the models with knowledge representations. Studies in this thesis find that incorporating relevant and up-to-date knowledge of entities benefits fake news detection. It is established that other general forms of knowledge, such as parametric and distilled knowledge, enhance multimodal and multilingual knowledge-intensive tasks.
  arXiv  Detail & Related papers  (2024-03-22T17:32:43Z)
• Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators [78.63553017938911]
  Large language models (LLMs) outperform information retrieval techniques for downstream knowledge-intensive tasks. However, community concerns abound regarding the factuality and potential implications of using this uncensored knowledge. We introduce CONNER, designed to evaluate generated knowledge from six important perspectives.
  arXiv  Detail & Related papers  (2023-10-11T08:22:37Z)
• Knowledge Rumination for Pre-trained Language Models [77.55888291165462]
  We propose a new paradigm dubbed Knowledge Rumination to help the pre-trained language model utilize related latent knowledge without retrieving it from the external corpus. We apply the proposed knowledge rumination to various language models, including RoBERTa, DeBERTa, and GPT-3.
  arXiv  Detail & Related papers  (2023-05-15T15:47:09Z)
• UNTER: A Unified Knowledge Interface for Enhancing Pre-trained Language Models [100.4659557650775]
  We propose a UNified knowledge inTERface, UNTER, to provide a unified perspective to exploit both structured knowledge and unstructured knowledge. With both forms of knowledge injected, UNTER gains continuous improvements on a series of knowledge-driven NLP tasks.
  arXiv  Detail & Related papers  (2023-05-02T17:33:28Z)
• LM-CORE: Language Models with Contextually Relevant External Knowledge [13.451001884972033]
  We argue that storing large amounts of knowledge in the model parameters is sub-optimal given the ever-growing amounts of knowledge and resource requirements. We present LM-CORE -- a general framework to achieve this -- that allows textitdecoupling of the language model training from the external knowledge source. Experimental results show that LM-CORE, having access to external knowledge, achieves significant and robust outperformance over state-of-the-art knowledge-enhanced language models on knowledge probing tasks.
  arXiv  Detail & Related papers  (2022-08-12T18:59:37Z)
• Generated Knowledge Prompting for Commonsense Reasoning [53.88983683513114]
  We propose generating knowledge statements directly from a language model with a generic prompt format. This approach improves performance of both off-the-shelf and finetuned language models on four commonsense reasoning tasks. Notably, we find that a model's predictions can improve when using its own generated knowledge.
  arXiv  Detail & Related papers  (2021-10-15T21:58:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.