Chain-of-Knowledge: Integrating Knowledge Reasoning into Large Language Models by Learning from Knowledge Graphs

• URL: http://arxiv.org/abs/2407.00653v1
• Date: Sun, 30 Jun 2024 10:49:32 GMT
• Title: Chain-of-Knowledge: Integrating Knowledge Reasoning into Large Language Models by Learning from Knowledge Graphs
• Authors: Yifei Zhang, Xintao Wang, Jiaqing Liang, Sirui Xia, Lida Chen, Yanghua Xiao,
• Abstract summary: Chain-of-Knowledge (CoK) is a comprehensive framework for knowledge reasoning. CoK includes methodologies for both dataset construction and model learning. We conduct extensive experiments with KnowReason.
• Score: 55.317267269115845
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) have exhibited impressive proficiency in various natural language processing (NLP) tasks, which involve increasingly complex reasoning. Knowledge reasoning, a primary type of reasoning, aims at deriving new knowledge from existing one.While it has been widely studied in the context of knowledge graphs (KGs), knowledge reasoning in LLMs remains underexplored. In this paper, we introduce Chain-of-Knowledge, a comprehensive framework for knowledge reasoning, including methodologies for both dataset construction and model learning. For dataset construction, we create KnowReason via rule mining on KGs. For model learning, we observe rule overfitting induced by naive training. Hence, we enhance CoK with a trial-and-error mechanism that simulates the human process of internal knowledge exploration. We conduct extensive experiments with KnowReason. Our results show the effectiveness of CoK in refining LLMs in not only knowledge reasoning, but also general reasoning benchmarkms.

Related papers

• Knowledge Mechanisms in Large Language Models: A Survey and Perspective [88.51320482620679]
  This paper reviews knowledge mechanism analysis from a novel taxonomy including knowledge utilization and evolution. We discuss what knowledge LLMs have learned, the reasons for the fragility of parametric knowledge, and the potential dark knowledge (hypothesis) that will be challenging to address.
  arXiv  Detail & Related papers  (2024-07-22T06:15:59Z)
• Large Language Models are Limited in Out-of-Context Knowledge Reasoning [65.72847298578071]
  Large Language Models (LLMs) possess extensive knowledge and strong capabilities in performing in-context reasoning. This paper focuses on a significant aspect of out-of-context reasoning: Out-of-Context Knowledge Reasoning (OCKR), which is to combine multiple knowledge to infer new knowledge.
  arXiv  Detail & Related papers  (2024-06-11T15:58:59Z)
• Large Knowledge Model: Perspectives and Challenges [37.42721596964844]
  emphLarge Language Models (LLMs) epitomize the pre-training of extensive, sequence-based world knowledge into neural networks. This article explores large models through the lens of "knowledge" Considering the intricate nature of human knowledge, we advocate for the creation of emphLarge Knowledge Models (LKM)
  arXiv  Detail & Related papers  (2023-12-05T12:07:30Z)
• Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators [78.63553017938911]
  Large language models (LLMs) outperform information retrieval techniques for downstream knowledge-intensive tasks. However, community concerns abound regarding the factuality and potential implications of using this uncensored knowledge. We introduce CONNER, designed to evaluate generated knowledge from six important perspectives.
  arXiv  Detail & Related papers  (2023-10-11T08:22:37Z)
• Knowledge Rumination for Pre-trained Language Models [77.55888291165462]
  We propose a new paradigm dubbed Knowledge Rumination to help the pre-trained language model utilize related latent knowledge without retrieving it from the external corpus. We apply the proposed knowledge rumination to various language models, including RoBERTa, DeBERTa, and GPT-3.
  arXiv  Detail & Related papers  (2023-05-15T15:47:09Z)
• Learning by Applying: A General Framework for Mathematical Reasoning via Enhancing Explicit Knowledge Learning [47.96987739801807]
  We propose a framework to enhance existing models (backbones) in a principled way by explicit knowledge learning. In LeAp, we perform knowledge learning in a novel problem-knowledge-expression paradigm. We show that LeAp improves all backbones' performances, learns accurate knowledge, and achieves a more interpretable reasoning process.
  arXiv  Detail & Related papers  (2023-02-11T15:15:41Z)
• Structured Knowledge Grounding for Question Answering [0.23068481501673416]
  We propose to leverage the language and knowledge for knowledge based question-answering with flexibility, breadth of coverage and structured reasoning. Specifically, we devise a knowledge construction method that retrieves the relevant context with a dynamic hop. And we devise a deep fusion mechanism to further bridge the information exchanging bottleneck between the language and the knowledge.
  arXiv  Detail & Related papers  (2022-09-17T08:48:50Z)
• Lexical Knowledge Internalization for Neural Dialog Generation [36.27946635687281]
  We propose knowledge internalization (KI), which aims to complement the lexical knowledge into neural dialog models. To tackle the challenge due to the large scale of lexical knowledge, we adopt the contrastive learning approach and create an effective token-level lexical knowledge retriever.
  arXiv  Detail & Related papers  (2022-05-04T08:23:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.