Memorizing is Not Enough: Deep Knowledge Injection Through Reasoning

• URL: http://arxiv.org/abs/2504.00472v1
• Date: Tue, 01 Apr 2025 06:59:59 GMT
• Title: Memorizing is Not Enough: Deep Knowledge Injection Through Reasoning
• Authors: Ruoxi Xu, Yunjie Ji, Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Ben He, Yingfei Sun, Xiangang Li, Le Sun,
• Abstract summary: This paper proposes a four-tier knowledge injection framework that defines the levels of knowledge injection: memorization, retrieval, reasoning, and association.<n>We then explore various knowledge injection scenarios and evaluate the depth of knowledge injection for each scenario on the benchmark.
• Score: 60.01714908976762
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Although large language models (LLMs) excel in knowledge recall and reasoning, their static nature leads to outdated information as the real world evolves or when adapting to domain-specific knowledge, highlighting the need for effective knowledge injection. However, current research on knowledge injection remains superficial, mainly focusing on knowledge memorization and retrieval. This paper proposes a four-tier knowledge injection framework that systematically defines the levels of knowledge injection: memorization, retrieval, reasoning, and association. Based on this framework, we introduce DeepKnowledge, a synthetic experimental testbed designed for fine-grained evaluation of the depth of knowledge injection across three knowledge types (novel, incremental, and updated). We then explore various knowledge injection scenarios and evaluate the depth of knowledge injection for each scenario on the benchmark. Experimental results reveal key factors to reach each level of knowledge injection for LLMs and establish a mapping between the levels of knowledge injection and the corresponding suitable injection methods, aiming to provide a comprehensive approach for efficient knowledge injection across various levels.

Related papers

• Deepfake Detection via Knowledge Injection [35.95604525443886]
  Existing deepfake detection methods tend to overlook the essential role of real data knowledge.<n>A knowledge injection module is proposed to learn and inject necessary knowledge into the backbone model.<n>Two layer-wise suppression and contrast losses are proposed to emphasize the knowledge of real data in the knowledge injection module.
  arXiv  Detail & Related papers  (2025-03-04T11:11:14Z)
• Latent Paraphrasing: Perturbation on Layers Improves Knowledge Injection in Language Models [54.385486006684495]
  LaPael is a latent-level paraphrasing method that applies input-dependent noise to early Large Language Models layers. Our experiments on question-answering benchmarks demonstrate that LaPael improves knowledge injection over standard fine-tuning and existing noise-based approaches.
  arXiv  Detail & Related papers  (2024-11-01T15:47:05Z)
• Synthetic Knowledge Ingestion: Towards Knowledge Refinement and Injection for Enhancing Large Language Models [1.753683416932648]
  Large language models (LLMs) are proficient in capturing factual knowledge across various domains. In this work, we propose a novel synthetic knowledge ingestion method called Ski. We then integrate Ski and its variations with three knowledge injection techniques to inject and refine knowledge in language models.
  arXiv  Detail & Related papers  (2024-10-12T19:38:09Z)
• Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators [78.63553017938911]
  Large language models (LLMs) outperform information retrieval techniques for downstream knowledge-intensive tasks. However, community concerns abound regarding the factuality and potential implications of using this uncensored knowledge. We introduce CONNER, designed to evaluate generated knowledge from six important perspectives.
  arXiv  Detail & Related papers  (2023-10-11T08:22:37Z)
• Deep Learning-Based Knowledge Injection for Metaphor Detection: A Comprehensive Review [24.968400793968417]
  This paper provides a review of research advances in the application of deep learning for knowledge injection in metaphor detection tasks. We will first systematically summarize and generalize the mainstream knowledge and knowledge injection principles. Then, the datasets, evaluation metrics, and benchmark models used in metaphor detection tasks are examined.
  arXiv  Detail & Related papers  (2023-08-08T14:51:16Z)
• UNTER: A Unified Knowledge Interface for Enhancing Pre-trained Language Models [100.4659557650775]
  We propose a UNified knowledge inTERface, UNTER, to provide a unified perspective to exploit both structured knowledge and unstructured knowledge. With both forms of knowledge injected, UNTER gains continuous improvements on a series of knowledge-driven NLP tasks.
  arXiv  Detail & Related papers  (2023-05-02T17:33:28Z)
• Knowledge-augmented Deep Learning and Its Applications: A Survey [60.221292040710885]
  knowledge-augmented deep learning (KADL) aims to identify domain knowledge and integrate it into deep models for data-efficient, generalizable, and interpretable deep learning. This survey subsumes existing works and offers a bird's-eye view of research in the general area of knowledge-augmented deep learning.
  arXiv  Detail & Related papers  (2022-11-30T03:44:15Z)
• Informed Learning by Wide Neural Networks: Convergence, Generalization and Sampling Complexity [27.84415856657607]
  We study how and why domain knowledge benefits the performance of informed learning. We propose a generalized informed training objective to better exploit the benefits of knowledge and balance the label and knowledge imperfectness.
  arXiv  Detail & Related papers  (2022-07-02T06:28:25Z)
• A Unified End-to-End Retriever-Reader Framework for Knowledge-based VQA [67.75989848202343]
  This paper presents a unified end-to-end retriever-reader framework towards knowledge-based VQA. We shed light on the multi-modal implicit knowledge from vision-language pre-training models to mine its potential in knowledge reasoning. Our scheme is able to not only provide guidance for knowledge retrieval, but also drop these instances potentially error-prone towards question answering.
  arXiv  Detail & Related papers  (2022-06-30T02:35:04Z)
• Ontology-enhanced Prompt-tuning for Few-shot Learning [41.51144427728086]
  Few-shot Learning is aimed to make predictions based on a limited number of samples. Structured data such as knowledge graphs and ontology libraries has been leveraged to benefit the few-shot setting in various tasks.
  arXiv  Detail & Related papers  (2022-01-27T05:41:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.