Evaluating Knowledge Graph Based Retrieval Augmented Generation Methods under Knowledge Incompleteness

• URL: http://arxiv.org/abs/2504.05163v1
• Date: Mon, 07 Apr 2025 15:08:03 GMT
• Title: Evaluating Knowledge Graph Based Retrieval Augmented Generation Methods under Knowledge Incompleteness
• Authors: Dongzhuoran Zhou, Yuqicheng Zhu, Yuan He, Jiaoyan Chen, Evgeny Kharlamov, Steffen Staab,
• Abstract summary: Knowledge Graph based Retrieval-Augmented Generation (KG-RAG) is a technique that enhances Large Language Model (LLM) inference in tasks like Question Answering (QA)<n>Existing benchmarks do not adequately capture the impact of KG incompleteness on KG-RAG performance.<n>We demonstrate that KG-RAG methods are sensitive to KG incompleteness, highlighting the need for more robust approaches in realistic settings.
• Score: 25.74411097212245
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Knowledge Graph based Retrieval-Augmented Generation (KG-RAG) is a technique that enhances Large Language Model (LLM) inference in tasks like Question Answering (QA) by retrieving relevant information from knowledge graphs (KGs). However, real-world KGs are often incomplete, meaning that essential information for answering questions may be missing. Existing benchmarks do not adequately capture the impact of KG incompleteness on KG-RAG performance. In this paper, we systematically evaluate KG-RAG methods under incomplete KGs by removing triples using different methods and analyzing the resulting effects. We demonstrate that KG-RAG methods are sensitive to KG incompleteness, highlighting the need for more robust approaches in realistic settings.

Related papers

• Question-Aware Knowledge Graph Prompting for Enhancing Large Language Models [51.47994645529258]
  We propose Question-Aware Knowledge Graph Prompting (QAP), which incorporates question embeddings into GNN aggregation to dynamically assess KG relevance. Experimental results demonstrate that QAP outperforms state-of-the-art methods across multiple datasets, highlighting its effectiveness.
  arXiv  Detail & Related papers  (2025-03-30T17:09:11Z)
• A Pilot Empirical Study on When and How to Use Knowledge Graphs as Retrieval Augmented Generation [28.099850239183638]
  This paper lays the foundation for systematically answering the question of when and how to use KG-RAG. We conduct a pilot empirical study of KG-RAG works to reimplement and evaluate 6 KG-RAG methods across 7 datasets in diverse scenarios. Our results underscore the critical role of appropriate application conditions and optimal configurations of KG-RAG components.
  arXiv  Detail & Related papers  (2025-02-28T08:53:08Z)
• GLTW: Joint Improved Graph Transformer and LLM via Three-Word Language for Knowledge Graph Completion [52.026016846945424]
  We propose a new method called GLTW, which encodes the structural information of KGs and merges it with Large Language Models.<n>Specifically, we introduce an improved Graph Transformer (iGT) that effectively encodes subgraphs with both local and global structural information.<n>Also, we develop a subgraph-based multi-classification training objective, using all entities within KG as classification objects, to boost learning efficiency.
  arXiv  Detail & Related papers  (2025-02-17T06:02:59Z)
• Distill-SynthKG: Distilling Knowledge Graph Synthesis Workflow for Improved Coverage and Efficiency [59.6772484292295]
  Knowledge graphs (KGs) generated by large language models (LLMs) are increasingly valuable for Retrieval-Augmented Generation (RAG) applications. Existing KG extraction methods rely on prompt-based approaches, which are inefficient for processing large-scale corpora. We propose SynthKG, a multi-step, document-level synthesis KG workflow based on LLMs. We also design a novel graph-based retrieval framework for RAG.
  arXiv  Detail & Related papers  (2024-10-22T00:47:54Z)
• Context Graph [8.02985792541121]
  We present a context graph reasoning textbfCGR$3$ paradigm that leverages large language models (LLMs) to retrieve candidate entities and related contexts. Our experimental results demonstrate that CGR$3$ significantly improves performance on KG completion (KGC) and KG question answering (KGQA) tasks.
  arXiv  Detail & Related papers  (2024-06-17T02:59:19Z)
• KG-FIT: Knowledge Graph Fine-Tuning Upon Open-World Knowledge [63.19837262782962]
  Knowledge Graph Embedding (KGE) techniques are crucial in learning compact representations of entities and relations within a knowledge graph. This study introduces KG-FIT, which builds a semantically coherent hierarchical structure of entity clusters. Experiments on the benchmark datasets FB15K-237, YAGO3-10, and PrimeKG demonstrate the superiority of KG-FIT over state-of-the-art pre-trained language model-based methods.
  arXiv  Detail & Related papers  (2024-05-26T03:04:26Z)
• Generate-on-Graph: Treat LLM as both Agent and KG in Incomplete Knowledge Graph Question Answering [87.67177556994525]
  We propose a training-free method called Generate-on-Graph (GoG) to generate new factual triples while exploring Knowledge Graphs (KGs) GoG performs reasoning through a Thinking-Searching-Generating framework, which treats LLM as both Agent and KG in IKGQA.
  arXiv  Detail & Related papers  (2024-04-23T04:47:22Z)
• Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Framework for Knowledge Graph Question Answering [16.434098552925427]
  We study the KG-augmented language model approach for solving the knowledge graph question answering (KGQA) task. We propose an answer-sensitive KG-to-Text approach that can transform KG knowledge into well-textualized statements.
  arXiv  Detail & Related papers  (2023-09-20T10:42:08Z)
• Collective Knowledge Graph Completion with Mutual Knowledge Distillation [11.922522192224145]
  We study the problem of multi-KG completion, where we focus on maximizing the collective knowledge from different KGs. We propose a novel method called CKGC-CKD that uses relation-aware graph convolutional network encoder models on both individual KGs and a large fused KG. Experimental results on multilingual datasets have shown that our method outperforms all state-of-the-art models in the KGC task.
  arXiv  Detail & Related papers  (2023-05-25T09:49:40Z)
• Explainable Sparse Knowledge Graph Completion via High-order Graph Reasoning Network [111.67744771462873]
  This paper proposes a novel explainable model for sparse Knowledge Graphs (KGs) It combines high-order reasoning into a graph convolutional network, namely HoGRN. It can not only improve the generalization ability to mitigate the information insufficiency issue but also provide interpretability.
  arXiv  Detail & Related papers  (2022-07-14T10:16:56Z)
• IterefinE: Iterative KG Refinement Embeddings using Symbolic Knowledge [10.689559910656474]
  Knowledge Graphs (KGs) extracted from text sources are often noisy and lead to poor performance in downstream application tasks such as KG-based question answering. Most successful techniques for KG refinement make use of inference rules and reasoning oversupervised. In this paper, we present a KG refinement framework called IterefinE which iteratively combines the two techniques.
  arXiv  Detail & Related papers  (2020-06-03T14:05:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.