A Pilot Empirical Study on When and How to Use Knowledge Graphs as Retrieval Augmented Generation

• URL: http://arxiv.org/abs/2502.20854v2
• Date: Mon, 03 Mar 2025 03:00:59 GMT
• Title: A Pilot Empirical Study on When and How to Use Knowledge Graphs as Retrieval Augmented Generation
• Authors: Xujie Yuan, Yongxu Liu, Shimin Di, Shiwen Wu, Libin Zheng, Rui Meng, Lei Chen, Xiaofang Zhou, Jian Yin,
• Abstract summary: This paper lays the foundation for systematically answering the question of when and how to use KG-RAG.<n>We conduct a pilot empirical study of KG-RAG works to reimplement and evaluate 6 KG-RAG methods across 7 datasets in diverse scenarios.<n>Our results underscore the critical role of appropriate application conditions and optimal configurations of KG-RAG components.
• Score: 28.099850239183638
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The integration of Knowledge Graphs (KGs) into the Retrieval Augmented Generation (RAG) framework has attracted significant interest, with early studies showing promise in mitigating hallucinations and improving model accuracy. However, a systematic understanding and comparative analysis of the rapidly emerging KG-RAG methods are still lacking. This paper seeks to lay the foundation for systematically answering the question of when and how to use KG-RAG by analyzing their performance in various application scenarios associated with different technical configurations. After outlining the mind map using KG-RAG framework and summarizing its popular pipeline, we conduct a pilot empirical study of KG-RAG works to reimplement and evaluate 6 KG-RAG methods across 7 datasets in diverse scenarios, analyzing the impact of 9 KG-RAG configurations in combination with 17 LLMs. Our results underscore the critical role of appropriate application conditions and optimal configurations of KG-RAG components.

Related papers

• RAKG:Document-level Retrieval Augmented Knowledge Graph Construction [10.013667560362565]
  This paper focuses on the task of automatic document-level knowledge graph construction. It proposes the Document-level Retrieval Augmented Knowledge Graph Construction (RAKG) framework.
  arXiv  Detail & Related papers  (2025-04-14T02:47:23Z)
• Evaluating Knowledge Graph Based Retrieval Augmented Generation Methods under Knowledge Incompleteness [25.74411097212245]
  Knowledge Graph based Retrieval-Augmented Generation (KG-RAG) is a technique that enhances Large Language Model (LLM) inference in tasks like Question Answering (QA) Existing benchmarks do not adequately capture the impact of KG incompleteness on KG-RAG performance. We demonstrate that KG-RAG methods are sensitive to KG incompleteness, highlighting the need for more robust approaches in realistic settings.
  arXiv  Detail & Related papers  (2025-04-07T15:08:03Z)
• Enhancing Large Language Models (LLMs) for Telecommunications using Knowledge Graphs and Retrieval-Augmented Generation [52.8352968531863]
  Large language models (LLMs) have made significant progress in general-purpose natural language processing tasks. This paper presents a novel framework that combines knowledge graph (KG) and retrieval-augmented generation (RAG) techniques to enhance LLM performance in the telecom domain.
  arXiv  Detail & Related papers  (2025-03-31T15:58:08Z)
• KG-IRAG: A Knowledge Graph-Based Iterative Retrieval-Augmented Generation Framework for Temporal Reasoning [18.96570718233786]
  GraphRAG has proven highly effective in enhancing the performance of Large Language Models (LLMs) on tasks that require external knowledge. This paper presents Knowledge Graph-Based Iterative Retrieval-Augmented Generation (KG-IRAG), a novel framework that integrates KGs with iterative reasoning. Three new datasets are formed to evaluate KG-IRAG's performance, demonstrating its potential beyond traditional RAG applications.
  arXiv  Detail & Related papers  (2025-03-18T13:11:43Z)
• Knowledge Graph-Guided Retrieval Augmented Generation [34.83235788116369]
  We propose a Knowledge Graph-Guided Retrieval Augmented Generation framework.<n> KG$2$RAG provides fact-level relationships between chunks, improving the diversity and coherence of the retrieved results.
  arXiv  Detail & Related papers  (2025-02-08T02:14:31Z)
• GFM-RAG: Graph Foundation Model for Retrieval Augmented Generation [84.41557981816077]
  We introduce GFM-RAG, a novel graph foundation model (GFM) for retrieval augmented generation.<n>GFM-RAG is powered by an innovative graph neural network that reasons over graph structure to capture complex query-knowledge relationships.<n>It achieves state-of-the-art performance while maintaining efficiency and alignment with neural scaling laws.
  arXiv  Detail & Related papers  (2025-02-03T07:04:29Z)
• Chain-of-Retrieval Augmented Generation [72.06205327186069]
  This paper introduces an approach for training o1-like RAG models that retrieve and reason over relevant information step by step before generating the final answer.<n>Our proposed method, CoRAG, allows the model to dynamically reformulate the query based on the evolving state.
  arXiv  Detail & Related papers  (2025-01-24T09:12:52Z)
• Distill-SynthKG: Distilling Knowledge Graph Synthesis Workflow for Improved Coverage and Efficiency [59.6772484292295]
  Knowledge graphs (KGs) generated by large language models (LLMs) are increasingly valuable for Retrieval-Augmented Generation (RAG) applications. Existing KG extraction methods rely on prompt-based approaches, which are inefficient for processing large-scale corpora. We propose SynthKG, a multi-step, document-level synthesis KG workflow based on LLMs. We also design a novel graph-based retrieval framework for RAG.
  arXiv  Detail & Related papers  (2024-10-22T00:47:54Z)
• KG-FIT: Knowledge Graph Fine-Tuning Upon Open-World Knowledge [63.19837262782962]
  Knowledge Graph Embedding (KGE) techniques are crucial in learning compact representations of entities and relations within a knowledge graph. This study introduces KG-FIT, which builds a semantically coherent hierarchical structure of entity clusters. Experiments on the benchmark datasets FB15K-237, YAGO3-10, and PrimeKG demonstrate the superiority of KG-FIT over state-of-the-art pre-trained language model-based methods.
  arXiv  Detail & Related papers  (2024-05-26T03:04:26Z)
• How to Build an Adaptive AI Tutor for Any Course Using Knowledge Graph-Enhanced Retrieval-Augmented Generation (KG-RAG) [5.305156933641317]
  Large Language Models (LLMs) in Intelligent Tutoring Systems (ITS) presents transformative opportunities for personalized education.<n>Current implementations face two critical challenges: maintaining factual accuracy and delivering coherent, context-aware instruction.<n>This paper introduces Knowledge Graph-enhanced Retrieval-Augmented Generation (RAG), a novel framework that integrates structured knowledge representation with context-aware retrieval.
  arXiv  Detail & Related papers  (2023-11-29T15:02:46Z)
• Bring Your Own KG: Self-Supervised Program Synthesis for Zero-Shot KGQA [16.248395545151745]
  BYOKG is a universal question-answering (QA) system that can operate on any knowledge graph (KG) BYOKG draws inspiration from the remarkable ability of humans to comprehend information present in an unseen KG through exploration.
  arXiv  Detail & Related papers  (2023-11-14T02:05:29Z)
• KG-GPT: A General Framework for Reasoning on Knowledge Graphs Using Large Language Models [18.20425100517317]
  We propose KG-GPT, a framework leveraging large language models for tasks employing knowledge graphs. KG-GPT comprises three steps: Sentence, Graph Retrieval, and Inference, each aimed at partitioning sentences, retrieving relevant graph components, and deriving logical conclusions. We evaluate KG-GPT using KG-based fact verification and KGQA benchmarks, with the model showing competitive and robust performance, even outperforming several fully-supervised models.
  arXiv  Detail & Related papers  (2023-10-17T12:51:35Z)
• Reasoning over Multi-view Knowledge Graphs [59.99051368907095]
  ROMA is a novel framework for answering logical queries over multi-view KGs. It scales up to KGs of large sizes (e.g., millions of facts) and fine-granular views. It generalizes to query structures and KG views that are unobserved during training.
  arXiv  Detail & Related papers  (2022-09-27T21:32:20Z)
• DSKReG: Differentiable Sampling on Knowledge Graph for Recommendation with Relational GNN [59.160401038969795]
  We propose differentiable sampling on Knowledge Graph for Recommendation with GNN (DSKReG) We devise a differentiable sampling strategy, which enables the selection of relevant items to be jointly optimized with the model training procedure. The experimental results demonstrate that our model outperforms state-of-the-art KG-based recommender systems.
  arXiv  Detail & Related papers  (2021-08-26T16:19:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.