RAG vs. GraphRAG: A Systematic Evaluation and Key Insights

• URL: http://arxiv.org/abs/2502.11371v1
• Date: Mon, 17 Feb 2025 02:36:30 GMT
• Title: RAG vs. GraphRAG: A Systematic Evaluation and Key Insights
• Authors: Haoyu Han, Harry Shomer, Yu Wang, Yongjia Lei, Kai Guo, Zhigang Hua, Bo Long, Hui Liu, Jiliang Tang,
• Abstract summary: We systematically evaluate Retrieval-Augmented Generation (RAG) and GraphRAG on text-based benchmarks. Our results highlight the distinct strengths of RAG and GraphRAG across different tasks and evaluation perspectives.
• Score: 42.31801859160484
• License:
• Abstract: Retrieval-Augmented Generation (RAG) enhances the performance of LLMs across various tasks by retrieving relevant information from external sources, particularly on text-based data. For structured data, such as knowledge graphs, GraphRAG has been widely used to retrieve relevant information. However, recent studies have revealed that structuring implicit knowledge from text into graphs can benefit certain tasks, extending the application of GraphRAG from graph data to general text-based data. Despite their successful extensions, most applications of GraphRAG for text data have been designed for specific tasks and datasets, lacking a systematic evaluation and comparison between RAG and GraphRAG on widely used text-based benchmarks. In this paper, we systematically evaluate RAG and GraphRAG on well-established benchmark tasks, such as Question Answering and Query-based Summarization. Our results highlight the distinct strengths of RAG and GraphRAG across different tasks and evaluation perspectives. Inspired by these observations, we investigate strategies to integrate their strengths to improve downstream tasks. Additionally, we provide an in-depth discussion of the shortcomings of current GraphRAG approaches and outline directions for future research.

Related papers

• CG-RAG: Research Question Answering by Citation Graph Retrieval-Augmented LLMs [9.718354494802002]
  Contextualized Graph Retrieval-Augmented Generation (CG-RAG) is a novel framework that integrates sparse and dense retrieval signals within graph structures. First, we propose a contextual graph representation for citation graphs, effectively capturing both explicit and implicit connections within and across documents. Second, we introduce Lexical-Semantic Graph Retrieval (LeSeGR), which seamlessly integrates sparse and dense retrieval signals with graph encoding. Third, we present a context-aware generation strategy that utilizes the retrieved graph-structured information to generate precise and contextually enriched responses.
  arXiv  Detail & Related papers  (2025-01-25T04:18:08Z)
• Revisiting Graph Neural Networks on Graph-level Tasks: Comprehensive Experiments, Analysis, and Improvements [54.006506479865344]
  We propose a unified evaluation framework for graph-level Graph Neural Networks (GNNs) This framework provides a standardized setting to evaluate GNNs across diverse datasets. We also propose a novel GNN model with enhanced expressivity and generalization capabilities.
  arXiv  Detail & Related papers  (2025-01-01T08:48:53Z)
• Retrieval-Augmented Generation with Graphs (GraphRAG) [84.29507404866257]
  Retrieval-augmented generation (RAG) is a powerful technique that enhances downstream task execution by retrieving additional information. Graph, by its intrinsic "nodes connected by edges" nature, encodes massive heterogeneous and relational information. Unlike conventional RAG, the uniqueness of graph-structured data, such as diverse-formatted and domain-specific relational knowledge, poses unique and significant challenges when designing GraphRAG for different domains.
  arXiv  Detail & Related papers  (2024-12-31T06:59:35Z)
• LEGO-GraphRAG: Modularizing Graph-based Retrieval-Augmented Generation for Design Space Exploration [17.514586423233872]
  We propose LEGO-GraphRAG, a modular framework that enables fine-grained decomposition of the GraphRAG workflow. Our framework facilitates comprehensive empirical studies of GraphRAG on large-scale real-world graphs and diverse query sets.
  arXiv  Detail & Related papers  (2024-11-06T15:32:28Z)
• RAGraph: A General Retrieval-Augmented Graph Learning Framework [35.25522856244149]
  We introduce a novel framework called General Retrieval-Augmented Graph Learning (RAGraph) RAGraph brings external graph data into the general graph foundation model to improve model generalization on unseen scenarios. During inference, the RAGraph adeptly retrieves similar toy graphs based on key similarities in downstream tasks.
  arXiv  Detail & Related papers  (2024-10-31T12:05:21Z)
• Graph Retrieval-Augmented Generation: A Survey [28.979898837538958]
  Retrieval-Augmented Generation (RAG) has achieved remarkable success in addressing the challenges of Large Language Models (LLMs) without necessitating retraining. This paper provides the first comprehensive overview of GraphRAG methodologies. We formalize the GraphRAG workflow, encompassing Graph-Based Indexing, Graph-Guided Retrieval, and Graph-Enhanced Generation.
  arXiv  Detail & Related papers  (2024-08-15T12:20:24Z)
• G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering [61.93058781222079]
  We develop a flexible question-answering framework targeting real-world textual graphs. We introduce the first retrieval-augmented generation (RAG) approach for general textual graphs. G-Retriever performs RAG over a graph by formulating this task as a Prize-Collecting Steiner Tree optimization problem.
  arXiv  Detail & Related papers  (2024-02-12T13:13:04Z)
• Challenging the Myth of Graph Collaborative Filtering: a Reasoned and Reproducibility-driven Analysis [50.972595036856035]
  We present a code that successfully replicates results from six popular and recent graph recommendation models. We compare these graph models with traditional collaborative filtering models that historically performed well in offline evaluations. By investigating the information flow from users' neighborhoods, we aim to identify which models are influenced by intrinsic features in the dataset structure.
  arXiv  Detail & Related papers  (2023-08-01T09:31:44Z)
• Graph Learning based Recommender Systems: A Review [111.43249652335555]
  Graph Learning based Recommender Systems (GLRS) employ advanced graph learning approaches to model users' preferences and intentions as well as items' characteristics for recommendations. We provide a systematic review of GLRS, by discussing how they extract important knowledge from graph-based representations to improve the accuracy, reliability and explainability of the recommendations.
  arXiv  Detail & Related papers  (2021-05-13T14:50:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.