When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation

• URL: http://arxiv.org/abs/2506.05690v1
• Date: Fri, 06 Jun 2025 02:37:47 GMT
• Title: When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation
• Authors: Zhishang Xiang, Chuanjie Wu, Qinggang Zhang, Shengyuan Chen, Zijin Hong, Xiao Huang, Jinsong Su,
• Abstract summary: Graph retrieval-augmented generation (GraphRAG) has emerged as a powerful paradigm for enhancing large language models (LLMs) with external knowledge.<n>Recent studies report that GraphRAG frequently underperforms vanilla RAG on many real-world tasks.<n>This raises a critical question: Is GraphRAG really effective, and in which scenarios do graph structures provide measurable benefits for RAG systems?
• Score: 25.508719115522645
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Graph retrieval-augmented generation (GraphRAG) has emerged as a powerful paradigm for enhancing large language models (LLMs) with external knowledge. It leverages graphs to model the hierarchical structure between specific concepts, enabling more coherent and effective knowledge retrieval for accurate reasoning.Despite its conceptual promise, recent studies report that GraphRAG frequently underperforms vanilla RAG on many real-world tasks. This raises a critical question: Is GraphRAG really effective, and in which scenarios do graph structures provide measurable benefits for RAG systems? To address this, we propose GraphRAG-Bench, a comprehensive benchmark designed to evaluate GraphRAG models onboth hierarchical knowledge retrieval and deep contextual reasoning. GraphRAG-Bench features a comprehensive dataset with tasks of increasing difficulty, coveringfact retrieval, complex reasoning, contextual summarization, and creative generation, and a systematic evaluation across the entire pipeline, from graph constructionand knowledge retrieval to final generation. Leveraging this novel benchmark, we systematically investigate the conditions when GraphRAG surpasses traditional RAG and the underlying reasons for its success, offering guidelines for its practical application. All related resources and analyses are collected for the community at https://github.com/GraphRAG-Bench/GraphRAG-Benchmark.

Related papers

• Graph-R1: Towards Agentic GraphRAG Framework via End-to-end Reinforcement Learning [20.05893083101089]
  Graph-R1 is an agentic GraphRAG framework via end-to-end reinforcement learning (RL)<n>It introduces lightweight knowledge hypergraph construction, models retrieval as a multi-turn agent-environment interaction.<n>Experiments on standard RAG datasets show that Graph-R1 outperforms traditional GraphRAG and RL-enhanced RAG methods in reasoning accuracy, retrieval efficiency, and generation quality.
  arXiv  Detail & Related papers  (2025-07-29T15:01:26Z)
• XGraphRAG: Interactive Visual Analysis for Graph-based Retrieval-Augmented Generation [16.068460356582648]
  This research proposes a visual analysis framework that helps RAG developers identify critical recalls of GraphRAG.<n>We develop XGraphRAG, a prototype system incorporating a set of interactive visualizations to facilitate users' analysis process.
  arXiv  Detail & Related papers  (2025-06-10T09:14:30Z)
• GraphRAG-Bench: Challenging Domain-Specific Reasoning for Evaluating Graph Retrieval-Augmented Generation [26.654064783342545]
  Graph Retrieval Augmented Generation (GraphRAG) has garnered increasing recognition for its potential to enhance large language models (LLMs)<n>Current evaluations of GraphRAG models predominantly rely on traditional question-answering datasets.<n>We introduce GraphRAG-Bench, a large-scale, domain-specific benchmark designed to rigorously evaluate GraphRAG models.
  arXiv  Detail & Related papers  (2025-06-03T03:44:26Z)
• Align-GRAG: Reasoning-Guided Dual Alignment for Graph Retrieval-Augmented Generation [75.9865035064794]
  Large language models (LLMs) have demonstrated remarkable capabilities, but still struggle with issues like hallucinations and outdated information.<n>Retrieval-augmented generation (RAG) addresses these issues by grounding LLM outputs in external knowledge with an Information Retrieval (IR) system.<n>We propose Align-GRAG, a novel reasoning-guided dual alignment framework in post-retrieval phrase.
  arXiv  Detail & Related papers  (2025-05-22T05:15:27Z)
• RGL: A Graph-Centric, Modular Framework for Efficient Retrieval-Augmented Generation on Graphs [58.10503898336799]
  We introduce the RAG-on-Graphs Library (RGL), a modular framework that seamlessly integrates the complete RAG pipeline.<n>RGL addresses key challenges by supporting a variety of graph formats and integrating optimized implementations for essential components.<n>Our evaluations demonstrate that RGL not only accelerates the prototyping process but also enhances the performance and applicability of graph-based RAG systems.
  arXiv  Detail & Related papers  (2025-03-25T03:21:48Z)
• G-OSR: A Comprehensive Benchmark for Graph Open-Set Recognition [54.45837774534411]
  We introduce textbfG-OSR, a benchmark for evaluating Graph Open-Set Recognition (GOSR) methods at both the node and graph levels.<n>Results offer critical insights into the generalizability and limitations of current GOSR methods.
  arXiv  Detail & Related papers  (2025-03-01T13:02:47Z)
• RAG vs. GraphRAG: A Systematic Evaluation and Key Insights [42.31801859160484]
  We systematically evaluate Retrieval-Augmented Generation (RAG) and GraphRAG on text-based benchmarks.<n>Our results highlight the distinct strengths of RAG and GraphRAG across different tasks and evaluation perspectives.
  arXiv  Detail & Related papers  (2025-02-17T02:36:30Z)
• Revisiting Graph Neural Networks on Graph-level Tasks: Comprehensive Experiments, Analysis, and Improvements [54.006506479865344]
  We propose a unified evaluation framework for graph-level Graph Neural Networks (GNNs)<n>This framework provides a standardized setting to evaluate GNNs across diverse datasets.<n>We also propose a novel GNN model with enhanced expressivity and generalization capabilities.
  arXiv  Detail & Related papers  (2025-01-01T08:48:53Z)
• Retrieval-Augmented Generation with Graphs (GraphRAG) [84.29507404866257]
  Retrieval-augmented generation (RAG) is a powerful technique that enhances downstream task execution by retrieving additional information.<n>Graph, by its intrinsic "nodes connected by edges" nature, encodes massive heterogeneous and relational information.<n>Unlike conventional RAG, the uniqueness of graph-structured data, such as diverse-formatted and domain-specific relational knowledge, poses unique and significant challenges when designing GraphRAG for different domains.
  arXiv  Detail & Related papers  (2024-12-31T06:59:35Z)
• LEGO-GraphRAG: Modularizing Graph-based Retrieval-Augmented Generation for Design Space Exploration [17.514586423233872]
  We propose LEGO-GraphRAG, a modular framework that enables fine-grained decomposition of the GraphRAG workflow.<n>Our framework facilitates comprehensive empirical studies of GraphRAG on large-scale real-world graphs and diverse query sets.
  arXiv  Detail & Related papers  (2024-11-06T15:32:28Z)
• Graph Retrieval-Augmented Generation: A Survey [28.979898837538958]
  Retrieval-Augmented Generation (RAG) has achieved remarkable success in addressing the challenges of Large Language Models (LLMs) without necessitating retraining. This paper provides the first comprehensive overview of GraphRAG methodologies. We formalize the GraphRAG workflow, encompassing Graph-Based Indexing, Graph-Guided Retrieval, and Graph-Enhanced Generation.
  arXiv  Detail & Related papers  (2024-08-15T12:20:24Z)
• GC-Bench: An Open and Unified Benchmark for Graph Condensation [54.70801435138878]
  We develop a comprehensive Graph Condensation Benchmark (GC-Bench) to analyze the performance of graph condensation. GC-Bench systematically investigates the characteristics of graph condensation in terms of the following dimensions: effectiveness, transferability, and complexity. We have developed an easy-to-use library for training and evaluating different GC methods to facilitate reproducible research.
  arXiv  Detail & Related papers  (2024-06-30T07:47:34Z)
• G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering [61.93058781222079]
  We develop a flexible question-answering framework targeting real-world textual graphs. We introduce the first retrieval-augmented generation (RAG) approach for general textual graphs. G-Retriever performs RAG over a graph by formulating this task as a Prize-Collecting Steiner Tree optimization problem.
  arXiv  Detail & Related papers  (2024-02-12T13:13:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.