A Knowledge-Centric Benchmarking Framework and Empirical Study for Retrieval-Augmented Generation

• URL: http://arxiv.org/abs/2409.13694v1
• Date: Tue, 3 Sep 2024 03:31:37 GMT
• Title: A Knowledge-Centric Benchmarking Framework and Empirical Study for Retrieval-Augmented Generation
• Authors: Shuo Yu, Mingyue Cheng, Jiqian Yang, Jie Ouyang,
• Abstract summary: Retrieval-Augmented Generation (RAG) enhances generative models by integrating retrieval mechanisms. Despite its advantages, RAG encounters significant challenges, particularly in effectively handling real-world queries. This paper proposes a novel RAG benchmark designed to address these challenges.
• Score: 4.359511178431438
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Retrieval-Augmented Generation (RAG) enhances generative models by integrating retrieval mechanisms, which allow these models to access and utilize external knowledge sources. Despite its advantages, RAG encounters significant challenges, particularly in effectively handling real-world queries and mitigating hallucinations. The KDD Cup 2024 CRAG competition brings these issues to the forefront by incorporating both web pages and a mock API as knowledge sources, adding the complexity of parsing HTML before large language models (LLMs) can process the information. In this paper, we propose a novel RAG benchmark designed to address these challenges. Our work provides a comprehensive set of experimental results, offering valuable insights for the study of RAG. We thoroughly examine the entire RAG process, including knowledge source selection, retrieval, organization, and reasoning. Key findings from our study include the impact of automated knowledge source selection using agents and the influence of noise chunks on RAG reasoning. Additionally, we conduct detailed experiments to analyze the effects of various hyperparameters on RAG performance. To support further research, we have made our results, the associated code, and a parsed version of the CRAG dataset publicly available\footnote{https://github.com/USTCAGI/RAG-X}, contributing to the advancement of RAG methodologies and establishing a solid foundation for future work in this domain.

Related papers

• A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions [0.0]
  RAG combines retrieval mechanisms with generative language models to enhance the accuracy of outputs. Recent research breakthroughs are discussed, highlighting novel methods for improving retrieval efficiency. Future research directions are proposed, focusing on improving the robustness of RAG models.
  arXiv  Detail & Related papers  (2024-10-03T22:29:47Z)
• Trustworthiness in Retrieval-Augmented Generation Systems: A Survey [59.26328612791924]
  Retrieval-Augmented Generation (RAG) has quickly grown into a pivotal paradigm in the development of Large Language Models (LLMs) We propose a unified framework that assesses the trustworthiness of RAG systems across six key dimensions: factuality, robustness, fairness, transparency, accountability, and privacy.
  arXiv  Detail & Related papers  (2024-09-16T09:06:44Z)
• RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation [61.14660526363607]
  We propose a fine-grained evaluation framework, RAGChecker, that incorporates a suite of diagnostic metrics for both the retrieval and generation modules. RAGChecker has significantly better correlations with human judgments than other evaluation metrics. The metrics of RAGChecker can guide researchers and practitioners in developing more effective RAG systems.
  arXiv  Detail & Related papers  (2024-08-15T10:20:54Z)
• CRAG -- Comprehensive RAG Benchmark [58.15980697921195]
  Retrieval-Augmented Generation (RAG) has recently emerged as a promising solution to alleviate Large Language Model (LLM)'s deficiency in lack of knowledge. Existing RAG datasets do not adequately represent the diverse and dynamic nature of real-world Question Answering (QA) tasks. To bridge this gap, we introduce the Comprehensive RAG Benchmark (CRAG) CRAG is a factual question answering benchmark of 4,409 question-answer pairs and mock APIs to simulate web and Knowledge Graph (KG) search.
  arXiv  Detail & Related papers  (2024-06-07T08:43:07Z)
• DuetRAG: Collaborative Retrieval-Augmented Generation [57.440772556318926]
  Collaborative Retrieval-Augmented Generation framework, DuetRAG, proposed. bootstrapping philosophy is to simultaneously integrate the domain fintuning and RAG models.
  arXiv  Detail & Related papers  (2024-05-12T09:48:28Z)
• A Survey on Retrieval-Augmented Text Generation for Large Language Models [1.4579344926652844]
  Retrieval-Augmented Generation (RAG) merges retrieval methods with deep learning advancements. This paper organizes the RAG paradigm into four categories: pre-retrieval, retrieval, post-retrieval, and generation. It outlines RAG's evolution and discusses the field's progression through the analysis of significant studies.
  arXiv  Detail & Related papers  (2024-04-17T01:27:42Z)
• Retrieval-Augmented Generation for AI-Generated Content: A Survey [38.50754568320154]
  Retrieval-Augmented Generation (RAG) has emerged as a paradigm to address such challenges. RAG introduces the information retrieval process, which enhances the generation process by retrieving relevant objects from available data stores. In this paper, we comprehensively review existing efforts that integrate RAG technique into AIGC scenarios.
  arXiv  Detail & Related papers  (2024-02-29T18:59:01Z)
• REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answering [122.62012375722124]
  In existing methods, large language models (LLMs) cannot precisely assess the relevance of retrieved documents. We propose REAR, a RElevance-Aware Retrieval-augmented approach for open-domain question answering (QA)
  arXiv  Detail & Related papers  (2024-02-27T13:22:51Z)
• CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models [49.16989035566899]
  Retrieval-Augmented Generation (RAG) is a technique that enhances the capabilities of large language models (LLMs) by incorporating external knowledge sources. This paper constructs a large-scale and more comprehensive benchmark, and evaluates all the components of RAG systems in various RAG application scenarios.
  arXiv  Detail & Related papers  (2024-01-30T14:25:32Z)
• Retrieval-Augmented Generation for Large Language Models: A Survey [17.82361213043507]
  Large Language Models (LLMs) showcase impressive capabilities but encounter challenges like hallucination. Retrieval-Augmented Generation (RAG) has emerged as a promising solution by incorporating knowledge from external databases.
  arXiv  Detail & Related papers  (2023-12-18T07:47:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.