MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop
Queries
- URL: http://arxiv.org/abs/2401.15391v1
- Date: Sat, 27 Jan 2024 11:41:48 GMT
- Title: MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop
Queries
- Authors: Yixuan Tang and Yi Yang
- Abstract summary: Retrieval-augmented generation (RAG) augments large language models (LLM) by retrieving relevant knowledge.
Existing RAG systems are inadequate in answering multi-hop queries, which require retrieving and reasoning over multiple pieces of supporting evidence.
We develop a novel dataset, MultiHop-RAG, which consists of a knowledge base, a large collection of multi-hop queries, their ground-truth answers, and the associated supporting evidence.
- Score: 22.4349439498591
- License: http://creativecommons.org/licenses/by-sa/4.0/
- Abstract: Retrieval-augmented generation (RAG) augments large language models (LLM) by
retrieving relevant knowledge, showing promising potential in mitigating LLM
hallucinations and enhancing response quality, thereby facilitating the great
adoption of LLMs in practice. However, we find that existing RAG systems are
inadequate in answering multi-hop queries, which require retrieving and
reasoning over multiple pieces of supporting evidence. Furthermore, to our
knowledge, no existing RAG benchmarking dataset focuses on multi-hop queries.
In this paper, we develop a novel dataset, MultiHop-RAG, which consists of a
knowledge base, a large collection of multi-hop queries, their ground-truth
answers, and the associated supporting evidence. We detail the procedure of
building the dataset, utilizing an English news article dataset as the
underlying RAG knowledge base. We demonstrate the benchmarking utility of
MultiHop-RAG in two experiments. The first experiment compares different
embedding models for retrieving evidence for multi-hop queries. In the second
experiment, we examine the capabilities of various state-of-the-art LLMs,
including GPT-4, PaLM, and Llama2-70B, in reasoning and answering multi-hop
queries given the evidence. Both experiments reveal that existing RAG methods
perform unsatisfactorily in retrieving and answering multi-hop queries. We hope
MultiHop-RAG will be a valuable resource for the community in developing
effective RAG systems, thereby facilitating greater adoption of LLMs in
practice. The MultiHop-RAG and implemented RAG system is publicly available at
https://github.com/yixuantt/MultiHop-RAG/.
Related papers
- Retrieval-Augmented Generation for Natural Language Processing: A Survey [25.11304732038443]
retrieval-augmented generation (RAG) leverages an external knowledge database to augment large language models.
This paper reviews all significant techniques of RAG, especially in the retriever and the retrieval fusions.
RAG is used in representative natural language processing tasks and industrial scenarios.
arXiv Detail & Related papers (2024-07-18T06:06:53Z) - BERGEN: A Benchmarking Library for Retrieval-Augmented Generation [26.158785168036662]
Retrieval-Augmented Generation allows to enhance Large Language Models with external knowledge.
Inconsistent benchmarking poses a major challenge in comparing approaches and understanding the impact of each component in the pipeline.
In this work, we study best practices that lay the groundwork for a systematic evaluation of RAG and present BERGEN, an end-to-end library for reproducible research standardizing RAG experiments.
arXiv Detail & Related papers (2024-07-01T09:09:27Z) - Multi-Meta-RAG: Improving RAG for Multi-Hop Queries using Database Filtering with LLM-Extracted Metadata [1.6574413179773757]
Retrieval-augmented generation (RAG) enables retrieval of relevant information from an external knowledge source.
Traditional RAG applications perform poorly in answering multi-hop questions.
We introduce a new method called Multi-Meta-RAG, which uses database filtering with LLM-extracted metadata.
arXiv Detail & Related papers (2024-06-19T04:53:48Z) - Multi-Head RAG: Solving Multi-Aspect Problems with LLMs [13.638439488923671]
Retrieval Augmented Generation (RAG) enhances the abilities of Large Language Models (LLMs)
Existing RAG solutions do not focus on queries that may require fetching multiple documents with substantially different contents.
This paper introduces Multi-Head RAG (MRAG), a novel scheme designed to address this gap with a simple yet powerful idea.
arXiv Detail & Related papers (2024-06-07T16:59:38Z) - ERBench: An Entity-Relationship based Automatically Verifiable
Hallucination Benchmark for Large Language Models [48.38966595131693]
Large language models (LLMs) have achieved unprecedented performance in various applications, yet their evaluation remains a critical issue.
We contend that utilizing existing relational databases is a promising approach for constructing benchmarks due to their accurate knowledge description.
We propose ERBench to automatically convert any relational database into a benchmark based on the entity-relationship (ER) model.
arXiv Detail & Related papers (2024-03-08T12:42:36Z) - CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models [49.16989035566899]
Retrieval-Augmented Generation (RAG) is a technique that enhances the capabilities of large language models (LLMs) by incorporating external knowledge sources.
This paper constructs a large-scale and more comprehensive benchmark, and evaluates all the components of RAG systems in various RAG application scenarios.
arXiv Detail & Related papers (2024-01-30T14:25:32Z) - Generative Multi-Modal Knowledge Retrieval with Large Language Models [75.70313858231833]
We propose an innovative end-to-end generative framework for multi-modal knowledge retrieval.
Our framework takes advantage of the fact that large language models (LLMs) can effectively serve as virtual knowledge bases.
We demonstrate significant improvements ranging from 3.0% to 14.6% across all evaluation metrics when compared to strong baselines.
arXiv Detail & Related papers (2024-01-16T08:44:29Z) - Parrot: Enhancing Multi-Turn Instruction Following for Large Language Models [79.32652077838046]
We introduce Parrot, a solution aiming to enhance multi-turn instruction following for large language models (LLMs)
First, we introduce an efficient but effective method for collecting multi-turn instructions that feature human-like queries, such as anaphora and ellipsis.
Second, we propose a context-aware preference optimization strategy to further enhance LLMs for complex queries in multi-turn interaction.
arXiv Detail & Related papers (2023-10-11T08:36:43Z) - Enhancing Multi-modal and Multi-hop Question Answering via Structured
Knowledge and Unified Retrieval-Generation [33.56304858796142]
Multi-modal multi-hop question answering involves answering a question by reasoning over multiple input sources from different modalities.
Existing methods often retrieve evidences separately and then use a language model to generate an answer based on the retrieved evidences.
We propose a Structured Knowledge and Unified Retrieval-Generation (RG) approach to address these issues.
arXiv Detail & Related papers (2022-12-16T18:12:04Z) - UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question
Answering Over Knowledge Graph [89.98762327725112]
Multi-hop Question Answering over Knowledge Graph(KGQA) aims to find the answer entities that are multiple hops away from the topic entities mentioned in a natural language question.
We propose UniKGQA, a novel approach for multi-hop KGQA task, by unifying retrieval and reasoning in both model architecture and parameter learning.
arXiv Detail & Related papers (2022-12-02T04:08:09Z) - Multi-hop Question Generation with Graph Convolutional Network [58.31752179830959]
Multi-hop Question Generation (QG) aims to generate answer-related questions by aggregating and reasoning over multiple scattered evidence from different paragraphs.
We propose Multi-Hop volution Fusion Network for Question Generation (MulQG), which does context encoding in multiple hops.
Our proposed model is able to generate fluent questions with high completeness and outperforms the strongest baseline by 20.8% in the multi-hop evaluation.
arXiv Detail & Related papers (2020-10-19T06:15:36Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.