Seven Failure Points When Engineering a Retrieval Augmented Generation
System
- URL: http://arxiv.org/abs/2401.05856v1
- Date: Thu, 11 Jan 2024 12:04:11 GMT
- Title: Seven Failure Points When Engineering a Retrieval Augmented Generation
System
- Authors: Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brannelly,
Mohamed Abdelrazek
- Abstract summary: RAG systems aim to reduce the problem of hallucinated responses from large language models.
RAG systems suffer from limitations inherent to information retrieval systems.
We present an experience report on the failure points of RAG systems from three case studies.
- Score: 1.8776685617612472
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Software engineers are increasingly adding semantic search capabilities to
applications using a strategy known as Retrieval Augmented Generation (RAG). A
RAG system involves finding documents that semantically match a query and then
passing the documents to a large language model (LLM) such as ChatGPT to
extract the right answer using an LLM. RAG systems aim to: a) reduce the
problem of hallucinated responses from LLMs, b) link sources/references to
generated responses, and c) remove the need for annotating documents with
meta-data. However, RAG systems suffer from limitations inherent to information
retrieval systems and from reliance on LLMs. In this paper, we present an
experience report on the failure points of RAG systems from three case studies
from separate domains: research, education, and biomedical. We share the
lessons learned and present 7 failure points to consider when designing a RAG
system. The two key takeaways arising from our work are: 1) validation of a RAG
system is only feasible during operation, and 2) the robustness of a RAG system
evolves rather than designed in at the start. We conclude with a list of
potential research directions on RAG systems for the software engineering
community.
Related papers
- Retrieval-Augmented Generation for Natural Language Processing: A Survey [25.11304732038443]
retrieval-augmented generation (RAG) leverages an external knowledge database to augment large language models.
This paper reviews all significant techniques of RAG, especially in the retriever and the retrieval fusions.
RAG is used in representative natural language processing tasks and industrial scenarios.
arXiv Detail & Related papers (2024-07-18T06:06:53Z) - Enhancing Retrieval and Managing Retrieval: A Four-Module Synergy for Improved Quality and Efficiency in RAG Systems [14.62114319247837]
Retrieval-augmented generation (RAG) techniques leverage the in-context learning capabilities of large language models (LLMs) to produce more accurate and relevant responses.
A critical component, the Query Rewriter module, enhances knowledge retrieval by generating a search-friendly query.
These four RAG modules synergistically improve the response quality and efficiency of the RAG system.
arXiv Detail & Related papers (2024-07-15T12:35:00Z) - Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting [68.90949377014742]
Speculative RAG is a framework that leverages a larger generalist LM to efficiently verify multiple RAG drafts produced in parallel by a smaller, distilled specialist LM.
Our method accelerates RAG by delegating drafting to the smaller specialist LM, with the larger generalist LM performing a single verification pass over the drafts.
It notably enhances accuracy by up to 12.97% while reducing latency by 51% compared to conventional RAG systems on PubHealth.
arXiv Detail & Related papers (2024-07-11T06:50:19Z) - R^2AG: Incorporating Retrieval Information into Retrieval Augmented Generation [11.890598082534577]
Retrieval augmented generation (RAG) has been applied in many scenarios to augment large language models (LLMs) with external documents provided by retrievers.
This paper proposes R$2$AG, a novel enhanced RAG framework that incorporates Retrieval information into Retrieval Augmented Generation.
arXiv Detail & Related papers (2024-06-19T06:19:48Z) - RaFe: Ranking Feedback Improves Query Rewriting for RAG [83.24385658573198]
We propose a framework for training query rewriting models free of annotations.
By leveraging a publicly available reranker, oursprovides feedback aligned well with the rewriting objectives.
arXiv Detail & Related papers (2024-05-23T11:00:19Z) - Towards a Search Engine for Machines: Unified Ranking for Multiple Retrieval-Augmented Large Language Models [21.115495457454365]
uRAG is a framework with a unified retrieval engine that serves multiple downstream retrieval-augmented generation (RAG) systems.
We build a large-scale experimentation ecosystem consisting of 18 RAG systems that engage in training and 18 unknown RAG systems that use the uRAG as the new users of the search engine.
arXiv Detail & Related papers (2024-04-30T19:51:37Z) - RAGGED: Towards Informed Design of Retrieval Augmented Generation Systems [51.171355532527365]
We introduce the RAGGED framework to analyze and optimize RAG systems.
We study two classic sparse and dense retrievers, and four top-performing LMs in encoder-decoder and decoder-only architectures.
arXiv Detail & Related papers (2024-03-14T02:26:31Z) - REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain
Question Answering [122.62012375722124]
In existing methods, large language models (LLMs) cannot precisely assess the relevance of retrieved documents.
We propose REAR, a RElevance-Aware Retrieval-augmented approach for open-domain question answering (QA)
arXiv Detail & Related papers (2024-02-27T13:22:51Z) - CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models [49.16989035566899]
Retrieval-Augmented Generation (RAG) is a technique that enhances the capabilities of large language models (LLMs) by incorporating external knowledge sources.
This paper constructs a large-scale and more comprehensive benchmark, and evaluates all the components of RAG systems in various RAG application scenarios.
arXiv Detail & Related papers (2024-01-30T14:25:32Z) - The Power of Noise: Redefining Retrieval for RAG Systems [19.387105120040157]
Retrieval-Augmented Generation (RAG) has emerged as a method to extend beyond the pre-trained knowledge of Large Language Models.
We focus on the type of passages IR systems within a RAG solution should retrieve.
arXiv Detail & Related papers (2024-01-26T14:14:59Z) - Self-RAG: Learning to Retrieve, Generate, and Critique through
Self-Reflection [74.51523859064802]
We introduce a new framework called Self-Reflective Retrieval-Augmented Generation (Self-RAG)
Self-RAG enhances an LM's quality and factuality through retrieval and self-reflection.
It significantly outperforms state-of-the-art LLMs and retrieval-augmented models on a diverse set of tasks.
arXiv Detail & Related papers (2023-10-17T18:18:32Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.