RE-RAG: Improving Open-Domain QA Performance and Interpretability with Relevance Estimator in Retrieval-Augmented Generation

• URL: http://arxiv.org/abs/2406.05794v3
• Date: Thu, 24 Oct 2024 14:57:52 GMT
• Title: RE-RAG: Improving Open-Domain QA Performance and Interpretability with Relevance Estimator in Retrieval-Augmented Generation
• Authors: Kiseung Kim, Jay-Yoon Lee,
• Abstract summary: We propose a relevance estimator (RE) that provides relative relevance between contexts as previous rerankers did. We show that RE trained with a small generator (sLM) can not only improve the sLM fine-tuned together with RE but also improve previously unreferenced large language models.
• Score: 5.10832476049103
• License:
• Abstract: The Retrieval Augmented Generation (RAG) framework utilizes a combination of parametric knowledge and external knowledge to demonstrate state-of-the-art performance on open-domain question answering tasks. However, the RAG framework suffers from performance degradation when the query is accompanied by irrelevant contexts. In this work, we propose the RE-RAG framework, which introduces a relevance estimator (RE) that not only provides relative relevance between contexts as previous rerankers did, but also provides confidence, which can be used to classify whether given context is useful for answering the given question. We propose a weakly supervised method for training the RE simply utilizing question-answer data without any labels for correct contexts. We show that RE trained with a small generator (sLM) can not only improve the sLM fine-tuned together with RE but also improve previously unreferenced large language models (LLMs). Furthermore, we investigate new decoding strategies that utilize the proposed confidence measured by RE such as choosing to let the user know that it is "unanswerable" to answer the question given the retrieved contexts or choosing to rely on LLM's parametric knowledge rather than unrelated contexts.

Related papers

• SMART-RAG: Selection using Determinantal Matrices for Augmented Retrieval [40.17823569905232]
  Retrieval-Augmented Generation (RAG) has greatly improved large language models (LLMs) by enabling them to generate accurate, contextually grounded responses. RAG approaches, which prioritize top-ranked documents based solely on query-context relevance, often introduce redundancy and conflicting information. We propose Selection using Matrices for Augmented Retrieval (RAG) in question answering tasks, a fully unsupervised and training-free framework designed to optimize context selection in RAG.
  arXiv  Detail & Related papers  (2024-09-21T03:03:09Z)
• VERA: Validation and Enhancement for Retrieval Augmented systems [0.0]
  We propose textbfVERA (textbfValidation and textbfEnhancement for textbfRetrieval textbfAugmented systems), a system designed to evaluate and enhance the retrieved context before response generation. VERA employs an evaluator-cum-enhancer LLM that first checks if external retrieval is necessary, evaluates the relevance and redundancy of the retrieved context, and refines it to eliminate non-essential information.
  arXiv  Detail & Related papers  (2024-09-18T16:10:47Z)
• SFR-RAG: Towards Contextually Faithful LLMs [57.666165819196486]
  Retrieval Augmented Generation (RAG) is a paradigm that integrates external contextual information with large language models (LLMs) to enhance factual accuracy and relevance. We introduce SFR-RAG, a small LLM that is instruction-textual with an emphasis on context-grounded generation and hallucination. We also present ConBench, a new evaluation framework compiling multiple popular and diverse RAG benchmarks.
  arXiv  Detail & Related papers  (2024-09-16T01:08:18Z)
• RaFe: Ranking Feedback Improves Query Rewriting for RAG [83.24385658573198]
  We propose a framework for training query rewriting models free of annotations. By leveraging a publicly available reranker, oursprovides feedback aligned well with the rewriting objectives.
  arXiv  Detail & Related papers  (2024-05-23T11:00:19Z)
• CONFLARE: CONFormal LArge language model REtrieval [0.0]
  Retrieval-augmented generation (RAG) frameworks enable large language models (LLMs) to retrieve relevant information from a knowledge base and incorporate it into the context for generating responses. RAG does not guarantee valid responses if retrieval fails to identify the necessary information as the context for response generation. We introduce a four-step framework for applying conformal prediction to quantify retrieval uncertainty in RAG frameworks.
  arXiv  Detail & Related papers  (2024-04-04T02:58:21Z)
• RAGGED: Towards Informed Design of Retrieval Augmented Generation Systems [51.171355532527365]
  Retrieval-augmented generation (RAG) can significantly improve the performance of language models (LMs) RAGGED is a framework for analyzing RAG configurations across various document-based question answering tasks.
  arXiv  Detail & Related papers  (2024-03-14T02:26:31Z)
• REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answering [122.62012375722124]
  In existing methods, large language models (LLMs) cannot precisely assess the relevance of retrieved documents. We propose REAR, a RElevance-Aware Retrieval-augmented approach for open-domain question answering (QA)
  arXiv  Detail & Related papers  (2024-02-27T13:22:51Z)
• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection [74.51523859064802]
  We introduce a new framework called Self-Reflective Retrieval-Augmented Generation (Self-RAG) Self-RAG enhances an LM's quality and factuality through retrieval and self-reflection. It significantly outperforms state-of-the-art LLMs and retrieval-augmented models on a diverse set of tasks.
  arXiv  Detail & Related papers  (2023-10-17T18:18:32Z)
• Generation-Augmented Retrieval for Open-domain Question Answering [134.27768711201202]
  Generation-Augmented Retrieval (GAR) for answering open-domain questions. We show that generating diverse contexts for a query is beneficial as fusing their results consistently yields better retrieval accuracy. GAR achieves state-of-the-art performance on Natural Questions and TriviaQA datasets under the extractive QA setup when equipped with an extractive reader.
  arXiv  Detail & Related papers  (2020-09-17T23:08:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.