RetrievalQA: Assessing Adaptive Retrieval-Augmented Generation for Short-form Open-Domain Question Answering

• URL: http://arxiv.org/abs/2402.16457v2
• Date: Wed, 5 Jun 2024 05:23:21 GMT
• Title: RetrievalQA: Assessing Adaptive Retrieval-Augmented Generation for Short-form Open-Domain Question Answering
• Authors: Zihan Zhang, Meng Fang, Ling Chen,
• Abstract summary: Adaptive retrieval-augmented generation (ARAG) aims to dynamically determine the necessity of retrieval for queries instead of retrieving indiscriminately. This work presents a benchmark, RetrievalQA, comprising 1,271 short-form questions covering new world and long-tail knowledge.
• Score: 42.66223628527439
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adaptive retrieval-augmented generation (ARAG) aims to dynamically determine the necessity of retrieval for queries instead of retrieving indiscriminately to enhance the efficiency and relevance of the sourced information. However, previous works largely overlook the evaluation of ARAG approaches, leading to their effectiveness being understudied. This work presents a benchmark, RetrievalQA, comprising 1,271 short-form questions covering new world and long-tail knowledge. The knowledge necessary to answer the questions is absent from LLMs; therefore, external information must be retrieved to answer correctly. This makes RetrievalQA a suitable testbed to evaluate existing ARAG methods. We observe that calibration-based methods heavily rely on threshold tuning, while vanilla prompting is inadequate for guiding LLMs to make reliable retrieval decisions. Based on our findings, we propose Time-Aware Adaptive Retrieval (TA-ARE), a simple yet effective method that helps LLMs assess the necessity of retrieval without calibration or additional training. The dataset and code will be available at https://github.com/hyintell/RetrievalQA

Related papers

• Post-training an LLM for RAG? Train on Self-Generated Demonstrations [18.8726880078299]
  Large language models (LLMs) often struggle with knowledge intensive NLP tasks. Retrieval augmented generation (RAG) allows the model to leverage in-context information. We propose a recipe for training RAG-enabled LLMs using self-generated demonstrations.
  arXiv  Detail & Related papers  (2025-02-14T23:00:49Z)
• Invar-RAG: Invariant LLM-aligned Retrieval for Better Generation [43.630437906898635]
  We propose a novel two-stage fine-tuning architecture called Invar-RAG. In the retrieval stage, an LLM-based retriever is constructed by integrating LoRA-based representation learning. In the generation stage, a refined fine-tuning method is employed to improve LLM accuracy in generating answers based on retrieved information.
  arXiv  Detail & Related papers  (2024-11-11T14:25:37Z)
• Retriever-and-Memory: Towards Adaptive Note-Enhanced Retrieval-Augmented Generation [72.70046559930555]
  We propose a generic RAG approach called Adaptive Note-Enhanced RAG (Adaptive-Note) for complex QA tasks. Specifically, Adaptive-Note introduces an overarching view of knowledge growth, iteratively gathering new information in the form of notes. In addition, we employ an adaptive, note-based stop-exploration strategy to decide "what to retrieve and when to stop" to encourage sufficient knowledge exploration.
  arXiv  Detail & Related papers  (2024-10-11T14:03:29Z)
• When to Retrieve: Teaching LLMs to Utilize Information Retrieval Effectively [3.705145020383824]
  We show how Large Language Models (LLMs) can learn to use an off-the-shelf information retrieval (IR) system specifically when additional context is required to answer a given question.
  arXiv  Detail & Related papers  (2024-04-30T16:52:55Z)
• CLARINET: Augmenting Language Models to Ask Clarification Questions for Retrieval [52.134133938779776]
  We present CLARINET, a system that asks informative clarification questions by choosing questions whose answers would maximize certainty in the correct candidate. Our approach works by augmenting a large language model (LLM) to condition on a retrieval distribution, finetuning end-to-end to generate the question that would have maximized the rank of the true candidate at each turn.
  arXiv  Detail & Related papers  (2024-04-28T18:21:31Z)
• SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs [85.54906813106683]
  We propose a simple yet effective framework to enhance open-domain question answering (ODQA) with large language models (LLMs) SuRe helps LLMs predict more accurate answers for a given question, which are well-supported by the summarized retrieval (SuRe) Experimental results on diverse ODQA benchmarks demonstrate the superiority of SuRe, with improvements of up to 4.6% in exact match (EM) and 4.0% in F1 score over standard prompting approaches.
  arXiv  Detail & Related papers  (2024-04-17T01:15:54Z)
• Embedding-Informed Adaptive Retrieval-Augmented Generation of Large Language Models [37.02290559379761]
  Retrieval-augmented large language models (LLMs) have been remarkably competent in various NLP tasks. Motivated by this, Adaptive Retrieval-Augmented Generation (ARAG) studies retrieving only when the knowledge asked by the query is absent in the LLM.
  arXiv  Detail & Related papers  (2024-04-04T15:21:22Z)
• Self-Knowledge Guided Retrieval Augmentation for Large Language Models [59.771098292611846]
  Large language models (LLMs) have shown superior performance without task-specific fine-tuning. Retrieval-based methods can offer non-parametric world knowledge and improve the performance on tasks such as question answering. Self-Knowledge guided Retrieval augmentation (SKR) is a simple yet effective method which can let LLMs refer to the questions they have previously encountered.
  arXiv  Detail & Related papers  (2023-10-08T04:22:33Z)
• Query Rewriting for Retrieval-Augmented Large Language Models [139.242907155883]
  Large Language Models (LLMs) play powerful, black-box readers in the retrieve-then-read pipeline. This work introduces a new framework, Rewrite-Retrieve-Read instead of the previous retrieve-then-read for the retrieval-augmented LLMs.
  arXiv  Detail & Related papers  (2023-05-23T17:27:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.