ScopeQA: A Framework for Generating Out-of-Scope Questions for RAG

• URL: http://arxiv.org/abs/2410.14567v2
• Date: Thu, 19 Dec 2024 19:49:04 GMT
• Title: ScopeQA: A Framework for Generating Out-of-Scope Questions for RAG
• Authors: Zhiyuan Peng, Jinming Nian, Alexandre Evfimievski, Yi Fang,
• Abstract summary: Conversational AI agents use Retrieval Augmented Generation (RAG) to provide verifiable document-grounded responses to user inquiries. This paper presents a novel guided hallucination-based method to efficiently generate a diverse set of borderline out-of-scope confusing questions.
• Score: 52.33835101586687
• License:
• Abstract: Conversational AI agents use Retrieval Augmented Generation (RAG) to provide verifiable document-grounded responses to user inquiries. However, many natural questions do not have good answers: about 25\% contain false assumptions~\cite{Yu2023:CREPE}, and over 50\% are ambiguous~\cite{DBLP:conf/emnlp/MinMHZ20}. RAG agents need high-quality data to improve their responses to confusing questions. This paper presents a novel guided hallucination-based method to efficiently generate a diverse set of borderline out-of-scope confusing questions for a given document corpus. We conduct an empirical comparative evaluation of several large language models as RAG agents to measure the accuracy of confusion detection and appropriate response generation. We contribute a benchmark dataset to the public domain.

Related papers

• Evidence Contextualization and Counterfactual Attribution for Conversational QA over Heterogeneous Data with RAG Systems [4.143039012104666]
  Retrieval Augmented Generation (RAG) works as a backbone for interacting with an enterprise's own data via Conversational Question Answering (ConvQA) In this work, we demonstrate RAGONITE, a RAG system that remedies the above concerns by: (i) contextualizing evidence with source metadata and surrounding text; and (ii) computing counterfactual attribution.
  arXiv  Detail & Related papers  (2024-12-13T21:28:17Z)
• RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answering [61.19126689470398]
  Long-form RobustQA (LFRQA) is a new dataset covering 26K queries and large corpora across seven different domains. We show via experiments that RAG-QA Arena and human judgments on answer quality are highly correlated. Only 41.3% of the most competitive LLM's answers are preferred to LFRQA's answers, demonstrating RAG-QA Arena as a challenging evaluation platform for future research.
  arXiv  Detail & Related papers  (2024-07-19T03:02:51Z)
• Optimization of Retrieval-Augmented Generation Context with Outlier Detection [0.0]
  We focus on methods to reduce the size and improve the quality of the prompt context required for question-answering systems. Our goal is to select the most semantically relevant documents, treating the discarded ones as outliers. It was found that the greatest improvements were achieved with increasing complexity of the questions and answers.
  arXiv  Detail & Related papers  (2024-07-01T15:53:29Z)
• Long-Span Question-Answering: Automatic Question Generation and QA-System Ranking via Side-by-Side Evaluation [65.16137964758612]
  We explore the use of long-context capabilities in large language models to create synthetic reading comprehension data from entire books. Our objective is to test the capabilities of LLMs to analyze, understand, and reason over problems that require a detailed comprehension of long spans of text.
  arXiv  Detail & Related papers  (2024-05-31T20:15:10Z)
• CONFLARE: CONFormal LArge language model REtrieval [0.0]
  Retrieval-augmented generation (RAG) frameworks enable large language models (LLMs) to retrieve relevant information from a knowledge base and incorporate it into the context for generating responses. RAG does not guarantee valid responses if retrieval fails to identify the necessary information as the context for response generation. We introduce a four-step framework for applying conformal prediction to quantify retrieval uncertainty in RAG frameworks.
  arXiv  Detail & Related papers  (2024-04-04T02:58:21Z)
• Narrowing the Knowledge Evaluation Gap: Open-Domain Question Answering with Multi-Granularity Answers [21.814007454504978]
  We present a novel evaluation setting where a predicted answer is evaluated in terms of accuracy and informativeness against a set of multi-granularity answers. Our experiments show that large language models with standard decoding tend to generate specific answers, which are often incorrect. When evaluated on multi-granularity answers, DRAG yields a nearly 20 point increase in accuracy on average, which further increases for rare entities.
  arXiv  Detail & Related papers  (2024-01-09T17:44:36Z)
• RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question [29.18544401904503]
  We propose a new metric, RQUGE, based on the answerability of the candidate question given the context. We demonstrate that RQUGE has a higher correlation with human judgment without relying on the reference question.
  arXiv  Detail & Related papers  (2022-11-02T21:10:09Z)
• Generation-Augmented Retrieval for Open-domain Question Answering [134.27768711201202]
  Generation-Augmented Retrieval (GAR) for answering open-domain questions. We show that generating diverse contexts for a query is beneficial as fusing their results consistently yields better retrieval accuracy. GAR achieves state-of-the-art performance on Natural Questions and TriviaQA datasets under the extractive QA setup when equipped with an extractive reader.
  arXiv  Detail & Related papers  (2020-09-17T23:08:01Z)
• Answering Any-hop Open-domain Questions with Iterative Document Reranking [62.76025579681472]
  We propose a unified QA framework to answer any-hop open-domain questions. Our method consistently achieves performance comparable to or better than the state-of-the-art on both single-hop and multi-hop open-domain QA datasets.
  arXiv  Detail & Related papers  (2020-09-16T04:31:38Z)
• ClarQ: A large-scale and diverse dataset for Clarification Question Generation [67.1162903046619]
  We devise a novel bootstrapping framework that assists in the creation of a diverse, large-scale dataset of clarification questions based on postcomments extracted from stackexchange. We quantitatively demonstrate the utility of the newly created dataset by applying it to the downstream task of question-answering. We release this dataset in order to foster research into the field of clarification question generation with the larger goal of enhancing dialog and question answering systems.
  arXiv  Detail & Related papers  (2020-06-10T17:56:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.