Diversify-verify-adapt: Efficient and Robust Retrieval-Augmented Ambiguous Question Answering

• URL: http://arxiv.org/abs/2409.02361v1
• Date: Wed, 4 Sep 2024 01:14:04 GMT
• Title: Diversify-verify-adapt: Efficient and Robust Retrieval-Augmented Ambiguous Question Answering
• Authors: Yeonjun In, Sungchul Kim, Ryan A. Rossi, Md Mehrab Tanjim, Tong Yu, Ritwik Sinha, Chanyoung Park,
• Abstract summary: The retrieval augmented generation (RAG) framework addresses an ambiguity in user queries in QA systems. RAG retrieves passages that cover all plausible interpretations and generates comprehensive responses. However, a single retrieval process often suffers from low quality results. We propose a diversify-verify-adapt (DIVA) framework to address this problem.
• Score: 45.154063285999015
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The retrieval augmented generation (RAG) framework addresses an ambiguity in user queries in QA systems by retrieving passages that cover all plausible interpretations and generating comprehensive responses based on the passages. However, our preliminary studies reveal that a single retrieval process often suffers from low quality results, as the retrieved passages frequently fail to capture all plausible interpretations. Although the iterative RAG approach has been proposed to address this problem, it comes at the cost of significantly reduced efficiency. To address these issues, we propose the diversify-verify-adapt (DIVA) framework. DIVA first diversifies the retrieved passages to encompass diverse interpretations. Subsequently, DIVA verifies the quality of the passages and adapts the most suitable approach tailored to their quality. This approach improves the QA systems accuracy and robustness by handling low quality retrieval issue in ambiguous questions, while enhancing efficiency.

Related papers

• Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coverage [74.70255719194819]
  We introduce a novel framework based on sub-question coverage, which measures how well a RAG system addresses different facets of a question. We use this framework to evaluate three commercial generative answer engines: You.com, Perplexity AI, and Bing Chat. We find that while all answer engines cover core sub-questions more often than background or follow-up ones, they still miss around 50% of core sub-questions.
  arXiv  Detail & Related papers  (2024-10-20T22:59:34Z)
• RAG-ConfusionQA: A Benchmark for Evaluating LLMs on Confusing Questions [52.33835101586687]
  Conversational AI agents use Retrieval Augmented Generation (RAG) to provide verifiable document-grounded responses to user inquiries. This paper presents a novel synthetic data generation method to efficiently create a diverse set of context-grounded confusing questions from a given document corpus.
  arXiv  Detail & Related papers  (2024-10-18T16:11:29Z)
• CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity [23.48167670445722]
  Retrieval-Augmented Generation (RAG) aims to generate more accurate and reliable answers with the help of the retrieved context from external knowledge sources. evaluating these systems remains a crucial research area due to the following issues. We propose a Comprehensive Full-chain Evaluation (CoFE-RAG) framework to facilitate thorough evaluation across the entire RAG pipeline.
  arXiv  Detail & Related papers  (2024-10-16T05:20:32Z)
• AdaCQR: Enhancing Query Reformulation for Conversational Search via Sparse and Dense Retrieval Alignment [16.62505706601199]
  We present a novel framework AdaCQR for conversational search reformulation. By aligning reformulation models with both term-based and semantic-based retrieval systems, AdaCQR enhances the generalizability of information-seeking queries. Experimental evaluations on the TopiOCQA and QReCC datasets demonstrate that AdaCQR significantly outperforms existing methods.
  arXiv  Detail & Related papers  (2024-07-02T05:50:16Z)
• On the Impact of Speech Recognition Errors in Passage Retrieval for Spoken Question Answering [13.013751306590303]
  We study the robustness of lexical and dense retrievers against questions with synthetic ASR noise. We create a new dataset with questions voiced by human users and use their transcriptions to show that the retrieval performance can further degrade when dealing with natural ASR noise instead of synthetic ASR noise.
  arXiv  Detail & Related papers  (2022-09-26T18:29:36Z)
• Better Retrieval May Not Lead to Better Question Answering [59.1892787017522]
  A popular approach to improve the system's performance is to improve the quality of the retrieved context from the IR stage. We show that for StrategyQA, a challenging open-domain QA dataset that requires multi-hop reasoning, this common approach is surprisingly ineffective.
  arXiv  Detail & Related papers  (2022-05-07T16:59:38Z)
• Joint Passage Ranking for Diverse Multi-Answer Retrieval [56.43443577137929]
  We study multi-answer retrieval, an under-explored problem that requires retrieving passages to cover multiple distinct answers for a question. This task requires joint modeling of retrieved passages, as models should not repeatedly retrieve passages containing the same answer at the cost of missing a different valid answer. In this paper, we introduce JPR, a joint passage retrieval model focusing on reranking. To model the joint probability of the retrieved passages, JPR makes use of an autoregressive reranker that selects a sequence of passages, equipped with novel training and decoding algorithms.
  arXiv  Detail & Related papers  (2021-04-17T04:48:36Z)
• Complementary Evidence Identification in Open-Domain Question Answering [66.17954897343456]
  We propose a new problem of complementary evidence identification for open-domain question answering (QA) The problem aims to efficiently find a small set of passages that covers full evidence from multiple aspects as to answer a complex question.
  arXiv  Detail & Related papers  (2021-03-22T08:04:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.