Diversify-verify-adapt: Efficient and Robust Retrieval-Augmented Ambiguous Question Answering

• URL: http://arxiv.org/abs/2409.02361v1
• Date: Wed, 4 Sep 2024 01:14:04 GMT
• Title: Diversify-verify-adapt: Efficient and Robust Retrieval-Augmented Ambiguous Question Answering
• Authors: Yeonjun In, Sungchul Kim, Ryan A. Rossi, Md Mehrab Tanjim, Tong Yu, Ritwik Sinha, Chanyoung Park,
• Abstract summary: The retrieval augmented generation (RAG) framework addresses an ambiguity in user queries in QA systems. RAG retrieves passages that cover all plausible interpretations and generates comprehensive responses. However, a single retrieval process often suffers from low quality results. We propose a diversify-verify-adapt (DIVA) framework to address this problem.
• Score: 45.154063285999015
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The retrieval augmented generation (RAG) framework addresses an ambiguity in user queries in QA systems by retrieving passages that cover all plausible interpretations and generating comprehensive responses based on the passages. However, our preliminary studies reveal that a single retrieval process often suffers from low quality results, as the retrieved passages frequently fail to capture all plausible interpretations. Although the iterative RAG approach has been proposed to address this problem, it comes at the cost of significantly reduced efficiency. To address these issues, we propose the diversify-verify-adapt (DIVA) framework. DIVA first diversifies the retrieved passages to encompass diverse interpretations. Subsequently, DIVA verifies the quality of the passages and adapts the most suitable approach tailored to their quality. This approach improves the QA systems accuracy and robustness by handling low quality retrieval issue in ambiguous questions, while enhancing efficiency.

Related papers

• Agentic Verification for Ambiguous Query Disambiguation [42.238086712267396]
  We tackle the challenge of disambiguating queries in retrieval-augmented generation (RAG) to diverse yet answerable interpretations. We propose a joint approach to unify diversification with verification by incorporating feedback from retriever and generator early on. We validate the efficiency and effectiveness of our method on the widely adopted ASQA benchmark to achieve diverse yet verifiable interpretations.
  arXiv  Detail & Related papers  (2025-02-14T18:31:39Z)
• SUGAR: Leveraging Contextual Confidence for Smarter Retrieval [28.552283701883766]
  We introduce Semantic Uncertainty Guided Adaptive Retrieval (SUGAR) We leverage context-based entropy to actively decide whether to retrieve and to further determine between single-step and multi-step retrieval. Our empirical results show that selective retrieval guided by semantic uncertainty estimation improves the performance across diverse question answering tasks, as well as achieves a more efficient inference.
  arXiv  Detail & Related papers  (2025-01-09T01:24:59Z)
• Unanswerability Evaluation for Retrieval Augmented Generation [74.3022365715597]
  UAEval4RAG is a framework designed to evaluate whether RAG systems can handle unanswerable queries effectively. We define a taxonomy with six unanswerable categories, and UAEval4RAG automatically synthesizes diverse and challenging queries.
  arXiv  Detail & Related papers  (2024-12-16T19:11:55Z)
• Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coverage [74.70255719194819]
  We introduce a novel framework based on sub-question coverage, which measures how well a RAG system addresses different facets of a question. We use this framework to evaluate three commercial generative answer engines: You.com, Perplexity AI, and Bing Chat. We find that while all answer engines cover core sub-questions more often than background or follow-up ones, they still miss around 50% of core sub-questions.
  arXiv  Detail & Related papers  (2024-10-20T22:59:34Z)
• ScopeQA: A Framework for Generating Out-of-Scope Questions for RAG [52.33835101586687]
  Conversational AI agents use Retrieval Augmented Generation (RAG) to provide verifiable document-grounded responses to user inquiries. This paper presents a novel guided hallucination-based method to efficiently generate a diverse set of borderline out-of-scope confusing questions.
  arXiv  Detail & Related papers  (2024-10-18T16:11:29Z)
• CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity [23.48167670445722]
  Retrieval-Augmented Generation (RAG) aims to generate more accurate and reliable answers with the help of the retrieved context from external knowledge sources. evaluating these systems remains a crucial research area due to the following issues. We propose a Comprehensive Full-chain Evaluation (CoFE-RAG) framework to facilitate thorough evaluation across the entire RAG pipeline.
  arXiv  Detail & Related papers  (2024-10-16T05:20:32Z)
• AdaCQR: Enhancing Query Reformulation for Conversational Search via Sparse and Dense Retrieval Alignment [16.62505706601199]
  We present a novel framework AdaCQR for conversational search reformulation. By aligning reformulation models with both term-based and semantic-based retrieval systems, AdaCQR enhances the generalizability of information-seeking queries. Experimental results on the TopiOCQA and QReCC datasets demonstrate that AdaCQR outperforms the existing methods in a more efficient framework.
  arXiv  Detail & Related papers  (2024-07-02T05:50:16Z)
• Better Retrieval May Not Lead to Better Question Answering [59.1892787017522]
  A popular approach to improve the system's performance is to improve the quality of the retrieved context from the IR stage. We show that for StrategyQA, a challenging open-domain QA dataset that requires multi-hop reasoning, this common approach is surprisingly ineffective.
  arXiv  Detail & Related papers  (2022-05-07T16:59:38Z)
• Complementary Evidence Identification in Open-Domain Question Answering [66.17954897343456]
  We propose a new problem of complementary evidence identification for open-domain question answering (QA) The problem aims to efficiently find a small set of passages that covers full evidence from multiple aspects as to answer a complex question.
  arXiv  Detail & Related papers  (2021-03-22T08:04:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.