Better Retrieval May Not Lead to Better Question Answering

• URL: http://arxiv.org/abs/2205.03685v1
• Date: Sat, 7 May 2022 16:59:38 GMT
• Title: Better Retrieval May Not Lead to Better Question Answering
• Authors: Zhengzhong Liang, Tushar Khot, Steven Bethard, Mihai Surdeanu, Ashish Sabharwal
• Abstract summary: A popular approach to improve the system's performance is to improve the quality of the retrieved context from the IR stage. We show that for StrategyQA, a challenging open-domain QA dataset that requires multi-hop reasoning, this common approach is surprisingly ineffective.
• Score: 59.1892787017522
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Considerable progress has been made recently in open-domain question answering (QA) problems, which require Information Retrieval (IR) and Reading Comprehension (RC). A popular approach to improve the system's performance is to improve the quality of the retrieved context from the IR stage. In this work we show that for StrategyQA, a challenging open-domain QA dataset that requires multi-hop reasoning, this common approach is surprisingly ineffective -- improving the quality of the retrieved context hardly improves the system's performance. We further analyze the system's behavior to identify potential reasons.

Related papers

• Context Awareness Gate For Retrieval Augmented Generation [2.749898166276854]
  Retrieval Augmented Generation (RAG) has emerged as a widely adopted approach to mitigate the limitations of large language models (LLMs) in answering domain-specific questions. Previous research has predominantly focused on improving the accuracy and quality of retrieved data chunks to enhance the overall performance of the generation pipeline. We investigate the impact of retrieving irrelevant information in open-domain question answering, highlighting its significant detrimental effect on the quality of LLM outputs.
  arXiv  Detail & Related papers  (2024-11-25T06:48:38Z)
• Enhancing Retrieval in QA Systems with Derived Feature Association [0.0]
  Retrieval augmented generation (RAG) has become the standard in long context question answering (QA) systems. We propose a novel extension to RAG systems, which we call Retrieval from AI Derived Documents (RAIDD)
  arXiv  Detail & Related papers  (2024-10-02T05:24:49Z)
• Diversify-verify-adapt: Efficient and Robust Retrieval-Augmented Ambiguous Question Answering [45.154063285999015]
  The retrieval augmented generation (RAG) framework addresses an ambiguity in user queries in QA systems. RAG retrieves passages that cover all plausible interpretations and generates comprehensive responses. However, a single retrieval process often suffers from low quality results. We propose a diversify-verify-adapt (DIVA) framework to address this problem.
  arXiv  Detail & Related papers  (2024-09-04T01:14:04Z)
• KaPQA: Knowledge-Augmented Product Question-Answering [59.096607961704656]
  We introduce two product question-answering (QA) datasets focused on Adobe Acrobat and Photoshop products. We also propose a novel knowledge-driven RAG-QA framework to enhance the performance of the models in the product QA task.
  arXiv  Detail & Related papers  (2024-07-22T22:14:56Z)
• AdaCQR: Enhancing Query Reformulation for Conversational Search via Sparse and Dense Retrieval Alignment [16.62505706601199]
  We present a novel framework AdaCQR for conversational search reformulation. By aligning reformulation models with both term-based and semantic-based retrieval systems, AdaCQR enhances the generalizability of information-seeking queries. Experimental evaluations on the TopiOCQA and QReCC datasets demonstrate that AdaCQR significantly outperforms existing methods.
  arXiv  Detail & Related papers  (2024-07-02T05:50:16Z)
• REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answering [115.72130322143275]
  REAR is a RElevance-Aware Retrieval-augmented approach for open-domain question answering (QA) We develop a novel architecture for LLM-based RAG systems, by incorporating a specially designed assessment module. Experiments on four open-domain QA tasks show that REAR significantly outperforms previous a number of competitive RAG approaches.
  arXiv  Detail & Related papers  (2024-02-27T13:22:51Z)
• QontSum: On Contrasting Salient Content for Query-focused Summarization [22.738731393540633]
  Query-focused summarization (QFS) is a challenging task in natural language processing that generates summaries to address specific queries. This paper highlights the role of QFS in Grounded Answer Generation (GAR) We propose QontSum, a novel approach for QFS that leverages contrastive learning to help the model attend to the most relevant regions of the input document.
  arXiv  Detail & Related papers  (2023-07-14T19:25:35Z)
• Improving the Question Answering Quality using Answer Candidate Filtering based on Natural-Language Features [117.44028458220427]
  We address the problem of how the Question Answering (QA) quality of a given system can be improved. Our main contribution is an approach capable of identifying wrong answers provided by a QA system. In particular, our approach has shown its potential while removing in many cases the majority of incorrect answers.
  arXiv  Detail & Related papers  (2021-12-10T11:09:44Z)
• NoiseQA: Challenge Set Evaluation for User-Centric Question Answering [68.67783808426292]
  We show that components in the pipeline that precede an answering engine can introduce varied and considerable sources of error. We conclude that there is substantial room for progress before QA systems can be effectively deployed.
  arXiv  Detail & Related papers  (2021-02-16T18:35:29Z)
• Global Optimization of Objective Functions Represented by ReLU Networks [77.55969359556032]
  Neural networks can learn complex, non- adversarial functions, and it is challenging to guarantee their correct behavior in safety-critical contexts. Many approaches exist to find failures in networks (e.g., adversarial examples), but these cannot guarantee the absence of failures. We propose an approach that integrates the optimization process into the verification procedure, achieving better performance than the naive approach.
  arXiv  Detail & Related papers  (2020-10-07T08:19:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.