Answerability in Retrieval-Augmented Open-Domain Question Answering

• URL: http://arxiv.org/abs/2403.01461v1
• Date: Sun, 3 Mar 2024 09:55:35 GMT
• Title: Answerability in Retrieval-Augmented Open-Domain Question Answering
• Authors: Rustam Abdumalikov, Pasquale Minervini and Yova Kementchedjhieva
• Abstract summary: Open-Domain Question Answering (ODQA) retrieval systems can exhibit sub-optimal behavior, providing text excerpts with varying degrees of irrelevance. Previous attempts to address this gap have relied on a simplistic approach of pairing questions with random text excerpts.
• Score: 17.177439885871788
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The performance of Open-Domain Question Answering (ODQA) retrieval systems can exhibit sub-optimal behavior, providing text excerpts with varying degrees of irrelevance. Unfortunately, many existing ODQA datasets lack examples specifically targeting the identification of irrelevant text excerpts. Previous attempts to address this gap have relied on a simplistic approach of pairing questions with random text excerpts. This paper aims to investigate the effectiveness of models trained using this randomized strategy, uncovering an important limitation in their ability to generalize to irrelevant text excerpts with high semantic overlap. As a result, we observed a substantial decrease in predictive accuracy, from 98% to 1%. To address this limitation, we discovered an efficient approach for training models to recognize such excerpts. By leveraging unanswerable pairs from the SQuAD 2.0 dataset, our models achieve a nearly perfect (~100%) accuracy when confronted with these challenging text excerpts.

Related papers

• pEBR: A Probabilistic Approach to Embedding Based Retrieval [4.8338111302871525]
  Embedding retrieval aims to learn a shared semantic representation space for both queries and items. In current industrial practice, retrieval systems typically retrieve a fixed number of items for different queries.
  arXiv  Detail & Related papers  (2024-10-25T07:14:12Z)
• Localizing Factual Inconsistencies in Attributable Text Generation [91.981439746404]
  We introduce QASemConsistency, a new formalism for localizing factual inconsistencies in attributable text generation. We first demonstrate the effectiveness of the QASemConsistency methodology for human annotation. We then implement several methods for automatically detecting localized factual inconsistencies.
  arXiv  Detail & Related papers  (2024-10-09T22:53:48Z)
• Long-Span Question-Answering: Automatic Question Generation and QA-System Ranking via Side-by-Side Evaluation [65.16137964758612]
  We explore the use of long-context capabilities in large language models to create synthetic reading comprehension data from entire books. Our objective is to test the capabilities of LLMs to analyze, understand, and reason over problems that require a detailed comprehension of long spans of text.
  arXiv  Detail & Related papers  (2024-05-31T20:15:10Z)
• SEMQA: Semi-Extractive Multi-Source Question Answering [94.04430035121136]
  We introduce a new QA task for answering multi-answer questions by summarizing multiple diverse sources in a semi-extractive fashion. We create the first dataset of this kind, QuoteSum, with human-written semi-extractive answers to natural and generated questions.
  arXiv  Detail & Related papers  (2023-11-08T18:46:32Z)
• Detrimental Contexts in Open-Domain Question Answering [9.059854023578508]
  We analyze how passages can have a detrimental effect on retrieve-then-read architectures used in question answering. Our findings demonstrate that model accuracy can be improved by 10% on two popular QA datasets by filtering out detrimental passages.
  arXiv  Detail & Related papers  (2023-10-27T11:45:16Z)
• Revisiting Sparse Retrieval for Few-shot Entity Linking [33.15662306409253]
  We propose an ELECTRA-based keyword extractor to denoise the mention context and construct a better query expression. For training the extractor, we propose a distant supervision method to automatically generate training data based on overlapping tokens between mention contexts and entity descriptions. Experimental results on the ZESHEL dataset demonstrate that the proposed method outperforms state-of-the-art models by a significant margin across all test domains.
  arXiv  Detail & Related papers  (2023-10-19T03:51:10Z)
• Short Answer Grading Using One-shot Prompting and Text Similarity Scoring Model [2.14986347364539]
  We developed an automated short answer grading model that provided both analytic scores and holistic scores. The accuracy and quadratic weighted kappa of our model were 0.67 and 0.71 on a subset of the publicly available ASAG dataset.
  arXiv  Detail & Related papers  (2023-05-29T22:05:29Z)
• Revisiting text decomposition methods for NLI-based factuality scoring of summaries [9.044665059626958]
  We show that fine-grained decomposition is not always a winning strategy for factuality scoring. We also show that small changes to previously proposed entailment-based scoring methods can result in better performance.
  arXiv  Detail & Related papers  (2022-11-30T09:54:37Z)
• Mixed-modality Representation Learning and Pre-training for Joint Table-and-Text Retrieval in OpenQA [85.17249272519626]
  An optimized OpenQA Table-Text Retriever (OTTeR) is proposed. We conduct retrieval-centric mixed-modality synthetic pre-training. OTTeR substantially improves the performance of table-and-text retrieval on the OTT-QA dataset.
  arXiv  Detail & Related papers  (2022-10-11T07:04:39Z)
• Toward the Understanding of Deep Text Matching Models for Information Retrieval [72.72380690535766]
  This paper aims at testing whether existing deep text matching methods satisfy some fundamental gradients in information retrieval. Specifically, four attributions are used in our study, i.e., term frequency constraint, term discrimination constraint, length normalization constraints, and TF-length constraint. Experimental results on LETOR 4.0 and MS Marco show that all the investigated deep text matching methods satisfy the above constraints with high probabilities in statistics.
  arXiv  Detail & Related papers  (2021-08-16T13:33:15Z)
• Tradeoffs in Sentence Selection Techniques for Open-Domain Question Answering [54.541952928070344]
  We describe two groups of models for sentence selection: QA-based approaches, which run a full-fledged QA system to identify answer candidates, and retrieval-based models, which find parts of each passage specifically related to each question. We show that very lightweight QA models can do well at this task, but retrieval-based models are faster still.
  arXiv  Detail & Related papers  (2020-09-18T23:39:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.