SituatedQA: Incorporating Extra-Linguistic Contexts into QA

• URL: http://arxiv.org/abs/2109.06157v1
• Date: Mon, 13 Sep 2021 17:53:21 GMT
• Title: SituatedQA: Incorporating Extra-Linguistic Contexts into QA
• Authors: Michael J.Q. Zhang and Eunsol Choi
• Abstract summary: We introduce SituatedQA, an open-retrieval QA dataset where systems must produce the correct answer to a question given the temporal or geographical context. We find that a significant proportion of information seeking questions have context-dependent answers. Our study shows that existing models struggle with producing answers that are frequently updated or from uncommon locations.
• Score: 7.495151447459443
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Answers to the same question may change depending on the extra-linguistic contexts (when and where the question was asked). To study this challenge, we introduce SituatedQA, an open-retrieval QA dataset where systems must produce the correct answer to a question given the temporal or geographical context. To construct SituatedQA, we first identify such questions in existing QA datasets. We find that a significant proportion of information seeking questions have context-dependent answers (e.g., roughly 16.5% of NQ-Open). For such context-dependent questions, we then crowdsource alternative contexts and their corresponding answers. Our study shows that existing models struggle with producing answers that are frequently updated or from uncommon locations. We further quantify how existing models, which are trained on data collected in the past, fail to generalize to answering questions asked in the present, even when provided with an updated evidence corpus (a roughly 15 point drop in accuracy). Our analysis suggests that open-retrieval QA benchmarks should incorporate extra-linguistic context to stay relevant globally and in the future. Our data, code, and datasheet are available at https://situatedqa.github.io/ .

Related papers

• Retrieving Contextual Information for Long-Form Question Answering using Weak Supervision [23.394961301584026]
  Long-form question answering (LFQA) aims at generating in-depth answers to end-user questions. We propose and compare different weak supervision techniques to optimize retrieval for contextual information. We show that long-form answers often anticipate likely follow-up questions.
  arXiv  Detail & Related papers  (2024-10-11T08:42:02Z)
• Long-form Question Answering: An Iterative Planning-Retrieval-Generation Approach [28.849548176802262]
  Long-form question answering (LFQA) poses a challenge as it involves generating detailed answers in the form of paragraphs. We propose an LFQA model with iterative Planning, Retrieval, and Generation. We find that our model outperforms the state-of-the-art models on various textual and factual metrics for the LFQA task.
  arXiv  Detail & Related papers  (2023-11-15T21:22:27Z)
• Diversity Enhanced Narrative Question Generation for Storybooks [4.043005183192124]
  We introduce a multi-question generation model (mQG) capable of generating multiple, diverse, and answerable questions. To validate the answerability of the generated questions, we employ a SQuAD2.0 fine-tuned question answering model. mQG shows promising results across various evaluation metrics, among strong baselines.
  arXiv  Detail & Related papers  (2023-10-25T08:10:04Z)
• UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Models [55.22048505787125]
  This paper contributes a comprehensive dataset, called UNK-VQA. We first augment the existing data via deliberate perturbations on either the image or question. We then extensively evaluate the zero- and few-shot performance of several emerging multi-modal large models.
  arXiv  Detail & Related papers  (2023-10-17T02:38:09Z)
• Question Answering Survey: Directions, Challenges, Datasets, Evaluation Matrices [0.0]
  The research directions of QA field are analyzed based on the type of question, answer type, source of evidence-answer, and modeling approach. This detailed followed by open challenges of the field like automatic question generation, similarity detection and, low resource availability for a language.
  arXiv  Detail & Related papers  (2021-12-07T08:53:40Z)
• ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers [93.55268936974971]
  We describe a Question Answering dataset that contains complex questions with conditional answers. We call this dataset ConditionalQA. We show that ConditionalQA is challenging for many of the existing QA models, especially in selecting answer conditions.
  arXiv  Detail & Related papers  (2021-10-13T17:16:46Z)
• Relation-Guided Pre-Training for Open-Domain Question Answering [67.86958978322188]
  We propose a Relation-Guided Pre-Training (RGPT-QA) framework to solve complex open-domain questions. We show that RGPT-QA achieves 2.2%, 2.4%, and 6.3% absolute improvement in Exact Match accuracy on Natural Questions, TriviaQA, and WebQuestions.
  arXiv  Detail & Related papers  (2021-09-21T17:59:31Z)
• Improving Unsupervised Question Answering via Summarization-Informed Question Generation [47.96911338198302]
  Question Generation (QG) is the task of generating a plausible question for a passage, answer> pair. We make use of freely available news summary data, transforming declarative sentences into appropriate questions using dependency parsing, named entity recognition and semantic role labeling. The resulting questions are then combined with the original news articles to train an end-to-end neural QG model.
  arXiv  Detail & Related papers  (2021-09-16T13:08:43Z)
• Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering [98.48363619128108]
  We propose an unsupervised approach to training QA models with generated pseudo-training data. We show that generating questions for QA training by applying a simple template on a related, retrieved sentence rather than the original context sentence improves downstream QA performance.
  arXiv  Detail & Related papers  (2020-04-24T17:57:45Z)
• AmbigQA: Answering Ambiguous Open-domain Questions [99.59747941602684]
  We introduce AmbigQA, a new open-domain question answering task which involves finding every plausible answer. To study this task, we construct AmbigNQ, a dataset covering 14,042 questions from NQ-open. We find that over half of the questions in NQ-open are ambiguous, with diverse sources of ambiguity such as event and entity references.
  arXiv  Detail & Related papers  (2020-04-22T15:42:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.