IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions

• URL: http://arxiv.org/abs/2305.14010v1
• Date: Tue, 23 May 2023 12:43:19 GMT
• Title: IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions
• Authors: Wenhao Yu, Meng Jiang, Peter Clark, Ashish Sabharwal
• Abstract summary: We introduce the first large-scale counterfactual open-domain question-answering (QA) benchmarks, named IfQA. The IfQA dataset contains over 3,800 questions that were annotated by crowdworkers on relevant Wikipedia passages. The unique challenges posed by the IfQA benchmark will push open-domain QA research on both retrieval and counterfactual reasoning fronts.
• Score: 54.23087908182134
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Although counterfactual reasoning is a fundamental aspect of intelligence, the lack of large-scale counterfactual open-domain question-answering (QA) benchmarks makes it difficult to evaluate and improve models on this ability. To address this void, we introduce the first such dataset, named IfQA, where each question is based on a counterfactual presupposition via an "if" clause. For example, if Los Angeles was on the east coast of the U.S., what would be the time difference between Los Angeles and Paris? Such questions require models to go beyond retrieving direct factual knowledge from the Web: they must identify the right information to retrieve and reason about an imagined situation that may even go against the facts built into their parameters. The IfQA dataset contains over 3,800 questions that were annotated annotated by crowdworkers on relevant Wikipedia passages. Empirical analysis reveals that the IfQA dataset is highly challenging for existing open-domain QA methods, including supervised retrieve-then-read pipeline methods (EM score 36.2), as well as recent few-shot approaches such as chain-of-thought prompting with GPT-3 (EM score 27.4). The unique challenges posed by the IfQA benchmark will push open-domain QA research on both retrieval and counterfactual reasoning fronts.

Related papers

• MythQA: Query-Based Large-Scale Check-Worthy Claim Detection through Multi-Answer Open-Domain Question Answering [8.70509665552136]
  Check-worthy claim detection aims at providing plausible misinformation to downstream fact-checking systems or human experts to check. Many efforts have been put into how to identify check-worthy claims from a small scale of pre-collected claims, but how to efficiently detect check-worthy claims directly from a large-scale information source, such as Twitter, remains underexplored. We introduce MythQA, a new multi-answer open-domain question answering(QA) task that involves contradictory stance mining for query-based large-scale check-worthy claim detection.
  arXiv  Detail & Related papers  (2023-07-21T18:35:24Z)
• RealTime QA: What's the Answer Right Now? [137.04039209995932]
  We introduce REALTIME QA, a dynamic question answering (QA) platform that announces questions and evaluates systems on a regular basis. We build strong baseline models upon large pretrained language models, including GPT-3 and T5. GPT-3 tends to return outdated answers when retrieved documents do not provide sufficient information to find an answer.
  arXiv  Detail & Related papers  (2022-07-27T07:26:01Z)
• Multifaceted Improvements for Conversational Open-Domain Question Answering [54.913313912927045]
  We propose a framework with Multifaceted Improvements for Conversational open-domain Question Answering (MICQA) Firstly, the proposed KL-divergence based regularization is able to lead to a better question understanding for retrieval and answer reading. Second, the added post-ranker module can push more relevant passages to the top placements and be selected for reader with a two-aspect constrains. Third, the well designed curriculum learning strategy effectively narrows the gap between the golden passage settings of training and inference, and encourages the reader to find true answer without the golden passage assistance.
  arXiv  Detail & Related papers  (2022-04-01T07:54:27Z)
• Relation-Guided Pre-Training for Open-Domain Question Answering [67.86958978322188]
  We propose a Relation-Guided Pre-Training (RGPT-QA) framework to solve complex open-domain questions. We show that RGPT-QA achieves 2.2%, 2.4%, and 6.3% absolute improvement in Exact Match accuracy on Natural Questions, TriviaQA, and WebQuestions.
  arXiv  Detail & Related papers  (2021-09-21T17:59:31Z)
• Improving Unsupervised Question Answering via Summarization-Informed Question Generation [47.96911338198302]
  Question Generation (QG) is the task of generating a plausible question for a passage, answer> pair. We make use of freely available news summary data, transforming declarative sentences into appropriate questions using dependency parsing, named entity recognition and semantic role labeling. The resulting questions are then combined with the original news articles to train an end-to-end neural QG model.
  arXiv  Detail & Related papers  (2021-09-16T13:08:43Z)
• SituatedQA: Incorporating Extra-Linguistic Contexts into QA [7.495151447459443]
  We introduce SituatedQA, an open-retrieval QA dataset where systems must produce the correct answer to a question given the temporal or geographical context. We find that a significant proportion of information seeking questions have context-dependent answers. Our study shows that existing models struggle with producing answers that are frequently updated or from uncommon locations.
  arXiv  Detail & Related papers  (2021-09-13T17:53:21Z)
• Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive Study [45.9120218818558]
  We benchmark the research on the NarrativeQA dataset with experiments with cutting-edge ODQA techniques. This quantifies the challenges Book QA poses, as well as advances the published state-of-the-art with a $sim$7% absolute improvement on Rouge-L. Our findings indicate that the event-centric questions dominate this task, which exemplifies the inability of existing QA models to handle event-oriented scenarios.
  arXiv  Detail & Related papers  (2021-06-07T17:46:09Z)
• AmbigQA: Answering Ambiguous Open-domain Questions [99.59747941602684]
  We introduce AmbigQA, a new open-domain question answering task which involves finding every plausible answer. To study this task, we construct AmbigNQ, a dataset covering 14,042 questions from NQ-open. We find that over half of the questions in NQ-open are ambiguous, with diverse sources of ambiguity such as event and entity references.
  arXiv  Detail & Related papers  (2020-04-22T15:42:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.