Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive Study

• URL: http://arxiv.org/abs/2106.03826v1
• Date: Mon, 7 Jun 2021 17:46:09 GMT
• Title: Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive Study
• Authors: Xiangyang Mou, Chenghao Yang, Mo Yu, Bingsheng Yao, Xiaoxiao Guo, Saloni Potdar, Hui Su
• Abstract summary: We benchmark the research on the NarrativeQA dataset with experiments with cutting-edge ODQA techniques. This quantifies the challenges Book QA poses, as well as advances the published state-of-the-art with a $sim$7% absolute improvement on Rouge-L. Our findings indicate that the event-centric questions dominate this task, which exemplifies the inability of existing QA models to handle event-oriented scenarios.
• Score: 45.9120218818558
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in open-domain question answering (ODQA), i.e., finding answers from large open-domain corpus like Wikipedia, have led to human-level performance on many datasets. However, progress in QA over book stories (Book QA) lags behind despite its similar task formulation to ODQA. This work provides a comprehensive and quantitative analysis about the difficulty of Book QA: (1) We benchmark the research on the NarrativeQA dataset with extensive experiments with cutting-edge ODQA techniques. This quantifies the challenges Book QA poses, as well as advances the published state-of-the-art with a $\sim$7\% absolute improvement on Rouge-L. (2) We further analyze the detailed challenges in Book QA through human studies.\footnote{\url{https://github.com/gorov/BookQA}.} Our findings indicate that the event-centric questions dominate this task, which exemplifies the inability of existing QA models to handle event-oriented scenarios.

Related papers

• DEXTER: A Benchmark for open-domain Complex Question Answering using LLMs [3.24692739098077]
  Open-domain complex Question Answering (QA) is a difficult task with challenges in evidence retrieval and reasoning. We evaluate state-of-the-art pre-trained dense and sparse retrieval models in an open-domain setting. We observe that late interaction models and surprisingly lexical models like BM25 perform well compared to other pre-trained dense retrieval models.
  arXiv  Detail & Related papers  (2024-06-24T22:09:50Z)
• IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions [54.23087908182134]
  We introduce the first large-scale counterfactual open-domain question-answering (QA) benchmarks, named IfQA. The IfQA dataset contains over 3,800 questions that were annotated by crowdworkers on relevant Wikipedia passages. The unique challenges posed by the IfQA benchmark will push open-domain QA research on both retrieval and counterfactual reasoning fronts.
  arXiv  Detail & Related papers  (2023-05-23T12:43:19Z)
• Closed-book Question Generation via Contrastive Learning [20.644215991166895]
  We propose a new QG model empowered by a contrastive learning module and an answer reconstruction module. We show how to leverage the proposed model to improve existing closed-book QA systems.
  arXiv  Detail & Related papers  (2022-10-13T06:45:46Z)
• Modern Question Answering Datasets and Benchmarks: A Survey [5.026863544662493]
  Question Answering (QA) is one of the most important natural language processing (NLP) tasks. It aims using NLP technologies to generate a corresponding answer to a given question based on the massive unstructured corpus. In this paper, we investigate influential QA datasets that have been released in the era of deep learning.
  arXiv  Detail & Related papers  (2022-06-30T05:53:56Z)
• Multifaceted Improvements for Conversational Open-Domain Question Answering [54.913313912927045]
  We propose a framework with Multifaceted Improvements for Conversational open-domain Question Answering (MICQA) Firstly, the proposed KL-divergence based regularization is able to lead to a better question understanding for retrieval and answer reading. Second, the added post-ranker module can push more relevant passages to the top placements and be selected for reader with a two-aspect constrains. Third, the well designed curriculum learning strategy effectively narrows the gap between the golden passage settings of training and inference, and encourages the reader to find true answer without the golden passage assistance.
  arXiv  Detail & Related papers  (2022-04-01T07:54:27Z)
• Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering [62.88322725956294]
  We review the latest research trends in OpenQA, with particular attention to systems that incorporate neural MRC techniques. We introduce modern OpenQA architecture named Retriever-Reader'' and analyze the various systems that follow this architecture. We then discuss key challenges to developing OpenQA systems and offer an analysis of benchmarks that are commonly used.
  arXiv  Detail & Related papers  (2021-01-04T04:47:46Z)
• Frustratingly Hard Evidence Retrieval for QA Over Books [42.451721693007954]
  We formulate BookQA as an open-domain QA task given its similar dependency on evidence retrieval. We investigate how state-of-the-art open-domain QA approaches can help BookQA.
  arXiv  Detail & Related papers  (2020-07-20T04:10:08Z)
• Harvesting and Refining Question-Answer Pairs for Unsupervised QA [95.9105154311491]
  We introduce two approaches to improve unsupervised Question Answering (QA) First, we harvest lexically and syntactically divergent questions from Wikipedia to automatically construct a corpus of question-answer pairs (named as RefQA) Second, we take advantage of the QA model to extract more appropriate answers, which iteratively refines data over RefQA.
  arXiv  Detail & Related papers  (2020-05-06T15:56:06Z)
• Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering [98.48363619128108]
  We propose an unsupervised approach to training QA models with generated pseudo-training data. We show that generating questions for QA training by applying a simple template on a related, retrieved sentence rather than the original context sentence improves downstream QA performance.
  arXiv  Detail & Related papers  (2020-04-24T17:57:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.