Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets

• URL: http://arxiv.org/abs/2008.02637v1
• Date: Thu, 6 Aug 2020 13:17:43 GMT
• Title: Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets
• Abstract summary: We study the test sets of three popular open-domain benchmark datasets. We find that 60-70% of test-time answers are also present somewhere in the training sets. We also find that 30% of test-set questions have a near-duplicate paraphrase in their corresponding training sets.
• Score: 37.02260241742407
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Ideally Open-Domain Question Answering models should exhibit a number of competencies, ranging from simply memorizing questions seen at training time, to answering novel question formulations with answers seen during training, to generalizing to completely novel questions with novel answers. However, single aggregated test set scores do not show the full picture of what capabilities models truly have. In this work, we perform a detailed study of the test sets of three popular open-domain benchmark datasets with respect to these competencies. We find that 60-70% of test-time answers are also present somewhere in the training sets. We also find that 30% of test-set questions have a near-duplicate paraphrase in their corresponding training sets. Using these findings, we evaluate a variety of popular open-domain models to obtain greater insight into what extent they can actually generalize, and what drives their overall performance. We find that all models perform dramatically worse on questions that cannot be memorized from training sets, with a mean absolute performance difference of 63% between repeated and non-repeated data. Finally we show that simple nearest-neighbor models out-perform a BART closed-book QA model, further highlighting the role that training set memorization plays in these benchmarks

Related papers

• No Single Best Model for Diversity: Learning a Router for Sample Diversity [69.53166985556759]
  In this paper, we study methods to elicit a comprehensive set of valid responses.<n>We introduce textbfdiversity coverage, a metric that measures the total quality scores assigned to each textbfunique answer.<n>Per each prompt, there exists a model that outperforms all other models significantly at generating a diverse answer set.
  arXiv  Detail & Related papers  (2026-04-02T17:58:37Z)
• Robust Training for Conversational Question Answering Models with Reinforced Reformulation Generation [26.752549844734034]
  We show that ConvQA models with robust training via reformulations, significantly outperform those with standard training from gold QA pairs only. We demonstrate the viability of training major model components on one benchmark and applying them zero-shot to another.
  arXiv  Detail & Related papers  (2023-10-20T13:51:08Z)
• QUADRo: Dataset and Models for QUestion-Answer Database Retrieval [97.84448420852854]
  Given a database (DB) of question/answer (q/a) pairs, it is possible to answer a target question by scanning the DB for similar questions. We build a large scale DB of 6.3M q/a pairs, using public questions, and design a new system based on neural IR and a q/a pair reranker. We show that our DB-based approach is competitive with Web-based methods, i.e., a QA system built on top the BING search engine.
  arXiv  Detail & Related papers  (2023-03-30T00:42:07Z)
• Challenges in Generalization in Open Domain Question Answering [16.63912089965166]
  We introduce and annotate questions according to three categories that measure different levels and kinds of generalization. Key question difficulty factors are cascading errors from the retrieval component, frequency of question pattern, and frequency of the entity.
  arXiv  Detail & Related papers  (2021-09-02T18:04:10Z)
• Learning Compositional Representation for Few-shot Visual Question Answering [93.4061107793983]
  Current methods of Visual Question Answering perform well on the answers with an amount of training data but have limited accuracy on the novel ones with few examples. We propose to extract the attributes from the answers with enough data, which are later composed to constrain the learning of the few-shot ones. Experimental results on the VQA v2.0 validation dataset demonstrate the effectiveness of our proposed attribute network.
  arXiv  Detail & Related papers  (2021-02-21T10:16:24Z)
• Selective Question Answering under Domain Shift [90.021577320085]
  Abstention policies based solely on the model's softmax probabilities fare poorly, since models are overconfident on out-of-domain inputs. We train a calibrator to identify inputs on which the QA model errs, and abstain when it predicts an error is likely. Our method answers 56% of questions while maintaining 80% accuracy; in contrast, directly using the model's probabilities only answers 48% at 80% accuracy.
  arXiv  Detail & Related papers  (2020-06-16T19:13:21Z)
• MultiReQA: A Cross-Domain Evaluation for Retrieval Question Answering Models [25.398047573530985]
  Retrieval question answering (ReQA) is the task of retrieving a sentence-level answer to a question from an open corpus. This paper presents MultiReQA, anew multi-domain ReQA evaluation suite com-posed of eight retrieval QA tasks drawn from publicly available QA datasets.
  arXiv  Detail & Related papers  (2020-05-05T21:30:16Z)
• ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoning [35.6375880208001]
  This paper introduces a new question answering dataset for training and evaluating common sense reasoning capabilities of artificial intelligence systems. The training set is gathered from an existing set of questions played in a long-running international game show FAMILY- FEUD. We also propose a generative evaluation task where a model has to output a ranked list of answers, ideally covering prototypical answers for a question.
  arXiv  Detail & Related papers  (2020-05-02T09:40:05Z)
• Robust Question Answering Through Sub-part Alignment [53.94003466761305]
  We model question answering as an alignment problem. We train our model on SQuAD v1.1 and test it on several adversarial and out-of-domain datasets.
  arXiv  Detail & Related papers  (2020-04-30T09:10:57Z)
• ManyModalQA: Modality Disambiguation and QA over Diverse Inputs [73.93607719921945]
  We present a new multimodal question answering challenge, ManyModalQA, in which an agent must answer a question by considering three distinct modalities. We collect our data by scraping Wikipedia and then utilize crowdsourcing to collect question-answer pairs.
  arXiv  Detail & Related papers  (2020-01-22T14:39:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.