Few-Shot Question Answering by Pretraining Span Selection

• URL: http://arxiv.org/abs/2101.00438v1
• Date: Sat, 2 Jan 2021 11:58:44 GMT
• Title: Few-Shot Question Answering by Pretraining Span Selection
• Authors: Ori Ram and Yuval Kirstain and Jonathan Berant and Amir Globerson and Omer Levy
• Abstract summary: We explore the more realistic few-shot setting, where only a few hundred training examples are available. We show that standard span selection models perform poorly, highlighting the fact that current pretraining objective are far removed from question answering. Our findings indicate that careful design of pretraining schemes and model architecture can have a dramatic effect on performance in the few-shot settings.
• Score: 58.31911597824848
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: In a number of question answering (QA) benchmarks, pretrained models have reached human parity through fine-tuning on an order of 100,000 annotated questions and answers. We explore the more realistic few-shot setting, where only a few hundred training examples are available. We show that standard span selection models perform poorly, highlighting the fact that current pretraining objective are far removed from question answering. To address this, we propose a new pretraining scheme that is more suitable for extractive question answering. Given a passage with multiple sets of recurring spans, we mask in each set all recurring spans but one, and ask the model to select the correct span in the passage for each masked span. Masked spans are replaced with a special token, viewed as a question representation, that is later used during fine-tuning to select the answer span. The resulting model obtains surprisingly good results on multiple benchmarks, e.g., 72.7 F1 with only 128 examples on SQuAD, while maintaining competitive (and sometimes better) performance in the high-resource setting. Our findings indicate that careful design of pretraining schemes and model architecture can have a dramatic effect on performance in the few-shot settings.

Related papers

• Listening to the Wise Few: Select-and-Copy Attention Heads for Multiple-Choice QA [19.78468832417275]
  We introduce new scores that better capture and reveal model's underlying knowledge. Based on these scores, our method improves knowledge extraction, yielding up to 16% gain for LLaMA2-7B. The accuracy on a simple synthetic dataset, where the model explicitly knows the right answer, increases by almost 60%.
  arXiv  Detail & Related papers  (2024-10-03T09:53:48Z)
• Answering Ambiguous Questions via Iterative Prompting [84.3426020642704]
  In open-domain question answering, due to the ambiguity of questions, multiple plausible answers may exist. One approach is to directly predict all valid answers, but this can struggle with balancing relevance and diversity. We present AmbigPrompt to address the imperfections of existing approaches to answering ambiguous questions.
  arXiv  Detail & Related papers  (2023-07-08T04:32:17Z)
• Momentum Contrastive Pre-training for Question Answering [54.57078061878619]
  MCROSS introduces a momentum contrastive learning framework to align the answer probability between cloze-like and natural query-passage sample pairs. Our method achieves noticeable improvement compared with all baselines in both supervised and zero-shot scenarios.
  arXiv  Detail & Related papers  (2022-12-12T08:28:22Z)
• CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Training [21.07506671340319]
  We propose a novel question-answering dataset based on the Common Crawl project in this paper. We extract around 130 million multilingual question-answer pairs, including about 60 million English data-points. With this previously unseen number of natural QA pairs, we pre-train popular language models to show the potential of large-scale in-domain pre-training for the task of question-answering.
  arXiv  Detail & Related papers  (2021-10-14T21:23:01Z)
• FewshotQA: A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text models [0.0]
  Current state-of-the-art pre-trained models typically need fine-tuning on tens of thousands of examples to obtain good results. We propose a simple fine-tuning framework that leverages pre-trained text-to-text models and is directly aligned with their pre-training framework. Our model outperforms the XLM-Roberta-large by an absolute margin of upto 40 F1 points and an average of 33 F1 points in a few-shot setting.
  arXiv  Detail & Related papers  (2021-09-04T23:08:57Z)
• True Few-Shot Learning with Language Models [78.42578316883271]
  We evaluate the few-shot ability of LMs when held-out examples are unavailable. Our findings suggest that prior work significantly overestimated the true few-shot ability of LMs.
  arXiv  Detail & Related papers  (2021-05-24T17:55:51Z)
• Self-supervised pre-training and contrastive representation learning for multiple-choice video QA [39.78914328623504]
  Video Question Answering (Video QA) requires fine-grained understanding of both video and language modalities to answer the given questions. We propose novel training schemes for multiple-choice video question answering with a self-supervised pre-training stage and a supervised contrastive learning in the main stage as an auxiliary learning. We evaluate our proposed model on highly competitive benchmark datasets related to multiple-choice video QA: TVQA, TVQA+, and DramaQA.
  arXiv  Detail & Related papers  (2020-09-17T03:37:37Z)
• Robust Question Answering Through Sub-part Alignment [53.94003466761305]
  We model question answering as an alignment problem. We train our model on SQuAD v1.1 and test it on several adversarial and out-of-domain datasets.
  arXiv  Detail & Related papers  (2020-04-30T09:10:57Z)
• ManyModalQA: Modality Disambiguation and QA over Diverse Inputs [73.93607719921945]
  We present a new multimodal question answering challenge, ManyModalQA, in which an agent must answer a question by considering three distinct modalities. We collect our data by scraping Wikipedia and then utilize crowdsourcing to collect question-answer pairs.
  arXiv  Detail & Related papers  (2020-01-22T14:39:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.