Training Question Answering Models From Synthetic Data

• URL: http://arxiv.org/abs/2002.09599v1
• Date: Sat, 22 Feb 2020 01:49:27 GMT
• Title: Training Question Answering Models From Synthetic Data
• Authors: Raul Puri, Ryan Spring, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro
• Abstract summary: This work aims to narrow the gap between synthetic and human-generated question-answer pairs. We synthesize questions and answers from a synthetic corpus generated by an 8.3 billion parameter GPT-2 model. With no access to human supervision and only access to other models, we are able to train state of the art question answering networks on entirely model-generated data.
• Score: 26.91650323300262
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Question and answer generation is a data augmentation method that aims to improve question answering (QA) models given the limited amount of human labeled data. However, a considerable gap remains between synthetic and human-generated question-answer pairs. This work aims to narrow this gap by taking advantage of large language models and explores several factors such as model size, quality of pretrained models, scale of data synthesized, and algorithmic choices. On the SQuAD1.1 question answering task, we achieve higher accuracy using solely synthetic questions and answers than when using the SQuAD1.1 training set questions alone. Removing access to real Wikipedia data, we synthesize questions and answers from a synthetic corpus generated by an 8.3 billion parameter GPT-2 model. With no access to human supervision and only access to other models, we are able to train state of the art question answering networks on entirely model-generated data that achieve 88.4 Exact Match (EM) and 93.9 F1 score on the SQuAD1.1 dev set. We further apply our methodology to SQuAD2.0 and show a 2.8 absolute gain on EM score compared to prior work using synthetic data.

Related papers

• Chatting Up Attachment: Using LLMs to Predict Adult Bonds [0.0]
  We use GPT-4 and Claude 3 Opus to create agents that simulate adults with varying profiles, childhood memories, and attachment styles. We evaluate our models using a transcript dataset from 9 humans who underwent the same interview protocol, analyzed and labeled by mental health professionals. Our findings indicate that training the models using only synthetic data achieves performance comparable to training the models on human data.
  arXiv  Detail & Related papers  (2024-08-31T04:29:19Z)
• GSQA: An End-to-End Model for Generative Spoken Question Answering [54.418723701886115]
  We introduce the first end-to-end Generative Spoken Question Answering (GSQA) model that empowers the system to engage in abstractive reasoning. Our model surpasses the previous extractive model by 3% on extractive QA datasets. Our GSQA model shows the potential to generalize to a broad spectrum of questions, thus further expanding the spoken question answering capabilities of abstractive QA.
  arXiv  Detail & Related papers  (2023-12-15T13:33:18Z)
• Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models [115.501751261878]
  Fine-tuning language models(LMs) on human-generated data remains a prevalent practice. We investigate whether we can go beyond human data on tasks where we have access to scalar feedback. We find that ReST$EM$ scales favorably with model size and significantly surpasses fine-tuning only on human data.
  arXiv  Detail & Related papers  (2023-12-11T18:17:43Z)
• A Lightweight Method to Generate Unanswerable Questions in English [18.323248259867356]
  We examine a simpler data augmentation method for unanswerable question generation in English. We perform antonym and entity swaps on answerable questions. Compared to the prior state-of-the-art, data generated with our training-free and lightweight strategy results in better models.
  arXiv  Detail & Related papers  (2023-10-30T10:14:52Z)
• QUADRo: Dataset and Models for QUestion-Answer Database Retrieval [97.84448420852854]
  Given a database (DB) of question/answer (q/a) pairs, it is possible to answer a target question by scanning the DB for similar questions. We build a large scale DB of 6.3M q/a pairs, using public questions, and design a new system based on neural IR and a q/a pair reranker. We show that our DB-based approach is competitive with Web-based methods, i.e., a QA system built on top the BING search engine.
  arXiv  Detail & Related papers  (2023-03-30T00:42:07Z)
• Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation [41.9785159975426]
  State-of-the-art question answering models remain susceptible to a variety of adversarial attacks and are still far from obtaining human-level language understanding. One proposed way forward is dynamic adversarial data collection, in which a human annotator attempts to create examples for which a model-in-the-loop fails. In this work, we investigate several answer selection, question generation, and filtering methods that form a synthetic adversarial data generation pipeline. Models trained on both synthetic and human-generated data outperform models not trained on synthetic adversarial data, and obtain state-of-the-art results on the Adversarial
  arXiv  Detail & Related papers  (2021-04-18T02:00:06Z)
• Stacking Neural Network Models for Automatic Short Answer Scoring [0.0]
  We propose the use of a stacking model based on neural network and XGBoost for classification process with sentence embedding feature. Best model obtained an F1-score of 0.821 exceeding the previous work at the same dataset.
  arXiv  Detail & Related papers  (2020-10-21T16:00:09Z)
• When in Doubt, Ask: Generating Answerable and Unanswerable Questions, Unsupervised [0.0]
  Question Answering (QA) is key for making possible a robust communication between human and machine. Modern language models used for QA have surpassed the human-performance in several essential tasks. This paper studies augmenting human-made datasets with synthetic data as a way of surmounting this problem.
  arXiv  Detail & Related papers  (2020-10-04T15:56:44Z)
• Selective Question Answering under Domain Shift [90.021577320085]
  Abstention policies based solely on the model's softmax probabilities fare poorly, since models are overconfident on out-of-domain inputs. We train a calibrator to identify inputs on which the QA model errs, and abstain when it predicts an error is likely. Our method answers 56% of questions while maintaining 80% accuracy; in contrast, directly using the model's probabilities only answers 48% at 80% accuracy.
  arXiv  Detail & Related papers  (2020-06-16T19:13:21Z)
• Harvesting and Refining Question-Answer Pairs for Unsupervised QA [95.9105154311491]
  We introduce two approaches to improve unsupervised Question Answering (QA) First, we harvest lexically and syntactically divergent questions from Wikipedia to automatically construct a corpus of question-answer pairs (named as RefQA) Second, we take advantage of the QA model to extract more appropriate answers, which iteratively refines data over RefQA.
  arXiv  Detail & Related papers  (2020-05-06T15:56:06Z)
• Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering [98.48363619128108]
  We propose an unsupervised approach to training QA models with generated pseudo-training data. We show that generating questions for QA training by applying a simple template on a related, retrieved sentence rather than the original context sentence improves downstream QA performance.
  arXiv  Detail & Related papers  (2020-04-24T17:57:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.