Better Distractions: Transformer-based Distractor Generation and Multiple Choice Question Filtering

• URL: http://arxiv.org/abs/2010.09598v1
• Date: Mon, 19 Oct 2020 15:23:24 GMT
• Title: Better Distractions: Transformer-based Distractor Generation and Multiple Choice Question Filtering
• Authors: Jeroen Offerijns, Suzan Verberne, Tessa Verhoef
• Abstract summary: We train a GPT-2 language model to generate three distractors for a given question and text context. Next, we train a BERT language model to answer multiple choice questions (MCQs) and use this model as a filter.
• Score: 4.168157981135697
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: For the field of education, being able to generate semantically correct and educationally relevant multiple choice questions (MCQs) could have a large impact. While question generation itself is an active research topic, generating distractors (the incorrect multiple choice options) receives much less attention. A missed opportunity, since there is still a lot of room for improvement in this area. In this work, we train a GPT-2 language model to generate three distractors for a given question and text context, using the RACE dataset. Next, we train a BERT language model to answer MCQs, and use this model as a filter, to select only questions that can be answered and therefore presumably make sense. To evaluate our work, we start by using text generation metrics, which show that our model outperforms earlier work on distractor generation (DG) and achieves state-of-the-art performance. Also, by calculating the question answering ability, we show that larger base models lead to better performance. Moreover, we conducted a human evaluation study, which confirmed the quality of the generated questions, but showed no statistically significant effect of the QA filter.

Related papers

• DisGeM: Distractor Generation for Multiple Choice Questions with Span Masking [0.0]
  We present a generic framework for distractor generation for multiple-choice questions (MCQ) Our framework relies solely on pre-trained language models and does not require additional training on specific datasets. Human evaluations confirm that our approach produces more effective and engaging distractors.
  arXiv  Detail & Related papers  (2024-09-26T20:15:46Z)
• Improving Automated Distractor Generation for Math Multiple-choice Questions with Overgenerate-and-rank [44.04217284677347]
  We propose a novel method to enhance the quality of generated distractors through overgenerate-and-rank. Our ranking model increases alignment with human-authored distractors, although human-authored ones are still preferred over generated ones.
  arXiv  Detail & Related papers  (2024-04-19T00:25:44Z)
• CDGP: Automatic Cloze Distractor Generation based on Pre-trained Language Model [2.2169618382995764]
  We explore the employment of pre-trained language models (PLMs) as an alternative for candidate distractor generation. Experiments show that the PLM-enhanced model brings a substantial performance improvement.
  arXiv  Detail & Related papers  (2024-03-15T14:14:26Z)
• Improving Question Generation with Multi-level Content Planning [70.37285816596527]
  This paper addresses the problem of generating questions from a given context and an answer, specifically focusing on questions that require multi-hop reasoning across an extended context. We propose MultiFactor, a novel QG framework based on multi-level content planning. Specifically, MultiFactor includes two components: FA-model, which simultaneously selects key phrases and generates full answers, and Q-model which takes the generated full answer as an additional input to generate questions.
  arXiv  Detail & Related papers  (2023-10-20T13:57:01Z)
• UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Models [55.22048505787125]
  This paper contributes a comprehensive dataset, called UNK-VQA. We first augment the existing data via deliberate perturbations on either the image or question. We then extensively evaluate the zero- and few-shot performance of several emerging multi-modal large models.
  arXiv  Detail & Related papers  (2023-10-17T02:38:09Z)
• Negated Complementary Commonsense using Large Language Models [3.42658286826597]
  This work focuses on finding answers to negated complementary questions in commonsense scenarios. We propose a model-agnostic methodology to improve the performance in negated complementary scenarios.
  arXiv  Detail & Related papers  (2023-07-13T15:03:48Z)
• Weakly Supervised Visual Question Answer Generation [2.7605547688813172]
  We present a weakly supervised method that synthetically generates question-answer pairs procedurally from visual information and captions. We perform an exhaustive experimental analysis on VQA dataset and see that our model significantly outperforms SOTA methods on BLEU scores.
  arXiv  Detail & Related papers  (2023-06-11T08:46:42Z)
• Read before Generate! Faithful Long Form Question Answering with Machine Reading [77.17898499652306]
  Long-form question answering (LFQA) aims to generate a paragraph-length answer for a given question. We propose a new end-to-end framework that jointly models answer generation and machine reading.
  arXiv  Detail & Related papers  (2022-03-01T10:41:17Z)
• Answer Generation for Retrieval-based Question Answering Systems [80.28727681633096]
  We train a sequence to sequence transformer model to generate an answer from a candidate set. Our tests on three English AS2 datasets show improvement up to 32 absolute points in accuracy over the state of the art.
  arXiv  Detail & Related papers  (2021-06-02T05:45:49Z)
• Stronger Transformers for Neural Multi-Hop Question Generation [48.06692942528804]
  We introduce a series of strong transformer models for multi-hop question generation. We show that we can substantially outperform the state-of-the-art by 5 BLEU points using a standard transformer architecture.
  arXiv  Detail & Related papers  (2020-10-22T01:51:09Z)
• Improving Multi-Turn Response Selection Models with Complementary Last-Utterance Selection by Instance Weighting [84.9716460244444]
  We consider utilizing the underlying correlation in the data resource itself to derive different kinds of supervision signals. We conduct extensive experiments in two public datasets and obtain significant improvement in both datasets.
  arXiv  Detail & Related papers  (2020-02-18T06:29:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.