Harnessing the Power of Prompt-based Techniques for Generating School-Level Questions using Large Language Models

• URL: http://arxiv.org/abs/2312.01032v1
• Date: Sat, 2 Dec 2023 05:13:28 GMT
• Title: Harnessing the Power of Prompt-based Techniques for Generating School-Level Questions using Large Language Models
• Authors: Subhankar Maity, Aniket Deroy, Sudeshna Sarkar
• Abstract summary: We propose a novel approach that utilizes prompt-based techniques to generate descriptive and reasoning-based questions. We curate a new QG dataset called EduProbe for school-level subjects, by leveraging the rich content of NCERT textbooks. We investigate several prompt-based QG methods by fine-tuning transformer-based large language models.
• Score: 0.5459032912385802
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Designing high-quality educational questions is a challenging and time-consuming task. In this work, we propose a novel approach that utilizes prompt-based techniques to generate descriptive and reasoning-based questions. However, current question-answering (QA) datasets are inadequate for conducting our experiments on prompt-based question generation (QG) in an educational setting. Therefore, we curate a new QG dataset called EduProbe for school-level subjects, by leveraging the rich content of NCERT textbooks. We carefully annotate this dataset as quadruples of 1) Context: a segment upon which the question is formed; 2) Long Prompt: a long textual cue for the question (i.e., a longer sequence of words or phrases, covering the main theme of the context); 3) Short Prompt: a short textual cue for the question (i.e., a condensed representation of the key information or focus of the context); 4) Question: a deep question that aligns with the context and is coherent with the prompts. We investigate several prompt-based QG methods by fine-tuning pre-trained transformer-based large language models (LLMs), namely PEGASUS, T5, MBART, and BART. Moreover, we explore the performance of two general-purpose pre-trained LLMs such as Text-Davinci-003 and GPT-3.5-Turbo without any further training. By performing automatic evaluation, we show that T5 (with long prompt) outperforms all other models, but still falls short of the human baseline. Under human evaluation criteria, TextDavinci-003 usually shows better results than other models under various prompt settings. Even in the case of human evaluation criteria, QG models mostly fall short of the human baseline. Our code and dataset are available at: https://github.com/my625/PromptQG

Related papers

• Diversity Enhanced Narrative Question Generation for Storybooks [4.043005183192124]
  We introduce a multi-question generation model (mQG) capable of generating multiple, diverse, and answerable questions. To validate the answerability of the generated questions, we employ a SQuAD2.0 fine-tuned question answering model. mQG shows promising results across various evaluation metrics, among strong baselines.
  arXiv  Detail & Related papers  (2023-10-25T08:10:04Z)
• Improving Question Generation with Multi-level Content Planning [70.37285816596527]
  This paper addresses the problem of generating questions from a given context and an answer, specifically focusing on questions that require multi-hop reasoning across an extended context. We propose MultiFactor, a novel QG framework based on multi-level content planning. Specifically, MultiFactor includes two components: FA-model, which simultaneously selects key phrases and generates full answers, and Q-model which takes the generated full answer as an additional input to generate questions.
  arXiv  Detail & Related papers  (2023-10-20T13:57:01Z)
• Prompting Large Language Models with Chain-of-Thought for Few-Shot Knowledge Base Question Generation [19.327008532572645]
  Question Generation over Knowledge Bases (KBQG) aims to convert a logical form into a natural language question. We propose Chain-of-Thought prompting, which is an in-context learning strategy for reasoning. We conduct extensive experiments over three public KBQG datasets.
  arXiv  Detail & Related papers  (2023-10-12T15:08:14Z)
• Answer Candidate Type Selection: Text-to-Text Language Model for Closed Book Question Answering Meets Knowledge Graphs [62.20354845651949]
  We present a novel approach which works on top of the pre-trained Text-to-Text QA system to address this issue. Our simple yet effective method performs filtering and re-ranking of generated candidates based on their types derived from Wikidata "instance_of" property.
  arXiv  Detail & Related papers  (2023-10-10T20:49:43Z)
• Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning [150.17907456113537]
  We present Tabular Math Word Problems (TabMWP), a new dataset containing 38,431 grade-level problems that require mathematical reasoning. We evaluate different pre-trained models on TabMWP, including the GPT-3 model in a few-shot setting. We propose a novel approach, PromptPG, which utilizes policy gradient to learn to select in-context examples from a small amount of training data.
  arXiv  Detail & Related papers  (2022-09-29T08:01:04Z)
• Towards Complex Document Understanding By Discrete Reasoning [77.91722463958743]
  Document Visual Question Answering (VQA) aims to understand visually-rich documents to answer questions in natural language. We introduce a new Document VQA dataset, named TAT-DQA, which consists of 3,067 document pages and 16,558 question-answer pairs. We develop a novel model named MHST that takes into account the information in multi-modalities, including text, layout and visual image, to intelligently address different types of questions.
  arXiv  Detail & Related papers  (2022-07-25T01:43:19Z)
• Fantastic Questions and Where to Find Them: FairytaleQA -- An Authentic Dataset for Narrative Comprehension [136.82507046638784]
  We introduce FairytaleQA, a dataset focusing on narrative comprehension of kindergarten to eighth-grade students. FairytaleQA consists of 10,580 explicit and implicit questions derived from 278 children-friendly stories.
  arXiv  Detail & Related papers  (2022-03-26T00:20:05Z)
• Inquisitive Question Generation for High Level Text Comprehension [60.21497846332531]
  We introduce INQUISITIVE, a dataset of 19K questions that are elicited while a person is reading through a document. We show that readers engage in a series of pragmatic strategies to seek information. We evaluate question generation models based on GPT-2 and show that our model is able to generate reasonable questions.
  arXiv  Detail & Related papers  (2020-10-04T19:03:39Z)
• Simplifying Paragraph-level Question Generation via Transformer Language Models [0.0]
  Question generation (QG) is a natural language generation task where a model is trained to ask questions corresponding to some input text. A single Transformer-based unidirectional language model leveraging transfer learning can be used to produce high quality questions. Our QG model, finetuned from GPT-2 Small, outperforms several paragraph-level QG baselines on the SQuAD dataset by 0.95 METEOR points.
  arXiv  Detail & Related papers  (2020-05-03T14:57:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.