Semantic-based Self-Critical Training For Question Generation

• URL: http://arxiv.org/abs/2108.12026v1
• Date: Thu, 26 Aug 2021 20:33:35 GMT
• Title: Semantic-based Self-Critical Training For Question Generation
• Authors: Lo\"ic, Kwate Dassi
• Abstract summary: We present a fully Transformer-based reinforcement learning generator-evaluator architecture for neural question generation. We come up with a semantic-based self-critical training layout in generator-evaluator architecture, which goes beyond typical maximum likelihood training.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present in this work a fully Transformer-based reinforcement learning generator-evaluator architecture for neural question generation. Question generation is a task that consists in generating questions given a context and answer. To improve the quality of the generated question, we came up with a semantic-based self-critical training layout in generator-evaluator architecture, which goes beyond typical maximum likelihood training. Evaluation metrics for language modeling only based on n-gram overlapping do not consider semantic relations between reference and candidate strings. To improve the evaluation step, we assess our model for both n-gram overlap using BLEU and semantically using BERTScore and NUBIA, a novel state-of-the-art evaluation metric for text generation. Question generation could be used in many downstream applications, including in extending question answering datasets, conversational systems, and educational assessment systems.

Related papers

• Exploring Precision and Recall to assess the quality and diversity of LLMs [82.21278402856079]
  We introduce a novel evaluation framework for Large Language Models (LLMs) such as textscLlama-2 and textscMistral. This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora.
  arXiv  Detail & Related papers  (2024-02-16T13:53:26Z)
• Evaluating Generative Ad Hoc Information Retrieval [58.800799175084286]
  generative retrieval systems often directly return a grounded generated text as a response to a query. Quantifying the utility of the textual responses is essential for appropriately evaluating such generative ad hoc retrieval.
  arXiv  Detail & Related papers  (2023-11-08T14:05:00Z)
• Scalable Learning of Latent Language Structure With Logical Offline Cycle Consistency [71.42261918225773]
  Conceptually, LOCCO can be viewed as a form of self-learning where the semantic being trained is used to generate annotations for unlabeled text. As an added bonus, the annotations produced by LOCCO can be trivially repurposed to train a neural text generation model.
  arXiv  Detail & Related papers  (2023-05-31T16:47:20Z)
• An Empirical Comparison of LM-based Question and Answer Generation Methods [79.31199020420827]
  Question and answer generation (QAG) consists of generating a set of question-answer pairs given a context. In this paper, we establish baselines with three different QAG methodologies that leverage sequence-to-sequence language model (LM) fine-tuning. Experiments show that an end-to-end QAG model, which is computationally light at both training and inference times, is generally robust and outperforms other more convoluted approaches.
  arXiv  Detail & Related papers  (2023-05-26T14:59:53Z)
• Automatic Short Math Answer Grading via In-context Meta-learning [2.0263791972068628]
  We study the problem of automatic short answer grading for students' responses to math questions. We use MathBERT, a variant of the popular language model BERT adapted to mathematical content, as our base model. Second, we use an in-context learning approach that provides scoring examples as input to the language model.
  arXiv  Detail & Related papers  (2022-05-30T16:26:02Z)
• Context Matters in Semantically Controlled Language Generation for Task-oriented Dialogue Systems [6.1478669848771546]
  This work combines information about the dialogue history encoded by pre-trained model with a meaning representation of the current system utterance to realize contextual language generation in task-oriented dialogues. We utilize the pre-trained multi-context ConveRT model for context representation in a model trained from scratch; and leverage the immediate preceding user utterance for context generation in a model adapted from the pre-trained GPT-2.
  arXiv  Detail & Related papers  (2021-11-28T11:48:02Z)
• Cooperative Learning of Zero-Shot Machine Reading Comprehension [9.868221447090855]
  We propose a cooperative, self-play learning model for question generation and answering. We can train question generation and answering models on any textual corpora without annotation. Our model outperforms the state-of-the-art pretrained language models on standard question answering benchmarks.
  arXiv  Detail & Related papers  (2021-03-12T18:22:28Z)
• Unsupervised Paraphrasing with Pretrained Language Models [85.03373221588707]
  We propose a training pipeline that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting. Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dynamic Blocking. We show with automatic and human evaluations that our approach achieves state-of-the-art performance on both the Quora Question Pair and the ParaNMT datasets.
  arXiv  Detail & Related papers  (2020-10-24T11:55:28Z)
• Simplifying Paragraph-level Question Generation via Transformer Language Models [0.0]
  Question generation (QG) is a natural language generation task where a model is trained to ask questions corresponding to some input text. A single Transformer-based unidirectional language model leveraging transfer learning can be used to produce high quality questions. Our QG model, finetuned from GPT-2 Small, outperforms several paragraph-level QG baselines on the SQuAD dataset by 0.95 METEOR points.
  arXiv  Detail & Related papers  (2020-05-03T14:57:24Z)
• NUBIA: NeUral Based Interchangeability Assessor for Text Generation [0.0]
  We present NUBIA, a methodology to build automatic evaluation metrics for text generation using only machine learning models as core components. A typical NUBIA model is composed of three modules: a neural feature extractor, an aggregator and a calibrator. We demonstrate an implementation of NUBIA which outperforms metrics currently used to evaluate machine translation, summaries and slightly exceeds/matches state of the art metrics on correlation with human judgement.
  arXiv  Detail & Related papers  (2020-04-30T10:11:33Z)
• PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation [92.7366819044397]
  Self-supervised pre-training has emerged as a powerful technique for natural language understanding and generation. This work presents PALM with a novel scheme that jointly pre-trains an autoencoding and autoregressive language model on a large unlabeled corpus. An extensive set of experiments show that PALM achieves new state-of-the-art results on a variety of language generation benchmarks.
  arXiv  Detail & Related papers  (2020-04-14T06:25:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.