QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation

• URL: http://arxiv.org/abs/2406.05707v2
• Date: Thu, 10 Oct 2024 15:12:23 GMT
• Title: QGEval: Benchmarking Multi-dimensional Evaluation for Question Generation
• Authors: Weiping Fu, Bifan Wei, Jianxiang Hu, Zhongmin Cai, Jun Liu,
• Abstract summary: Human evaluation is widely used in the field of question generation (QG) and serves as the gold standard for automatic metrics. There is a lack of unified human evaluation criteria, which hampers consistent evaluations of both QG models and automatic metrics. We propose QGEval, a multi-dimensional Evaluation benchmark for Question Generation, which evaluates both generated questions and existing automatic metrics across 7 dimensions.
• Score: 9.001613702628253
• License:
• Abstract: Automatically generated questions often suffer from problems such as unclear expression or factual inaccuracies, requiring a reliable and comprehensive evaluation of their quality. Human evaluation is widely used in the field of question generation (QG) and serves as the gold standard for automatic metrics. However, there is a lack of unified human evaluation criteria, which hampers consistent and reliable evaluations of both QG models and automatic metrics. To address this, we propose QGEval, a multi-dimensional Evaluation benchmark for Question Generation, which evaluates both generated questions and existing automatic metrics across 7 dimensions: fluency, clarity, conciseness, relevance, consistency, answerability, and answer consistency. We demonstrate the appropriateness of these dimensions by examining their correlations and distinctions. Through consistent evaluations of QG models and automatic metrics with QGEval, we find that 1) most QG models perform unsatisfactorily in terms of answerability and answer consistency, and 2) existing metrics fail to align well with human judgments when evaluating generated questions across the 7 dimensions. We expect this work to foster the development of both QG technologies and their evaluation.

Related papers

• A Step Towards Mixture of Grader: Statistical Analysis of Existing Automatic Evaluation Metrics [6.571049277167304]
  We study the statistics of the existing evaluation metrics for a better understanding of their limitations. As a potential solution, we discuss how a Mixture Of Grader could potentially improve the auto QA evaluator quality.
  arXiv  Detail & Related papers  (2024-10-13T22:10:42Z)
• An Automatic Question Usability Evaluation Toolkit [1.2499537119440245]
  evaluating multiple-choice questions (MCQs) involves either labor intensive human assessments or automated methods that prioritize readability. We introduce SAQUET, an open-source tool that leverages the Item-Writing Flaws (IWF) rubric for a comprehensive and automated quality evaluation of MCQs. With an accuracy rate of over 94%, our findings emphasize the limitations of existing evaluation methods and showcase potential in improving the quality of educational assessments.
  arXiv  Detail & Related papers  (2024-05-30T23:04:53Z)
• Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation [64.64849950642619]
  We develop an evaluation framework inspired by formal semantics for evaluating text-to-image models. We show that Davidsonian Scene Graph (DSG) produces atomic and unique questions organized in dependency graphs. We also present DSG-1k, an open-sourced evaluation benchmark that includes 1,060 prompts.
  arXiv  Detail & Related papers  (2023-10-27T16:20:10Z)
• QUDEVAL: The Evaluation of Questions Under Discussion Discourse Parsing [87.20804165014387]
  Questions Under Discussion (QUD) is a versatile linguistic framework in which discourse progresses as continuously asking questions and answering them. This work introduces the first framework for the automatic evaluation of QUD parsing. We present QUDeval, a dataset of fine-grained evaluation of 2,190 QUD questions generated from both fine-tuned systems and LLMs.
  arXiv  Detail & Related papers  (2023-10-23T03:03:58Z)
• Automatic Answerability Evaluation for Question Generation [32.1067137848404]
  This work proposes PMAN, a novel automatic evaluation metric to assess whether the generated questions are answerable by the reference answers. Our implementation of a GPT-based QG model achieves state-of-the-art performance in generating answerable questions.
  arXiv  Detail & Related papers  (2023-09-22T00:13:07Z)
• SQUARE: Automatic Question Answering Evaluation using Multiple Positive and Negative References [73.67707138779245]
  We propose a new evaluation metric: SQuArE (Sentence-level QUestion AnsweRing Evaluation) We evaluate SQuArE on both sentence-level extractive (Answer Selection) and generative (GenQA) QA systems.
  arXiv  Detail & Related papers  (2023-09-21T16:51:30Z)
• QAScore -- An Unsupervised Unreferenced Metric for the Question Generation Evaluation [6.697751970080859]
  Question Generation (QG) aims to automate the task of composing questions for a passage with a set of chosen answers. We propose a new reference-free evaluation metric that has the potential to provide a better mechanism for evaluating QG systems, called QAScore.
  arXiv  Detail & Related papers  (2022-10-09T19:00:39Z)
• GO FIGURE: A Meta Evaluation of Factuality in Summarization [131.1087461486504]
  We introduce GO FIGURE, a meta-evaluation framework for evaluating factuality evaluation metrics. Our benchmark analysis on ten factuality metrics reveals that our framework provides a robust and efficient evaluation. It also reveals that while QA metrics generally improve over standard metrics that measure factuality across domains, performance is highly dependent on the way in which questions are generated.
  arXiv  Detail & Related papers  (2020-10-24T08:30:20Z)
• KPQA: A Metric for Generative Question Answering Using Keyphrase Weights [64.54593491919248]
  KPQA-metric is a new metric for evaluating correctness of generative question answering systems. Our new metric assigns different weights to each token via keyphrase prediction. We show that our proposed metric has a significantly higher correlation with human judgments than existing metrics.
  arXiv  Detail & Related papers  (2020-05-01T03:24:36Z)
• Asking and Answering Questions to Evaluate the Factual Consistency of Summaries [80.65186293015135]
  We propose an automatic evaluation protocol called QAGS (pronounced "kags") to identify factual inconsistencies in a generated summary. QAGS is based on the intuition that if we ask questions about a summary and its source, we will receive similar answers if the summary is factually consistent with the source. We believe QAGS is a promising tool in automatically generating usable and factually consistent text.
  arXiv  Detail & Related papers  (2020-04-08T20:01:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.