From Model to Classroom: Evaluating Generated MCQs for Portuguese with Narrative and Difficulty Concerns

• URL: http://arxiv.org/abs/2506.15598v1
• Date: Wed, 18 Jun 2025 16:19:46 GMT
• Title: From Model to Classroom: Evaluating Generated MCQs for Portuguese with Narrative and Difficulty Concerns
• Abstract summary: This paper investigates the capabilities of current generative models in producing multiple choice questions (McQs) for reading comprehension in Portuguese.<n>Our results show that current models can generate MCQs of comparable quality to human-authored ones.<n>However, we identify issues related to semantic clarity and answerability.
• Score: 0.22585387137796725
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While MCQs are valuable for learning and evaluation, manually creating them with varying difficulty levels and targeted reading skills remains a time-consuming and costly task. Recent advances in generative AI provide an opportunity to automate MCQ generation efficiently. However, assessing the actual quality and reliability of generated MCQs has received limited attention -- particularly regarding cases where generation fails. This aspect becomes particularly important when the generated MCQs are meant to be applied in real-world settings. Additionally, most MCQ generation studies focus on English, leaving other languages underexplored. This paper investigates the capabilities of current generative models in producing MCQs for reading comprehension in Portuguese, a morphologically rich language. Our study focuses on generating MCQs that align with curriculum-relevant narrative elements and span different difficulty levels. We evaluate these MCQs through expert review and by analyzing the psychometric properties extracted from student responses to assess their suitability for elementary school students. Our results show that current models can generate MCQs of comparable quality to human-authored ones. However, we identify issues related to semantic clarity and answerability. Also, challenges remain in generating distractors that engage students and meet established criteria for high-quality MCQ option design.

Related papers

• Orchestrating LLM Agents for Scientific Research: A Pilot Study of Multiple Choice Question (MCQ) Generation and Evaluation [0.0]
  Large language models (LLMs) are rapidly transforming scientific work, yet empirical evidence on how these systems reshape research activities remains limited.<n>We report a mixed-methods pilot evaluation of an AI-orchestrated research workflow in which a human researcher coordinated multiple LLM-based agents to perform data extraction, corpus construction, artifact generation, and artifact evaluation.
  arXiv  Detail & Related papers  (2026-02-21T16:20:07Z)
• Cognitively Diverse Multiple-Choice Question Generation: A Hybrid Multi-Agent Framework with Large Language Models [4.155649113742267]
  ReQUESTA is a hybrid, multi-agent framework for generating cognitively diverse multiple-choice questions (MCQs)<n>We evaluated the framework in a large-scale reading comprehension study using academic expository texts.<n>Results showed that ReQUESTA-generated items were consistently more challenging, more discriminative, and more strongly aligned with overall reading comprehension performance.
  arXiv  Detail & Related papers  (2026-02-03T16:26:47Z)
• FarsiMCQGen: a Persian Multiple-choice Question Generation Framework [2.026379197206863]
  This paper introduces FarsiMCQGen, an innovative approach for generating Persian-language multiple-choice questions (MCQs)<n>Our methodology combines candidate generation, filtering, and ranking techniques to build a model that generates answer choices resembling those in real MCQs.<n>We leverage advanced methods, including Transformers and knowledge graphs, integrated with rule-based approaches to craft credible distractors that challenge test-takers.
  arXiv  Detail & Related papers  (2025-10-16T20:52:07Z)
• Q-Mirror: Unlocking the Multi-Modal Potential of Scientific Text-Only QA Pairs [60.0988889107102]
  We explore the potential for transforming Text-Only QA Pairs (TQAs) into high-quality Multi-Modal QA Pairs (MMQAs)<n>We develop a TQA-to-MMQA framework and establish a comprehensive, multi-dimensional MMQA quality that provides principles for the transformation.<n>We develop an agentic system (Q-Mirror) which operationalizes our framework by integrating MMQA generation and evaluation into a closed loop for iterative refinement.
  arXiv  Detail & Related papers  (2025-09-29T05:22:10Z)
• HeQ: a Large and Diverse Hebrew Reading Comprehension Benchmark [54.73504952691398]
  We set out to deliver a Hebrew Machine Reading dataset as extractive Questioning.<n>The morphologically rich nature of Hebrew poses a challenge to this endeavor.<n>We devise a novel set of guidelines, a controlled crowdsourcing protocol, and revised evaluation metrics.
  arXiv  Detail & Related papers  (2025-08-03T15:53:01Z)
• Grounding-IQA: Multimodal Language Grounding Model for Image Quality Assessment [69.07445098168344]
  We introduce a new image quality assessment (IQA) task paradigm, grounding-IQA.<n>Grounding-IQA comprises two subtasks: grounding-IQA-description (GIQA-DES) and visual question answering (GIQA-VQA)<n>To realize grounding-IQA, we construct a corresponding dataset, GIQA-160K, through our proposed automated annotation pipeline.<n>Experiments demonstrate that our proposed task paradigm, dataset, and benchmark facilitate the more fine-grained IQA application.
  arXiv  Detail & Related papers  (2024-11-26T09:03:16Z)
• Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph [83.90988015005934]
  Uncertainty quantification is a key element of machine learning applications.<n>We introduce a novel benchmark that implements a collection of state-of-the-art UQ baselines.<n>We conduct a large-scale empirical investigation of UQ and normalization techniques across eleven tasks, identifying the most effective approaches.
  arXiv  Detail & Related papers  (2024-06-21T20:06:31Z)
• Math Multiple Choice Question Generation via Human-Large Language Model Collaboration [5.081508251092439]
  Multiple choice questions (MCQs) are a popular method for evaluating students' knowledge. Recent advances in large language models (LLMs) have sparked interest in automating MCQ creation. This paper introduces a prototype tool designed to facilitate collaboration between LLMs and educators.
  arXiv  Detail & Related papers  (2024-05-01T20:53:13Z)
• Exploring Automated Distractor Generation for Math Multiple-choice Questions via Large Language Models [40.50115385623107]
  Multiple-choice questions (MCQs) are ubiquitous in almost all levels of education since they are easy to administer, grade, and reliable format in assessments and practices. One of the most important aspects of MCQs is the distractors, i.e., incorrect options that are designed to target common errors or misconceptions among real students. To date, the task of crafting high-quality distractors largely remains a labor and time-intensive process for teachers and learning content designers, which has limited scalability.
  arXiv  Detail & Related papers  (2024-04-02T17:31:58Z)
• PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models [72.57329554067195]
  ProxyQA is an innovative framework dedicated to assessing longtext generation. It comprises in-depth human-curated meta-questions spanning various domains, each accompanied by specific proxy-questions with pre-annotated answers. It assesses the generated content's quality through the evaluator's accuracy in addressing the proxy-questions.
  arXiv  Detail & Related papers  (2024-01-26T18:12:25Z)
• Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning [43.83422798569986]
  Multiple-choice questions (MCQs) are ubiquitous in almost all levels of education since they are easy to administer, grade, and reliable form of assessment. To date, the task of crafting high-quality distractors has largely remained a labor-intensive process for teachers and learning content designers. We propose a simple, in-context learning-based solution for automated distractor and corresponding feedback message generation.
  arXiv  Detail & Related papers  (2023-08-07T01:03:04Z)
• EMBRACE: Evaluation and Modifications for Boosting RACE [0.0]
  RACE is a dataset of English texts and corresponding multiple-choice questions (MCQs) RACE was constructed by Chinese teachers of English for human reading comprehension. This article provides a detailed analysis of the test set of RACE for high-school students.
  arXiv  Detail & Related papers  (2023-05-15T08:21:32Z)
• SkillQG: Learning to Generate Question for Reading Comprehension Assessment [54.48031346496593]
  We present a question generation framework with controllable comprehension types for assessing and improving machine reading comprehension models. We first frame the comprehension type of questions based on a hierarchical skill-based schema, then formulate $textttSkillQG$ as a skill-conditioned question generator. Empirical results demonstrate that $textttSkillQG$ outperforms baselines in terms of quality, relevance, and skill-controllability.
  arXiv  Detail & Related papers  (2023-05-08T14:40:48Z)
• Learning to Reuse Distractors to support Multiple Choice Question Generation in Education [19.408786425460498]
  This paper studies how a large existing set of manually created answers and distractors can be leveraged to help teachers in creating new multiple choice questions (MCQs) We built several data-driven models based on context-aware question and distractor representations, and compared them with static feature-based models. Both automatic and human evaluations indicate that context-aware models consistently outperform a static feature-based approach.
  arXiv  Detail & Related papers  (2022-10-25T12:48:56Z)
• Fantastic Questions and Where to Find Them: FairytaleQA -- An Authentic Dataset for Narrative Comprehension [136.82507046638784]
  We introduce FairytaleQA, a dataset focusing on narrative comprehension of kindergarten to eighth-grade students. FairytaleQA consists of 10,580 explicit and implicit questions derived from 278 children-friendly stories.
  arXiv  Detail & Related papers  (2022-03-26T00:20:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.