How Well Does AI-Generated Feedback Work? Intrinsic and Extrinsic Evaluation across more than 20,000 EFL Essay Drafts

• URL: http://arxiv.org/abs/2607.14591v1
• Date: Thu, 16 Jul 2026 05:38:53 GMT
• Title: How Well Does AI-Generated Feedback Work? Intrinsic and Extrinsic Evaluation across more than 20,000 EFL Essay Drafts
• Abstract summary: This study examines feedback in English as a Foreign Language (EFL) writing contexts, focusing on written corrective feedback (WCF)<n>Large language models (LLMs) can provide WCF at scale, but aligning them with pedagogical best practices remains an ongoing challenge.
• Score: 23.21912420340251
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This study examines feedback in English as a Foreign Language (EFL) writing contexts, focusing on written corrective feedback (WCF). Large language models (LLMs) can provide WCF at scale, but aligning them with pedagogical best practices remains an ongoing challenge. WCF meeting criteria like factuality or relevance may still be unsuitable for learning contexts, highlighting the need for extrinsic evaluation based on the learner's perspective. We deployed WCF systems in a university-level EFL class with nearly 2,000 students, collecting over 20,000 drafts. We evaluated the generated WCF from two perspectives: intrinsic evaluation by experienced English teachers using a rubric, and extrinsic evaluation via student feedback and engagement metrics. Results revealed low alignment between teacher expert ratings and student feedback. These findings suggest that traditional expert evaluation alone may not fully capture WCF's usability or helpfulness from the learner's perspective, highlighting the importance of learner-centered evaluation frameworks for AI-based applications in language education.

Related papers

• Learning in Blocks: A Multi Agent Debate Assisted Personalized Adaptive Learning Framework for Language Learning [0.0]
  We introduce Learning in Blocks, a framework that grounds progression in demonstrated conversational competence evaluated using CEFR-aligned rubrics.<n>We benchmark four scoring and recommendation methods on CEFR A2 conversations annotated by ESL experts.<n>An 8-week study with 180 CEFR A2 learners demonstrates that combining rubric-aligned scoring and recommendation with spaced review and mastery-based progression produces better learning outcomes than feedback alone.
  arXiv  Detail & Related papers  (2026-03-29T09:38:41Z)
• Beyond Accuracy: Towards a Robust Evaluation Methodology for AI Systems for Language Education [0.0]
  The rapid adoption of large language models in AI-powered language education has created an urgent need for evaluations that assess pedagogical effectiveness.<n>We introduce L2-Bench, a novel evaluation benchmark grounded in a validated "language learning experience designer" construct.<n>Our methodology integrates pedagogical theory, sociotechnical AI evaluation methods, and operationalizes a hierarchical taxonomy to structure an expert-curated dataset.
  arXiv  Detail & Related papers  (2026-03-20T16:13:03Z)
• Exposía: Academic Writing Assessment of Exposés and Peer Feedback [56.428320613219306]
  We present Exposa, the first public dataset that connects writing and feedback assessment in higher education.<n>We use Exposa to benchmark state-of-the-art open-source large language models (LLMs) for two tasks: automated scoring of (1) the proposals and (2) the student reviews.
  arXiv  Detail & Related papers  (2026-01-10T11:33:26Z)
• Disentangling Language and Culture for Evaluating Multilingual Large Language Models [48.06219053598005]
  This paper introduces a Dual Evaluation Framework to comprehensively assess the multilingual capabilities of LLMs.<n>By decomposing the evaluation along the dimensions of linguistic medium and cultural context, this framework enables a nuanced analysis of LLMs' ability to process questions cross-lingually.
  arXiv  Detail & Related papers  (2025-05-30T14:25:45Z)
• LLM-as-a-Judge & Reward Model: What They Can and Cannot Do [2.2469442203227863]
  We conduct a comprehensive analysis of automated evaluators, reporting several key findings on their behavior. We discover that English evaluation capabilities significantly influence language-specific evaluation capabilities, enabling evaluators trained in English to easily transfer their skills to other languages. We find that state-of-the-art evaluators struggle with challenging prompts, in either English or Korean, underscoring their limitations in assessing or generating complex reasoning questions.
  arXiv  Detail & Related papers  (2024-09-17T14:40:02Z)
• STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond [68.47402386668846]
  We introduce Structured Reasoning In Critical Text Assessment (STRICTA) to model text assessment as an explicit, step-wise reasoning process.<n>STRICTA breaks down the assessment into a graph of interconnected reasoning steps drawing on causality theory.<n>We apply STRICTA to a dataset of over 4000 reasoning steps from roughly 40 biomedical experts on more than 20 papers.
  arXiv  Detail & Related papers  (2024-09-09T06:55:37Z)
• Pragmatic Competence Evaluation of Large Language Models for the Korean Language [0.6757476692230009]
  This study evaluates how well Large Language Models (LLMs) understand context-dependent expressions from a pragmatic standpoint, specifically in Korean. We use both Multiple-Choice Questions (MCQs) for automatic evaluation and Open-Ended Questions (OEQs) assessed by human experts.
  arXiv  Detail & Related papers  (2024-03-19T12:21:20Z)
• F-Eval: Assessing Fundamental Abilities with Refined Evaluation Methods [102.98899881389211]
  We propose F-Eval, a bilingual evaluation benchmark to evaluate the fundamental abilities, including expression, commonsense and logic. For reference-free subjective tasks, we devise new evaluation methods, serving as alternatives to scoring by API models.
  arXiv  Detail & Related papers  (2024-01-26T13:55:32Z)
• FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets [69.91340332545094]
  We introduce FLASK, a fine-grained evaluation protocol for both human-based and model-based evaluation. We experimentally observe that the fine-graininess of evaluation is crucial for attaining a holistic view of model performance.
  arXiv  Detail & Related papers  (2023-07-20T14:56:35Z)
• Multi-Dimensional Evaluation of Text Summarization with In-Context Learning [79.02280189976562]
  In this paper, we study the efficacy of large language models as multi-dimensional evaluators using in-context learning. Our experiments show that in-context learning-based evaluators are competitive with learned evaluation frameworks for the task of text summarization. We then analyze the effects of factors such as the selection and number of in-context examples on performance.
  arXiv  Detail & Related papers  (2023-06-01T23:27:49Z)
• CEFR-Based Sentence Difficulty Annotation and Assessment [25.71796445061561]
  The CEFR-based Sentence Profile (CEFR-SP) corpus contains 17k English sentences annotated with the levels based on the Common European Framework of Reference for Languages. In the experiments in this study, our method achieved a macro-F1 score of 84.5% in the level assessment, thus outperforming strong baselines employed in readability assessment.
  arXiv  Detail & Related papers  (2022-10-21T07:03:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.