Consistently Good vs. Occasionally Great: A Rubric for Open-Ended Feedback Quality from Humans and Machines

• URL: http://arxiv.org/abs/2608.21850v1
• Date: Sat, 22 Aug 2026 08:48:53 GMT
• Title: Consistently Good vs. Occasionally Great: A Rubric for Open-Ended Feedback Quality from Humans and Machines
• Abstract summary: We focus on feedback for open-ended short answer questions in introductory programming.<n>We develop a five-criteria rubric grounded in educational literature for evaluating feedback quality.
• Score: 3.4020399898813145
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Providing high-quality feedback on student work is essential for learning, yet delivering such feedback at scale remains challenging. In this paper, we focus on feedback for open-ended short answer questions in introductory programming, with the goal of nudging students toward success on reattempts without revealing the correct answer. We develop a five-criteria rubric grounded in educational literature for evaluating feedback quality: (1) acknowledging correct portions of the student answer, (2) identifying at least one flaw (if present), (3) providing actionable guidance for improvement, (4) maintaining appropriate concealment of the answer, and (5) using an appropriate conversational tone. Using this rubric, we compare feedback generated by a frontier LLM (OpenAI o1) to feedback from nine teaching assistants across 90 student responses, with three researchers and an LLM independently scoring all feedback. Our results show that while one TA often produced the best feedback, the LLM demonstrated consistently higher average performance than TAs, as evaluated by humans. However, we also uncover significant self-preference bias when using LLMs to evaluate feedback quality: the LLM systematically rated its own outputs higher than human experts did. This bias, which research suggests persists even in cross-model evaluation, raises important methodological concerns for researchers employing LLM-based evaluation. We provide detailed characterization of both TA and LLM performance, analyze sources of variance in TA feedback quality, and discuss implications for deploying LLM-generated feedback in educational settings.

Related papers

• LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks [89.13372484640392]
  We propose Experiential Learning (EL), which repurposes the feedback model from an LLM-as-a-Judge into an LLM-as-a-Coach.<n>EL consistently outperforms rubric-based RL on held-out and unseen open-ended tasks.<n>These findings establish experiential knowledge as a richer and more generalizable learning signal for post-training on non-verifiable tasks.
  arXiv  Detail & Related papers  (2026-07-20T16:08:49Z)
• A Systematic Study of Pseudo-Relevance Feedback with LLMs [49.53124785319461]
  Pseudo-relevance feedback (PRF) methods built on large language models (LLMs) can be organized along two key design dimensions.<n>We study how the choice of feedback source and feedback model impact PRF effectiveness through controlled experimentation.
  arXiv  Detail & Related papers  (2026-03-11T17:31:50Z)
• FeedEval: Pedagogically Aligned Evaluation of LLM-Generated Essay Feedback [6.88204255655161]
  We propose FeedEval, a framework for evaluating LLM-generated essay feedback along three pedagogically grounded dimensions: specificity, helpfulness, and validity.<n>Experiments on the ASAP++ benchmark show that FeedEval closely aligns with human expert judgments and that essay scoring models trained with FeedEval-filtered high-quality feedback achieve superior scoring performance.
  arXiv  Detail & Related papers  (2026-01-08T04:04:29Z)
• Dean of LLM Tutors: Exploring Comprehensive and Automated Evaluation of LLM-generated Educational Feedback via LLM Feedback Evaluators [5.838566576554449]
  We propose a method that uses LLM feedback evaluators to automatically and comprehensively evaluate feedback generated by LLM tutors.<n>This allows low-quality feedback to be rejected and enables LLM tutors to improve the feedback they generated based on the evaluation results.<n>Our findings show that o3-pro demonstrated the best performance in zero-shot labelling of feedback while o4-mini demonstrated the best performance in few-shot labelling of feedback.
  arXiv  Detail & Related papers  (2025-08-08T02:36:23Z)
• DeepCritic: Deliberate Critique with Large Language Models [77.5516314477878]
  We focus on studying and enhancing the math critique ability of Large Language Models (LLMs)<n>Our developed critique model built on Qwen2.5-7B-Instruct significantly outperforms existing LLM critics on various error identification benchmarks.
  arXiv  Detail & Related papers  (2025-05-01T17:03:17Z)
• Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation [57.380464382910375]
  We show that the choice of feedback protocol for evaluation can significantly affect evaluation reliability and induce systematic biases.<n>We find that generator models can flip preferences by embedding distractor features.<n>We offer recommendations for choosing feedback protocols based on dataset characteristics and evaluation objectives.
  arXiv  Detail & Related papers  (2025-04-20T19:05:59Z)
• Zero-Shot LLMs in Human-in-the-Loop RL: Replacing Human Feedback for Reward Shaping [2.427844597259453]
  Reinforcement learning (RL) often struggles with reward misalignment.<n>Human-in-the-loop (HITL) methods can mitigate this issue, but they also introduce biases.<n>We propose two key contributions to address these challenges.
  arXiv  Detail & Related papers  (2025-03-26T03:17:12Z)
• Video SimpleQA: Towards Factuality Evaluation in Large Video Language Models [77.96693360763925]
  Video SimpleQA is the first comprehensive benchmark tailored for factuality evaluation in video contexts.<n>Our work differs from existing video benchmarks through the following key features: Knowledge required: demanding integration of external knowledge beyond the video's explicit narrative.<n>Short-form definitive answer: Answers are crafted as unambiguous and definitively correct in a short format with minimal scoring variance.
  arXiv  Detail & Related papers  (2025-03-24T17:46:09Z)
• Towards Adaptive Feedback with AI: Comparing the Feedback Quality of LLMs and Teachers on Experimentation Protocols [8.71931996488953]
  This study evaluates and compares the feedback quality of large language models (LLMs) with that of human teachers and science education experts.<n>Our results indicate that LLM-generated feedback shows no significant difference to that of teachers and experts in overall quality.
  arXiv  Detail & Related papers  (2025-02-18T13:22:14Z)
• Automated Feedback in Math Education: A Comparative Analysis of LLMs for Open-Ended Responses [0.0]
  This study aims to explore the potential of Large Language Models (LLMs) in facilitating automated feedback in math education. We employ Mistral, a version of Llama catered to math, and fine-tune this model for evaluating student responses by leveraging a dataset of student responses and teacher-written feedback for middle-school math problems. We evaluate the model's performance in scoring accuracy and the quality of feedback by utilizing judgments from 2 teachers.
  arXiv  Detail & Related papers  (2024-10-29T16:57:45Z)
• Large Language Model as an Assignment Evaluator: Insights, Feedback, and Challenges in a 1000+ Student Course [49.296957552006226]
  Using large language models (LLMs) for automatic evaluation has become an important evaluation method in NLP research. This report shares how we use GPT-4 as an automatic assignment evaluator in a university course with 1,028 students.
  arXiv  Detail & Related papers  (2024-07-07T00:17:24Z)
• Improving the Validity of Automatically Generated Feedback via Reinforcement Learning [46.667783153759636]
  We propose a framework for feedback generation that optimize both correctness and alignment using reinforcement learning (RL)<n>Specifically, we use GPT-4's annotations to create preferences over feedback pairs in an augmented dataset for training via direct preference optimization (DPO)
  arXiv  Detail & Related papers  (2024-03-02T20:25:50Z)
• Can Large Language Models Be an Alternative to Human Evaluations? [80.81532239566992]
  Large language models (LLMs) have demonstrated exceptional performance on unseen tasks when only the task instructions are provided. We show that the result of LLM evaluation is consistent with the results obtained by expert human evaluation.
  arXiv  Detail & Related papers  (2023-05-03T07:28:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.