The Missing Evaluation Axis: What 10,000 Student Submissions Reveal About AI Tutor Effectiveness

• URL: http://arxiv.org/abs/2605.05648v1
• Date: Thu, 07 May 2026 03:58:04 GMT
• Title: The Missing Evaluation Axis: What 10,000 Student Submissions Reveal About AI Tutor Effectiveness
• Abstract summary: We argue that AI tutor evaluation should be extended with a behavioral dimension grounded in student interaction data.<n>We propose an evaluation framework and apply it to 10,235 code submissions with corresponding AI tutor feedback.<n>Using this framework to compare two deployed AI tutors across different semesters in a large-scale introductory computer science course reveals substantial differences in student engagement patterns.
• Score: 9.482202294953526
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current Artificial Intelligence (AI)-based tutoring systems (AI tutors) are primarily evaluated based on the pedagogical quality of their feedback messages. While important, pedagogy alone is insufficient because it ignores a critical question: what do students actually do with the feedback they receive? We argue that AI tutor evaluation should be extended with a behavioral dimension grounded in student interaction data, which complements pedagogical assessment. We propose an evaluation framework and apply it to 10,235 code submissions with corresponding AI tutor feedback from an introductory undergraduate programming course to measure whether students act on tutor feedback and whether those actions are applied correctly. Using this framework to compare two deployed AI tutors across different semesters in a large-scale introductory computer science course reveals substantial differences in student engagement patterns that are not captured by pedagogy-only evaluation. Moreover, these engagement-based behavioral signals are more strongly associated with student perception of helpful feedback than pedagogical quality alone, providing a more complete and actionable picture of AI tutor performance.

Related papers

• Who Should Grade My Work? Student Perspectives on Transparent AI-Assisted Writing Assessment in Higher Education [0.0]
  The integration of GenAI tools into higher education assessment raises important questions about how students understand, interpret, and respond to AI-mediated evaluation.<n>This study reports findings from a qualitative pedagogical inquiry conducted in an undergraduate technical communication course for computing students at a Saudi public university.<n>Students were explicitly informed that ChatGPT had generated the score and feedback and were invited to reflect on the evaluation in writing.
  arXiv  Detail & Related papers  (2026-09-04T16:49:33Z)
• UCO: A Multi-Turn Interactive Reinforcement Learning Method for Adaptive Teaching with Large Language Models [59.693733170193944]
  Large language models (LLMs) are shifting from answer providers to intelligent tutors in educational settings.<n>Recent reinforcement learning approaches address this limitation but face two critical challenges.<n>We propose the Unidirectional Cognitive Optimization (UCO) method to address these challenges.
  arXiv  Detail & Related papers  (2025-11-12T01:27:02Z)
• Human or AI? Comparing Design Thinking Assessments by Teaching Assistants and Bots [0.38233569758620045]
  This study investigates the reliability and perceived accuracy of AI-assisted assessment compared to TA-assisted assessment in evaluating student posters in design thinking education.<n>Results showed low statistical agreement between instructor and AI scores for empathy and pain points, with slightly higher alignment for visual communication.<n>The study underscores the need for hybrid assessment models that integrate computational efficiency with human insights.
  arXiv  Detail & Related papers  (2025-10-17T07:09:21Z)
• Bridging Gaps Between Student and Expert Evaluations of AI-Generated Programming Hints [21.254611931654132]
  We study mismatches in perceived hint quality from students' and experts' perspectives.<n>We propose and discuss preliminary results on potential methods to bridge these gaps.
  arXiv  Detail & Related papers  (2025-09-03T12:38:35Z)
• MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors [82.91830877219822]
  We present MathTutorBench, an open-source benchmark for holistic tutoring model evaluation.<n>MathTutorBench contains datasets and metrics that broadly cover tutor abilities as defined by learning sciences research in dialog-based teaching.<n>We evaluate a wide set of closed- and open-weight models and find that subject expertise, indicated by solving ability, does not immediately translate to good teaching.
  arXiv  Detail & Related papers  (2025-02-26T08:43:47Z)
• Representational Alignment Supports Effective Machine Teaching [81.19197059407121]
  GRADE is a new controlled experimental setting to study pedagogy and representational alignment.<n>We find that improved representational alignment with a student improves student learning outcomes.<n>However, this effect is moderated by the size and representational diversity of the class being taught.
  arXiv  Detail & Related papers  (2024-06-06T17:48:24Z)
• Evaluating and Optimizing Educational Content with Large Language Model Judgments [52.33701672559594]
  We use Language Models (LMs) as educational experts to assess the impact of various instructions on learning outcomes. We introduce an instruction optimization approach in which one LM generates instructional materials using the judgments of another LM as a reward function. Human teachers' evaluations of these LM-generated worksheets show a significant alignment between the LM judgments and human teacher preferences.
  arXiv  Detail & Related papers  (2024-03-05T09:09:15Z)
• Identifying Student Profiles Within Online Judge Systems Using Explainable Artificial Intelligence [6.638206014723678]
  Online Judge (OJ) systems are typically considered within programming-related courses as they yield fast and objective assessments of the code developed by the students. This work aims to tackle this limitation by considering the further exploitation of the information gathered by the OJ and automatically inferring feedback for both the student and the instructor.
  arXiv  Detail & Related papers  (2024-01-29T12:11:30Z)
• Revealing Networks: Understanding Effective Teacher Practices in AI-Supported Classrooms using Transmodal Ordered Network Analysis [0.9187505256430948]
  The present study uses transmodal ordered network analysis to understand effective teacher practices in relationship to traditional metrics of in-system learning in a mathematics classroom working with AI tutors. Comparing teacher practices by student learning rates, we find that students with low learning rates exhibited more hint use after monitoring. Students with low learning rates showed learning behavior similar to their high learning rate peers, achieving repeated correct attempts in the tutor.
  arXiv  Detail & Related papers  (2023-12-17T21:50:02Z)
• MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems [74.73881579517055]
  We propose a framework to generate such dialogues by pairing human teachers with a Large Language Model prompted to represent common student errors. We describe how we use this framework to collect MathDial, a dataset of 3k one-to-one teacher-student tutoring dialogues.
  arXiv  Detail & Related papers  (2023-05-23T21:44:56Z)
• Giving Feedback on Interactive Student Programs with Meta-Exploration [74.5597783609281]
  Developing interactive software, such as websites or games, is a particularly engaging way to learn computer science. Standard approaches require instructors to manually grade student-implemented interactive programs. Online platforms that serve millions, like Code.org, are unable to provide any feedback on assignments for implementing interactive programs.
  arXiv  Detail & Related papers  (2022-11-16T10:00:23Z)
• Assessment Modeling: Fundamental Pre-training Tasks for Interactive Educational Systems [3.269851859258154]
  A common way of circumventing label-scarce problems is pre-training a model to learn representations of the contents of learning items. We propose Assessment Modeling, a class of fundamental pre-training tasks for general interactive educational systems.
  arXiv  Detail & Related papers  (2020-01-01T02:00:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.