Simulating Disengaged Students to Evaluate LLM-based Tutors

• URL: http://arxiv.org/abs/2609.12331v1
• Date: Fri, 11 Sep 2026 01:28:48 GMT
• Title: Simulating Disengaged Students to Evaluate LLM-based Tutors
• Abstract summary: Disengagement-Aware Student Simulators (DAS2) models five learner-engagement states: engaged, gaming, wheel-spinning, off-task, and mixed.<n>DAS2 provides a pre-deployment framework for evaluating how AI tutors respond to diverse learner-engagement states before deployment.
• Score: 0.40105987447353786
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Simulated students generated by computational models provide a practical way to evaluate tutoring strategies and pedagogical approaches used by human and AI tutors. However, such simulations should account for disengaged behaviors, including gaming the system, wheel-spinning, and off-task behavior, because tutors may need different responses for different learner states. We present Disengagement-Aware Student Simulators (DAS2), a reproducible pre-deployment protocol that models five learner-engagement states: engaged, gaming, wheel-spinning, off-task, and mixed, and evaluates AI tutor performance across these states. Using ASSISTments09, two coders independently labeled 100 sampled tutoring sessions based on anonymized interaction-log summaries. They achieved 84% agreement (Cohen's kappa = 0.78), and among agreed cases, human consensus labels matched DAS2 rule-based labels in 81% of cases (kappa = 0.75). Conditioning simulations on intended learner states reduced the correctness-rate gap between simulated and authentic sessions from 0.54 to 0.20 for gaming and from 0.51 to 0.18 for wheel-spinning. Fine-tuned Qwen2.5-7B better matched authentic response-time distributions, while prompt-only GPT-4o generated more distinguishable learner states. Evaluation of five AI tutors from the Claude, Llama, Gemini, Qwen, and GPT families showed that relative rankings remained stable across learner states and interaction lengths, while absolute performance varied, revealing state-specific differences in tutor support. Human validation further showed that automated tutor evaluation does not fully align with human judgment. DAS2 provides a pre-deployment framework for evaluating how AI tutors respond to diverse learner-engagement states before deployment.

Related papers

• StudentSim: Training LLM-based Student Simulators [67.06331010637008]
  StudentSim is a training framework that turns sparse per-student data into individualized simulators.<n>The resulting simulators both mirror a student's own responses and update them under tutor guidance.<n>Across all three domains, StudentSim outperforms GPT-5.4 on both metrics.
  arXiv  Detail & Related papers  (2026-09-01T17:55:10Z)
• Which CS1 Students Will Fail? Identifying Digital Markers from Learning Analytics in Computer Systems and Architecture Using Weighted Academic Momentum and Interaction Logs [0.0]
  Digital learning platforms generate rich behavioural traces (digital markers) that offer the potential to identify struggling students early.<n>This paper investigates whether a combination of traditional and digital markers can predict failure in a first-year CS1 course.
  arXiv  Detail & Related papers  (2026-07-16T08:32:54Z)
• AI-Driven Assessment of Human Tutors: Linking Training Performance to Real-Life Practice [1.3597553728687461]
  We present an AI-driven system that assesses both open responses during training and authentic real-life tutoring.<n>We use Generative AI (Gemini-2.5-pro) to analyze transcriptions of authentic tutoring, measuring the transfer of tutor skills to real-life application.
  arXiv  Detail & Related papers  (2026-06-17T02:28:10Z)
• Faster Completion, Less Learning: Generative AI Reduced Study Time on Math Problems and the Knowledge They Build [0.6524460254566904]
  Self-report surveys show little change, while small-scale behavioral studies report widespread AI use without the scale or duration to measure learning consequences.<n>We use a ten-year panel of $3.2$ million ALEKS learning interactions for the time-on-task analysis.<n>These results are among the first large-scale behavioral and outcome evidence that generative AI has altered how students study and the knowledge they build.
  arXiv  Detail & Related papers  (2026-05-20T18:40:53Z)
• Reinforcing Human Behavior Simulation via Verbal Feedback [105.68319269895653]
  We present DITTO, a model trained by treating verbal feedback as a first-class signal in reinforcement learning.<n>We also introduce SOUL, a unified benchmark and training data suite spanning 10 tasks across six categories: Theory of Mind, character role play, social skill, learner simulation, user simulation, and persona simulation.<n>DitTO achieves an average 36% improvement over the base model and exceeds GPT-5.4 on 6 of 10 SOUL benchmarks.
  arXiv  Detail & Related papers  (2026-05-19T21:23:24Z)
• Simulating Students or Sycophantic Problem Solving? On Misconception Faithfulness of LLM Simulators [55.617099475539305]
  Large language models (LLMs) can fluently generate student-like responses, making them attractive as simulated students for training and evaluating AI tutors and human educators.<n>Yet such simulators are typically evaluated by output similarity to real students, not by whether they behave like students with coherent misconceptions during interaction.<n>We introduce a controlled framework for evaluating misconception faithfulness, whether a simulator maintains a misconception-driven belief state and updates selectively when feedback addresses the underlying misconception.
  arXiv  Detail & Related papers  (2026-05-12T20:55:23Z)
• Personalized AI Practice Replicates Learning Rate Regularity at Scale [0.0]
  Students exhibit consistent learning rates across diverse educational contexts.<n>We test these findings using a dataset of 1.8 million (366k post-filtering) student interactions from the digital platform Campus AI.
  arXiv  Detail & Related papers  (2026-03-09T13:33:28Z)
• UCO: A Multi-Turn Interactive Reinforcement Learning Method for Adaptive Teaching with Large Language Models [59.693733170193944]
  Large language models (LLMs) are shifting from answer providers to intelligent tutors in educational settings.<n>Recent reinforcement learning approaches address this limitation but face two critical challenges.<n>We propose the Unidirectional Cognitive Optimization (UCO) method to address these challenges.
  arXiv  Detail & Related papers  (2025-11-12T01:27:02Z)
• Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning [52.07170679746533]
  Large Language Models (LLMs) are increasingly used to simulate human users in interactive settings such as therapy, education, and social role-play.<n>We introduce a unified framework for evaluating and improving persona consistency in LLM-generated dialogue.<n>We define three automatic metrics: prompt-to-line consistency, line-to-line consistency, and Q&A consistency, that capture different types of persona drift and validate each against human annotations.
  arXiv  Detail & Related papers  (2025-10-31T19:40:41Z)
• SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants? [61.07963107032645]
  Large language models (LLMs) are increasingly used in interactive applications.<n>Human evaluation remains the gold standard for assessing their performance in multi-turn conversations.<n>We introduce SimulatorArena, a benchmark of 909 annotated human-LLM conversations on two interactive tasks.
  arXiv  Detail & Related papers  (2025-10-06T23:17:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.