Your Students Don't Use LLMs Like You Wish They Did

• URL: http://arxiv.org/abs/2604.23486v1
• Date: Sun, 26 Apr 2026 01:23:21 GMT
• Title: Your Students Don't Use LLMs Like You Wish They Did
• Abstract summary: We introduce six computational metrics for automated evaluation of pedagogical alignment in student-AI dialogue.<n>We validate our metrics through analysis of 12,650 messages across 500 conversations from four courses.<n>Our metrics will enable researchers building educational dialogue systems to measure whether they are achieving their pedagogical goals.
• Score: 8.643506838321354
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Educational NLP systems are typically evaluated using engagement metrics and satisfaction surveys, which are at best a proxy for meeting pedagogical goals. We introduce six computational metrics for automated evaluation of pedagogical alignment in student-AI dialogue. We validate our metrics through analysis of 12,650 messages across 500 conversations from four courses. Using our metrics, we identify a fundamental misalignment: educators design conversational tutors for sustained learning dialogue, but students mainly use them for answer-extraction. Deployment context is the strongest predictor of usage patterns, outweighing student preference or system design: when AI tools are optional, usage concentrates around deadlines; when integrated into course structure, students ask for solutions to verbatim assignment questions. Whole-dialogue evaluation misses these turn-by-turn patterns. Our metrics will enable researchers building educational dialogue systems to measure whether they are achieving their pedagogical goals.

Related papers

• PredAct-Bench: Benchmarking Tool-Augmented Dialogue under Controlled Tool Noise [11.530655589494868]
  We introduce PREDACTBENCH, a benchmark for evaluating dialogue agents paired with statistically imperfect tools.<n>We build a benchmark for AI-assisted human decision-making, where the AI uses noisy predictors to help guide a user.<n>We evaluate 13 state-of-the-art closed and open source LLMs on two educational datasets.
  arXiv  Detail & Related papers  (2026-08-03T15:15:55Z)
• "How Do I ...?": Procedural Questions Predominate Student-LLM Chatbot Conversations [39.146761527401424]
  This paper focuses on such student questions from two datasets of distinct learning contexts: formative self-study, and summative assessed coursework.<n>We analysed 6,113 messages from both learning contexts using 11 different Large Language Models (LLM) and three human raters.<n>Results show that 'procedural' questions predominated in both learning contexts, but more so when students prepare for summative assessment.
  arXiv  Detail & Related papers  (2026-02-20T17:27:41Z)
• From Words to Wisdom: Discourse Annotation and Baseline Models for Student Dialogue Understanding [5.459797813771498]
  This work introduces an annotated educational dialogue dataset of student conversations featuring knowledge construction and task production discourse.<n>We also establish baseline models for automatically predicting these discourse properties for each turn of talk within conversations, using pre-trained large language models GPT-3.5 and Llama-3.1.<n> Experimental results indicate that these state-of-the-art models perform suboptimally on this task, indicating the potential for future research.
  arXiv  Detail & Related papers  (2025-11-25T17:46:00Z)
• Supporting Students' Reading and Cognition with AI [12.029238454394445]
  We analyzed text from 124 sessions with AI tools to understand users' reading processes and cognitive engagement.<n>We propose design implications for future AI reading-support systems, including structured scaffolds for lower-level cognitive tasks.<n>We advocate for adaptive, human-in-the-loop features that allow students and instructors to tailor their reading experiences with AI.
  arXiv  Detail & Related papers  (2025-04-07T17:51:27Z)
• Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs [49.18567856499736]
  We investigate whether large language models (LLMs) can be supportive of open-ended dialogue tutoring.<n>We apply a range of knowledge tracing (KT) methods on the resulting labeled data to track student knowledge levels over an entire dialogue.<n>We conduct experiments on two tutoring dialogue datasets, and show that a novel yet simple LLM-based method, LLMKT, significantly outperforms existing KT methods in predicting student response correctness in dialogues.
  arXiv  Detail & Related papers  (2024-09-24T22:31:39Z)
• ChEDDAR: Student-ChatGPT Dialogue in EFL Writing Education [14.524728335166703]
  The integration of generative AI in education is expanding, yet empirical analyses of large-scale, real-world interactions between students and AI systems remain limited. This study collects a semester-long longitudinal experiment involving 212 college students enrolled in English as Foreign Langauge (EFL) writing courses. ChEDDAR includes a conversation log, utterance-level essay edit history, self-rated satisfaction, and students' intent, in addition to session-level pre-and-post surveys documenting their objectives and overall experiences.
  arXiv  Detail & Related papers  (2023-09-23T03:28:25Z)
• Cue-CoT: Chain-of-thought Prompting for Responding to In-depth Dialogue Questions with LLMs [59.74002011562726]
  We propose a novel linguistic cue-based chain-of-thoughts (textitCue-CoT) to provide a more personalized and engaging response. We build a benchmark with in-depth dialogue questions, consisting of 6 datasets in both Chinese and English. Empirical results demonstrate our proposed textitCue-CoT method outperforms standard prompting methods in terms of both textithelpfulness and textitacceptability on all datasets.
  arXiv  Detail & Related papers  (2023-05-19T16:27:43Z)
• CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning [85.3987745097806]
  offline reinforcement learning can be used to train dialogue agents entirely using static datasets collected from human speakers. Experiments show that recently developed offline RL methods can be combined with language models to yield realistic dialogue agents.
  arXiv  Detail & Related papers  (2022-04-18T17:43:21Z)
• What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation [73.03318027164605]
  We propose to use information that can be automatically extracted from the next user utterance as a proxy to measure the quality of the previous system response. Our model generalizes across both spoken and written open-domain dialog corpora collected from real and paid users.
  arXiv  Detail & Related papers  (2022-03-25T22:09:52Z)
• Few-Shot Bot: Prompt-Based Learning for Dialogue Systems [58.27337673451943]
  Learning to converse using only a few examples is a great challenge in conversational AI. The current best conversational models are either good chit-chatters (e.g., BlenderBot) or goal-oriented systems (e.g., MinTL) We propose prompt-based few-shot learning which does not require gradient-based fine-tuning but instead uses a few examples as the only source of learning.
  arXiv  Detail & Related papers  (2021-10-15T14:36:45Z)
• Learning an Unreferenced Metric for Online Dialogue Evaluation [53.38078951628143]
  We propose an unreferenced automated evaluation metric that uses large pre-trained language models to extract latent representations of utterances. We show that our model achieves higher correlation with human annotations in an online setting, while not requiring true responses for comparison during inference.
  arXiv  Detail & Related papers  (2020-05-01T20:01:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.