A Large-Scale Real-World Evaluation of LLM-Based Virtual Teaching Assistant

• URL: http://arxiv.org/abs/2506.17363v1
• Date: Fri, 20 Jun 2025 10:59:57 GMT
• Title: A Large-Scale Real-World Evaluation of LLM-Based Virtual Teaching Assistant
• Abstract summary: Virtual Teaching Assistants (VTAs) have the potential to enhance student learning by providing instant feedback and facilitating multi-turn interactions.<n>This study develops an LLM-based VTA and deploys it in an AI programming course with 477 graduate students.<n>We assess the feasibility of deploying VTAs in real-world classrooms and identify key challenges for broader adoption.
• Score: 19.026750427901423
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Virtual Teaching Assistants (VTAs) powered by Large Language Models (LLMs) have the potential to enhance student learning by providing instant feedback and facilitating multi-turn interactions. However, empirical studies on their effectiveness and acceptance in real-world classrooms are limited, leaving their practical impact uncertain. In this study, we develop an LLM-based VTA and deploy it in an introductory AI programming course with 477 graduate students. To assess how student perceptions of the VTA's performance evolve over time, we conduct three rounds of comprehensive surveys at different stages of the course. Additionally, we analyze 3,869 student--VTA interaction pairs to identify common question types and engagement patterns. We then compare these interactions with traditional student--human instructor interactions to evaluate the VTA's role in the learning process. Through a large-scale empirical study and interaction analysis, we assess the feasibility of deploying VTAs in real-world classrooms and identify key challenges for broader adoption. Finally, we release the source code of our VTA system, fostering future advancements in AI-driven education: \texttt{https://github.com/sean0042/VTA}.

Related papers

• UCO: A Multi-Turn Interactive Reinforcement Learning Method for Adaptive Teaching with Large Language Models [59.693733170193944]
  Large language models (LLMs) are shifting from answer providers to intelligent tutors in educational settings.<n>Recent reinforcement learning approaches address this limitation but face two critical challenges.<n>We propose the Unidirectional Cognitive Optimization (UCO) method to address these challenges.
  arXiv  Detail & Related papers  (2025-11-12T01:27:02Z)
• Bringing Pedagogy into Focus: Evaluating Virtual Teaching Assistants' Question-Answering in Asynchronous Learning Environments [17.371971971294503]
  Asynchronous learning environments (ALEs) are widely adopted for formal and informal learning, but timely and personalized support is often limited.<n>Existing assessments often rely on surface-level metrics and lack sufficient grounding in educational theories.<n>We propose an evaluation framework rooted in learning sciences and tailored to asynchronous forum discussions.
  arXiv  Detail & Related papers  (2025-09-22T16:15:58Z)
• AI-driven formative assessment and adaptive learning in data-science education: Evaluating an LLM-powered virtual teaching assistant [6.874351093155318]
  VITA (Virtual Teaching Assistants) is an adaptive distributed learning platform that embeds a large language model (LLM)-powered bot (BotCaptain)<n>The paper describes an end-to-end data pipeline that transforms chat logs into Experience API (xAPI) statements, instructor dashboards that surface outliers for just-in-time intervention.<n>Future work will refine the platform's adaptive intelligence and examine applicability across varied educational settings.
  arXiv  Detail & Related papers  (2025-09-17T11:27:45Z)
• Improving Student-AI Interaction Through Pedagogical Prompting: An Example in Computer Science Education [1.1517315048749441]
  Large language model (LLM) applications have sparked both excitement and concern.<n>Recent studies consistently highlight students' (mis)use of LLMs can hinder learning outcomes.<n>This work aims to teach students how to effectively prompt LLMs to improve their learning.
  arXiv  Detail & Related papers  (2025-06-23T20:39:17Z)
• Unveiling the Learning Mind of Language Models: A Cognitive Framework and Empirical Study [50.065744358362345]
  Large language models (LLMs) have shown impressive capabilities across tasks such as mathematics, coding, and reasoning.<n>Yet their learning ability, which is crucial for adapting to dynamic environments and acquiring new knowledge, remains underexplored.
  arXiv  Detail & Related papers  (2025-06-16T13:24:50Z)
• An Empirical Study of Federated Prompt Learning for Vision Language Model [50.73746120012352]
  This paper systematically investigates behavioral differences between language prompt learning and vision prompt learning.<n>We conduct experiments to evaluate the impact of various fl and prompt configurations, such as client scale, aggregation strategies, and prompt length.<n>We explore strategies for enhancing prompt learning in complex scenarios where label skew and domain shift coexist.
  arXiv  Detail & Related papers  (2025-05-29T03:09:15Z)
• Examining the Role of LLM-Driven Interactions on Attention and Cognitive Engagement in Virtual Classrooms [9.241265477406078]
  We investigate how peer question-asking behaviors influenced student engagement, attention, cognitive load, and learning outcomes.<n>Our results suggest that peer questions did not introduce extraneous cognitive load directly, as the cognitive load is strongly correlated with increased attention to the learning material.
  arXiv  Detail & Related papers  (2025-05-12T09:21:19Z)
• The StudyChat Dataset: Student Dialogues With ChatGPT in an Artificial Intelligence Course [1.9116784879310025]
  Large language models (LLMs) have significantly impacted education, raising both opportunities and challenges.<n>We introduce StudyChat, a dataset capturing real-world student interactions with an LLM.<n>We deploy a web application that replicates ChatGPTs core functionalities, and use it to log student interactions with the LLM.<n>We analyze these interactions, highlight usage trends, and analyze how specific student behavior correlates with their course outcome.
  arXiv  Detail & Related papers  (2025-03-11T00:17:07Z)
• LLMs as Educational Analysts: Transforming Multimodal Data Traces into Actionable Reading Assessment Reports [6.523137821124204]
  This study investigates the use of multimodal data sources to derive meaningful reading insights.<n>We employ unsupervised learning techniques to identify distinct reading behavior patterns.<n>A large language model (LLM) synthesizes the derived information into actionable reports for educators.
  arXiv  Detail & Related papers  (2025-03-03T22:34:08Z)
• Student-AI Interaction in an LLM-Empowered Learning Environment: A Cluster Analysis of Engagement Profiles [28.794946431719392]
  This study explored diverse learner profiles within a multi-agent, LLM-empowered learning environment.<n>Students exhibit varied behavioral, cognitive, and emotional engagement tendencies.
  arXiv  Detail & Related papers  (2025-03-03T16:08:28Z)
• INTERACT: Enabling Interactive, Question-Driven Learning in Large Language Models [15.825663946923289]
  Large language models (LLMs) excel at answering questions but remain passive learners-absorbing static data without the ability to question and refine knowledge.<n>This paper explores how LLMs can transition to interactive, question-driven learning through student-teacher dialogues.
  arXiv  Detail & Related papers  (2024-12-16T02:28:53Z)
• When LLMs Learn to be Students: The SOEI Framework for Modeling and Evaluating Virtual Student Agents in Educational Interaction [12.070907646464537]
  We propose the SOEI framework for constructing and evaluating personality-aligned Virtual Student Agents (LVSAs) in classroom scenarios.<n>We generate five LVSAs based on Big Five traits through LoRA fine-tuning and expert-informed prompt design.<n>Our results provide: (1) an educationally and psychologically grounded generation pipeline for LLM-based student agents; (2) a hybrid, scalable evaluation framework for behavioral realism; and (3) empirical insights into the pedagogical utility of LVSAs in shaping instructional adaptation.
  arXiv  Detail & Related papers  (2024-10-21T07:18:24Z)
• Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs [49.18567856499736]
  We investigate whether large language models (LLMs) can be supportive of open-ended dialogue tutoring.<n>We apply a range of knowledge tracing (KT) methods on the resulting labeled data to track student knowledge levels over an entire dialogue.<n>We conduct experiments on two tutoring dialogue datasets, and show that a novel yet simple LLM-based method, LLMKT, significantly outperforms existing KT methods in predicting student response correctness in dialogues.
  arXiv  Detail & Related papers  (2024-09-24T22:31:39Z)
• YODA: Teacher-Student Progressive Learning for Language Models [82.0172215948963]
  This paper introduces YODA, a teacher-student progressive learning framework. It emulates the teacher-student education process to improve the efficacy of model fine-tuning. Experiments show that training LLaMA2 with data from YODA improves SFT with significant performance gain.
  arXiv  Detail & Related papers  (2024-01-28T14:32:15Z)
• UKP-SQuARE: An Interactive Tool for Teaching Question Answering [61.93372227117229]
  The exponential growth of question answering (QA) has made it an indispensable topic in any Natural Language Processing (NLP) course. We introduce UKP-SQuARE as a platform for QA education. Students can run, compare, and analyze various QA models from different perspectives.
  arXiv  Detail & Related papers  (2023-05-31T11:29:04Z)
• Single-Modal Entropy based Active Learning for Visual Question Answering [75.1682163844354]
  We address Active Learning in the multi-modal setting of Visual Question Answering (VQA) In light of the multi-modal inputs, image and question, we propose a novel method for effective sample acquisition. Our novel idea is simple to implement, cost-efficient, and readily adaptable to other multi-modal tasks.
  arXiv  Detail & Related papers  (2021-10-21T05:38:45Z)
• Explainable Active Learning (XAL): An Empirical Study of How Local Explanations Impact Annotator Experience [76.9910678786031]
  We propose a novel paradigm of explainable active learning (XAL), by introducing techniques from the recently surging field of explainable AI (XAI) into an Active Learning setting. Our study shows benefits of AI explanation as interfaces for machine teaching--supporting trust calibration and enabling rich forms of teaching feedback, and potential drawbacks--anchoring effect with the model judgment and cognitive workload.
  arXiv  Detail & Related papers  (2020-01-24T22:52:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.