Investigating Student Interaction Patterns with Large Language Model-Powered Course Assistants in Computer Science Courses

• URL: http://arxiv.org/abs/2509.08862v1
• Date: Wed, 10 Sep 2025 02:21:11 GMT
• Title: Investigating Student Interaction Patterns with Large Language Model-Powered Course Assistants in Computer Science Courses
• Authors: Chang Liu, Loc Hoang, Andrew Stolman, Rene F. Kizilcec, Bo Wu,
• Abstract summary: Large language models (LLMs) are promising for bridging this gap, but interactions between students and LLMs are rarely overseen by educators.<n>We developed and studied an LLM-powered course assistant deployed across multiple computer science courses to characterize real-world use and understand pedagogical implications.
• Score: 4.761218834684297
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Providing students with flexible and timely academic support is a challenge at most colleges and universities, leaving many students without help outside scheduled hours. Large language models (LLMs) are promising for bridging this gap, but interactions between students and LLMs are rarely overseen by educators. We developed and studied an LLM-powered course assistant deployed across multiple computer science courses to characterize real-world use and understand pedagogical implications. By Spring 2024, our system had been deployed to approximately 2,000 students across six courses at three institutions. Analysis of the interaction data shows that usage remains strong in the evenings and nights and is higher in introductory courses, indicating that our system helps address temporal support gaps and novice learner needs. We sampled 200 conversations per course for manual annotation: most sampled responses were judged correct and helpful, with a small share unhelpful or erroneous; few responses included dedicated examples. We also examined an inquiry-based learning strategy: only around 11% of sampled conversations contained LLM-generated follow-up questions, which were often ignored by students in advanced courses. A Bloom's taxonomy analysis reveals that current LLM capabilities are limited in generating higher-order cognitive questions. These patterns suggest opportunities for pedagogically oriented LLM-based educational systems and greater educator involvement in configuring prompts, content, and policies.

Related papers

• "How Do I ...?": Procedural Questions Predominate Student-LLM Chatbot Conversations [39.146761527401424]
  This paper focuses on such student questions from two datasets of distinct learning contexts: formative self-study, and summative assessed coursework.<n>We analysed 6,113 messages from both learning contexts using 11 different Large Language Models (LLM) and three human raters.<n>Results show that 'procedural' questions predominated in both learning contexts, but more so when students prepare for summative assessment.
  arXiv  Detail & Related papers  (2026-02-20T17:27:41Z)
• Prompting the Professoriate: A Qualitative Study of Instructor Perspectives on LLMs in Data Science Education [0.0]
  Large Language Models (LLMs) have shifted in just a few years from novelty to ubiquity, raising fundamental questions for data science education.<n>To understand how instructors are adapting, we conducted semi-structured interviews with 42 instructors from 33 institutions in 10 countries.
  arXiv  Detail & Related papers  (2025-09-14T17:32:45Z)
• The StudyChat Dataset: Student Dialogues With ChatGPT in an Artificial Intelligence Course [1.9116784879310025]
  Large language models (LLMs) have significantly impacted education, raising both opportunities and challenges.<n>We introduce StudyChat, a dataset capturing real-world student interactions with an LLM.<n>We deploy a web application that replicates ChatGPTs core functionalities, and use it to log student interactions with the LLM.<n>We analyze these interactions, highlight usage trends, and analyze how specific student behavior correlates with their course outcome.
  arXiv  Detail & Related papers  (2025-03-11T00:17:07Z)
• Embracing AI in Education: Understanding the Surge in Large Language Model Use by Secondary Students [53.20318273452059]
  Large language models (LLMs) like OpenAI's ChatGPT have opened up new avenues in education.<n>Despite school restrictions, our survey of over 300 middle and high school students revealed that a remarkable 70% of students have utilized LLMs.<n>We propose a few ideas to address such issues, including subject-specific models, personalized learning, and AI classrooms.
  arXiv  Detail & Related papers  (2024-11-27T19:19:34Z)
• Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs [49.18567856499736]
  We investigate whether large language models (LLMs) can be supportive of open-ended dialogue tutoring.<n>We apply a range of knowledge tracing (KT) methods on the resulting labeled data to track student knowledge levels over an entire dialogue.<n>We conduct experiments on two tutoring dialogue datasets, and show that a novel yet simple LLM-based method, LLMKT, significantly outperforms existing KT methods in predicting student response correctness in dialogues.
  arXiv  Detail & Related papers  (2024-09-24T22:31:39Z)
• How Do Students Interact with an LLM-powered Virtual Teaching Assistant in Different Educational Settings? [3.9134031118910264]
  Jill Watson, a virtual teaching assistant powered by LLMs, answers student questions and engages them in extended conversations on courseware provided by the instructors. In this paper, we analyze student interactions with Jill across multiple courses and colleges. We find that, by supporting a wide range of cognitive demands, Jill encourages students to engage in sophisticated, higher-order cognitive questions.
  arXiv  Detail & Related papers  (2024-07-15T01:22:50Z)
• MathChat: Benchmarking Mathematical Reasoning and Instruction Following in Multi-Turn Interactions [58.57255822646756]
  This paper introduces MathChat, a benchmark designed to evaluate large language models (LLMs) across a broader spectrum of mathematical tasks. We evaluate the performance of various SOTA LLMs on the MathChat benchmark, and we observe that while these models excel in single turn question answering, they significantly underperform in more complex scenarios. We develop MathChat sync, a synthetic dialogue based math dataset for LLM finetuning, focusing on improving models' interaction and instruction following capabilities in conversations.
  arXiv  Detail & Related papers  (2024-05-29T18:45:55Z)
• CourseAssist: Pedagogically Appropriate AI Tutor for Computer Science Education [1.052788652996288]
  This poster introduces CourseAssist, a novel LLM-based tutoring system tailored for computer science education. Unlike generic LLM systems, CourseAssist uses retrieval-augmented generation, user intent classification, and question decomposition to align AI responses with specific course materials and learning objectives.
  arXiv  Detail & Related papers  (2024-05-01T20:43:06Z)
• Analyzing LLM Usage in an Advanced Computing Class in India [4.580708389528142]
  This study examines the use of large language models (LLMs) by undergraduate and graduate students for programming assignments in advanced computing classes. We conducted a comprehensive analysis involving 411 students from a Distributed Systems class at an Indian university.
  arXiv  Detail & Related papers  (2024-04-06T12:06:56Z)
• Automate Knowledge Concept Tagging on Math Questions with LLMs [48.5585921817745]
  Knowledge concept tagging for questions plays a crucial role in contemporary intelligent educational applications. Traditionally, these annotations have been conducted manually with help from pedagogical experts. In this paper, we explore the automating the tagging task using Large Language Models (LLMs)
  arXiv  Detail & Related papers  (2024-03-26T00:09:38Z)
• Adapting Large Language Models for Education: Foundational Capabilities, Potentials, and Challenges [60.62904929065257]
  Large language models (LLMs) offer possibility for resolving this issue by comprehending individual requests. This paper reviews the recently emerged LLM research related to educational capabilities, including mathematics, writing, programming, reasoning, and knowledge-based question answering.
  arXiv  Detail & Related papers  (2023-12-27T14:37:32Z)
• Multimodal Lecture Presentations Dataset: Understanding Multimodality in Educational Slides [57.86931911522967]
  We test the capabilities of machine learning models in multimodal understanding of educational content. Our dataset contains aligned slides and spoken language, for 180+ hours of video and 9000+ slides, with 10 lecturers from various subjects. We introduce PolyViLT, a multimodal transformer trained with a multi-instance learning loss that is more effective than current approaches.
  arXiv  Detail & Related papers  (2022-08-17T05:30:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.