The StudyChat Dataset: Student Dialogues With ChatGPT in an Artificial Intelligence Course

• URL: http://arxiv.org/abs/2503.07928v2
• Date: Sat, 12 Apr 2025 02:42:06 GMT
• Title: The StudyChat Dataset: Student Dialogues With ChatGPT in an Artificial Intelligence Course
• Authors: Hunter McNichols, Andrew Lan,
• Abstract summary: textbfStudyChat is a publicly available dataset capturing real-world student interactions with an LLM-powered tutor.<n>We deploy a web application that replicates ChatGPT's core functionalities, and use it to log student interactions with the LLM.<n>We analyze these interactions, highlight behavioral trends, and analyze how specific usage patterns relate to course outcomes.
• Score: 2.1485350418225244
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The widespread availability of large language models (LLMs), such as ChatGPT, has significantly impacted education, raising both opportunities and challenges. Students can frequently interact with LLM-powered, interactive learning tools, but their usage patterns need to be analyzed to ensure ethical usage of these tools. To better understand how students interact with LLMs in an academic setting, we introduce \textbf{StudyChat}, a publicly available dataset capturing real-world student interactions with an LLM-powered tutoring chatbot in a semester-long, university-level artificial intelligence (AI) course. We deploy a web application that replicates ChatGPT's core functionalities, and use it to log student interactions with the LLM while working on programming assignments. We collect 1,197 conversations, which we annotate using a dialogue act labeling schema inspired by observed interaction patterns and prior research. Additionally, we analyze these interactions, highlight behavioral trends, and analyze how specific usage patterns relate to course outcomes. \textbf{StudyChat} provides a rich resource for the learning sciences and AI in education communities, enabling further research into the evolving role of LLMs in education.

Related papers

• Towards Anthropomorphic Conversational AI Part I: A Practical Framework [49.62013440962072]
  We introduce a multi- module framework designed to replicate the key aspects of human intelligence involved in conversations.<n>In the second stage of our approach, these conversational data, after filtering and labeling, can serve as training and testing data for reinforcement learning.
  arXiv  Detail & Related papers  (2025-02-28T03:18:39Z)
• INTERACT: Enabling Interactive, Question-Driven Learning in Large Language Models [15.825663946923289]
  Large language models (LLMs) excel at answering questions but remain passive learners--absorbing static data without the ability to question and refine knowledge.<n>This paper explores how LLMs can transition to interactive, question-driven learning through student-teacher dialogues.
  arXiv  Detail & Related papers  (2024-12-16T02:28:53Z)
• Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMs [49.18567856499736]
  We investigate whether large language models (LLMs) can be supportive of open-ended dialogue tutoring.<n>We apply a range of knowledge tracing (KT) methods on the resulting labeled data to track student knowledge levels over an entire dialogue.<n>We conduct experiments on two tutoring dialogue datasets, and show that a novel yet simple LLM-based method, LLMKT, significantly outperforms existing KT methods in predicting student response correctness in dialogues.
  arXiv  Detail & Related papers  (2024-09-24T22:31:39Z)
• Impacts of Anthropomorphizing Large Language Models in Learning Environments [0.0]
  Large Language Models (LLMs) are increasingly being used in learning environments to support teaching-be it as learning companions or as tutors. With our contribution, we aim to discuss the implications of the anthropomorphization of LLMs in learning environments on educational theory.
  arXiv  Detail & Related papers  (2024-07-22T06:28:54Z)
• LLM Roleplay: Simulating Human-Chatbot Interaction [52.03241266241294]
  We propose a goal-oriented, persona-based method to automatically generate diverse multi-turn dialogues simulating human-chatbot interaction. Our method can simulate human-chatbot dialogues with a high indistinguishability rate.
  arXiv  Detail & Related papers  (2024-07-04T14:49:46Z)
• MathChat: Benchmarking Mathematical Reasoning and Instruction Following in Multi-Turn Interactions [58.57255822646756]
  This paper introduces MathChat, a benchmark designed to evaluate large language models (LLMs) across a broader spectrum of mathematical tasks. We evaluate the performance of various SOTA LLMs on the MathChat benchmark, and we observe that while these models excel in single turn question answering, they significantly underperform in more complex scenarios. We develop MathChat sync, a synthetic dialogue based math dataset for LLM finetuning, focusing on improving models' interaction and instruction following capabilities in conversations.
  arXiv  Detail & Related papers  (2024-05-29T18:45:55Z)
• Generating Situated Reflection Triggers about Alternative Solution Paths: A Case Study of Generative AI for Computer-Supported Collaborative Learning [3.2721068185888127]
  We present a proof-of-concept application to offer students dynamic and contextualized feedback. Specifically, we augment an Online Programming Exercise bot for a college-level Cloud Computing course with ChatGPT. We demonstrate that LLMs can be used to generate highly situated reflection triggers that incorporate details of the collaborative discussion happening in context.
  arXiv  Detail & Related papers  (2024-04-28T17:56:14Z)
• Analyzing LLM Usage in an Advanced Computing Class in India [4.580708389528142]
  This study examines the use of large language models (LLMs) by undergraduate and graduate students for programming assignments in advanced computing classes. We conducted a comprehensive analysis involving 411 students from a Distributed Systems class at an Indian university.
  arXiv  Detail & Related papers  (2024-04-06T12:06:56Z)
• Investigation of the effectiveness of applying ChatGPT in Dialogic Teaching Using Electroencephalography [6.34494999013996]
  Large language models (LLMs) possess the capability to interpret knowledge, answer questions, and consider context. This research recruited 34 undergraduate students as participants, who were randomly divided into two groups. The experimental group engaged in dialogic teaching using ChatGPT, while the control group interacted with human teachers.
  arXiv  Detail & Related papers  (2024-03-25T12:23:12Z)
• LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models [56.25156596019168]
  This paper introduces the LMRL-Gym benchmark for evaluating multi-turn RL for large language models (LLMs) Our benchmark consists of 8 different language tasks, which require multiple rounds of language interaction and cover a range of tasks in open-ended dialogue and text games.
  arXiv  Detail & Related papers  (2023-11-30T03:59:31Z)
• Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations [70.7884839812069]
  Large language models (LLMs) have emerged as powerful and general solutions to many natural language tasks. However, many of the most important applications of language generation are interactive, where an agent has to talk to a person to reach a desired outcome. In this work, we explore a new method for adapting LLMs with RL for such goal-directed dialogue.
  arXiv  Detail & Related papers  (2023-11-09T18:45:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.