Students' Feedback Requests and Interactions with the SCRIPT Chatbot: Do They Get What They Ask For?
- URL: http://arxiv.org/abs/2507.17258v1
- Date: Wed, 23 Jul 2025 06:56:26 GMT
- Title: Students' Feedback Requests and Interactions with the SCRIPT Chatbot: Do They Get What They Ask For?
- Authors: Andreas Scholl, Natalie Kiesler,
- Abstract summary: Building on prior research on Generative AI (GenAI) and related tools for programming education, we developed SCRIPT to support novice learners.<n>SCRIPT allows for open-ended interactions and structured guidance through predefined prompts.<n>We analyzed how students interacted with SCRIPT while solving programming tasks with a focus on their feedback preferences.
- Score: 0.0
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Building on prior research on Generative AI (GenAI) and related tools for programming education, we developed SCRIPT, a chatbot based on ChatGPT-4o-mini, to support novice learners. SCRIPT allows for open-ended interactions and structured guidance through predefined prompts. We evaluated the tool via an experiment with 136 students from an introductory programming course at a large German university and analyzed how students interacted with SCRIPT while solving programming tasks with a focus on their feedback preferences. The results reveal that students' feedback requests seem to follow a specific sequence. Moreover, the chatbot responses aligned well with students' requested feedback types (in 75%), and it adhered to the system prompt constraints. These insights inform the design of GenAI-based learning support systems and highlight challenges in balancing guidance and flexibility in AI-assisted tools.
Related papers
- Exploring Student-AI Interactions in Vibe Coding [6.086654284173657]
The purpose of this study is to understand how students in introductory programming and advanced software engineering classes interact with a vibe coding platform (Replit) when creating software.<n>Interview participants were asked to think-aloud while building a web application using Replit.<n>For both groups, the majority of student interactions with Replit were to test or debug the prototype and only rarely did students visit code.
arXiv Detail & Related papers (2025-07-30T12:35:20Z) - That's Not the Feedback I Need! -- Student Engagement with GenAI Feedback in the Tutor Kai [0.0]
We build a custom web application providing students with Python programming tasks, a code editor, GenAI feedback, and compiler feedback.<n>We investigate how much attention the generated feedback received from learners and to what extent the generated feedback is helpful (or not)<n>The findings indicate that GenAI feedback generally receives a lot of visual attention, with inexperienced students spending twice as much fixation time.
arXiv Detail & Related papers (2025-06-25T13:43:23Z) - Sakshm AI: Advancing AI-Assisted Coding Education for Engineering Students in India Through Socratic Tutoring and Comprehensive Feedback [1.9841192743072902]
Existing AI tools for programming education struggle with key challenges, including the lack of Socratic guidance.<n>This study examines 1170 registered participants, analyzing platform logs, engagement trends, and problem-solving behavior to assess Sakshm AI's impact.
arXiv Detail & Related papers (2025-03-16T12:13:29Z) - How Novice Programmers Use and Experience ChatGPT when Solving Programming Exercises in an Introductory Course [0.0]
This research paper contributes to the computing education research community's understanding of Generative AI (GenAI) in the context of introductory programming.
This study is guided by the following research questions:.
What do students report on their use pattern of ChatGPT in the context of introductory programming exercises?
How do students perceive ChatGPT in the context of introductory programming exercises?
arXiv Detail & Related papers (2024-07-30T12:55:42Z) - Students' Perceptions and Preferences of Generative Artificial
Intelligence Feedback for Programming [15.372316943507506]
We generated automated feedback using the ChatGPT API for four lab assignments in an introductory computer science class.
Students perceived the feedback as aligning well with formative feedback guidelines established by Shute.
Students generally expected specific and corrective feedback with sufficient code examples, but had diverged opinions on the tone of the feedback.
arXiv Detail & Related papers (2023-12-17T22:26:53Z) - Exploring the Potential of Large Language Models to Generate Formative
Programming Feedback [0.5371337604556311]
We explore the potential of large language models (LLMs) for computing educators and learners.
To achieve these goals, we used students' programming sequences from a dataset gathered within a CS1 course as input for ChatGPT.
Results show that ChatGPT performs reasonably well for some of the introductory programming tasks and student errors.
However, educators should provide guidance on how to use the provided feedback, as it can contain misleading information for novices.
arXiv Detail & Related papers (2023-08-31T15:22:11Z) - Automated Distractor and Feedback Generation for Math Multiple-choice
Questions via In-context Learning [43.83422798569986]
Multiple-choice questions (MCQs) are ubiquitous in almost all levels of education since they are easy to administer, grade, and reliable form of assessment.
To date, the task of crafting high-quality distractors has largely remained a labor-intensive process for teachers and learning content designers.
We propose a simple, in-context learning-based solution for automated distractor and corresponding feedback message generation.
arXiv Detail & Related papers (2023-08-07T01:03:04Z) - Developing Effective Educational Chatbots with ChatGPT prompts: Insights
from Preliminary Tests in a Case Study on Social Media Literacy (with
appendix) [43.55994393060723]
Recent advances in language learning models with zero-shot learning capabilities, such as ChatGPT, suggest a new possibility for developing educational chatbots.
We present a case study with a simple system that enables mixed-turn chatbots interactions.
We examine ChatGPT's ability to pursue multiple interconnected learning objectives, adapt the educational activity to users' characteristics, such as culture, age, and level of education, and its ability to use diverse educational strategies and conversational styles.
arXiv Detail & Related papers (2023-06-18T22:23:18Z) - UKP-SQuARE: An Interactive Tool for Teaching Question Answering [61.93372227117229]
The exponential growth of question answering (QA) has made it an indispensable topic in any Natural Language Processing (NLP) course.
We introduce UKP-SQuARE as a platform for QA education.
Students can run, compare, and analyze various QA models from different perspectives.
arXiv Detail & Related papers (2023-05-31T11:29:04Z) - Giving Feedback on Interactive Student Programs with Meta-Exploration [74.5597783609281]
Developing interactive software, such as websites or games, is a particularly engaging way to learn computer science.
Standard approaches require instructors to manually grade student-implemented interactive programs.
Online platforms that serve millions, like Code.org, are unable to provide any feedback on assignments for implementing interactive programs.
arXiv Detail & Related papers (2022-11-16T10:00:23Z) - Play to Grade: Testing Coding Games as Classifying Markov Decision
Process [45.147473767394104]
We formalize the challenge of providing feedback to interactive programs as a task of classifying Markov Decision Processes (MDPs)
Our method enables an automatic feedback system for interactive code assignments.
We release a dataset of 711,274 anonymized student submissions to a single assignment with hand-coded bug labels to support future research.
arXiv Detail & Related papers (2021-10-27T17:37:33Z) - ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback [54.142719510638614]
In this paper, we frame the problem of providing feedback as few-shot classification.
A meta-learner adapts to give feedback to student code on a new programming question from just a few examples by instructors.
Our approach was successfully deployed to deliver feedback to 16,000 student exam-solutions in a programming course offered by a tier 1 university.
arXiv Detail & Related papers (2021-07-23T22:41:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.