Exploring Student Behaviors and Motivations using AI TAs with Optional Guardrails

• URL: http://arxiv.org/abs/2504.11146v1
• Date: Tue, 15 Apr 2025 12:48:22 GMT
• Title: Exploring Student Behaviors and Motivations using AI TAs with Optional Guardrails
• Abstract summary: "Guardrails" are features that provide scaffolded support rather than direct solutions.<n>"See Solution" feature disabled the guardrails and generated a verbatim response from the underlying model.<n>We found that 50% of the 885 students used the "See Solution" feature for at least one problem and 14% used it for all three problems.
• Score: 2.097564039749475
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: AI-powered chatbots and digital teaching assistants (AI TAs) are gaining popularity in programming education, offering students timely and personalized feedback. Despite their potential benefits, concerns about student over-reliance and academic misconduct have prompted the introduction of "guardrails" into AI TAs - features that provide scaffolded support rather than direct solutions. However, overly restrictive guardrails may lead students to bypass these tools and use unconstrained AI models, where interactions are not observable, thus limiting our understanding of students' help-seeking behaviors. To investigate this, we designed and deployed a novel AI TA tool with optional guardrails in one lab of a large introductory programming course. As students completed three code writing and debugging tasks, they had the option to receive guardrailed help or use a "See Solution" feature which disabled the guardrails and generated a verbatim response from the underlying model. We investigate students' motivations and use of this feature and examine the association between usage and their course performance. We found that 50% of the 885 students used the "See Solution" feature for at least one problem and 14% used it for all three problems. Additionally, low-performing students were more likely to use this feature and use it close to the deadline as they started assignments later. The predominant factors that motivated students to disable the guardrails were assistance in solving problems, time pressure, lack of self-regulation, and curiosity. Our work provides insights into students' solution-seeking motivations and behaviors, which has implications for the design of AI TAs that balance pedagogical goals with student preferences.

Related papers

• "Why Put in This Much Effort?": How AI Availability Shapes Students' Motivation in Introductory Programming [0.15293427903448018]
  We study how engineering students experience and respond to AI availability.<n>Our findings complicate the assumption that students need external constraints to protect their learning.<n>Students who managed the tension found motivation in the learning process itself.
  arXiv  Detail & Related papers  (2026-06-29T15:41:43Z)
• AI-Assisted Help-Seeking Trajectories in Programming Education from an SRL-Informed Perspective [11.163797770670683]
  This study analyzed AI-assisted help-seeking trajectories in university-level programming.<n>Results indicate that many students primarily used AI for reactive troubleshooting rather than for planned, self-regulated problem-solving.
  arXiv  Detail & Related papers  (2026-06-22T03:37:58Z)
• AI-Integrated Learning Management System for Middle School: A Longitudinal Study of Learning Outcomes Through High School and Beyond [0.0]
  Middle school is a key window for building core academic skills.<n>Many students fall behind because help is often limited and comes too late, after they have already been stuck for a while.<n>We propose an AI-integrated LMS for middle school instruction, paired with a longitudinal study design to test whether sustained, bounded AI support changes outcomes.
  arXiv  Detail & Related papers  (2026-05-02T05:10:36Z)
• An Experience Report on a Pedagogically Controlled, Curriculum-Constrained AI Tutor for SE Education [4.976713294177978]
  This paper presents the design and pilot evaluation of RockStartIT Tutor, an AI-powered assistant developed for a digital programming and computational thinking course within the RockStartIT initiative.<n> Powered by GPT-4 via OpenAI's Assistant API, the tutor employs a novel prompting strategy and a modular, semantically tagged knowledge base to deliver context-aware, personalized, and curriculum-constrained support for secondary school students.
  arXiv  Detail & Related papers  (2025-12-08T12:54:37Z)
• UCO: A Multi-Turn Interactive Reinforcement Learning Method for Adaptive Teaching with Large Language Models [59.693733170193944]
  Large language models (LLMs) are shifting from answer providers to intelligent tutors in educational settings.<n>Recent reinforcement learning approaches address this limitation but face two critical challenges.<n>We propose the Unidirectional Cognitive Optimization (UCO) method to address these challenges.
  arXiv  Detail & Related papers  (2025-11-12T01:27:02Z)
• Closing the Loop: An Instructor-in-the-Loop AI Assistance System for Supporting Student Help-Seeking in Programming Education [21.729610894273563]
  We present a hybrid help framework that integrates AI-generated hints with an escalation mechanism.<n>We observed that out of the total 673 AI-generated hints, students rated 146 (22%) as unhelpful.<n>This finding suggests that when AI support fails, even instructors with expertise may need to pay greater attention to avoid making mistakes.
  arXiv  Detail & Related papers  (2025-10-16T08:57:05Z)
• Integrating Generative AI in BIM Education: Insights from Classroom Implementation [0.4805964026801514]
  This study evaluates the implementation of a Generative AI-powered rule checking workflow within a graduate-level Building Information Modeling course.<n>Over two semesters, 55 students participated in a classroom-based pilot exploring the use of GenAI for BIM compliance tasks.
  arXiv  Detail & Related papers  (2025-07-06T03:41:04Z)
• Improving Question Embeddings with Cognitiv Representation Optimization for Knowledge Tracing [77.14348157016518]
  The Knowledge Tracing (KT) aims to track changes in students' knowledge status and predict their future answers based on their historical answer records.<n>Current research on KT modeling focuses on predicting student' future performance based on existing, unupdated records of student learning interactions.<n>We propose a Cognitive Representation Optimization for Knowledge Tracing model, which utilizes a dynamic programming algorithm to optimize structure of cognitive representations.
  arXiv  Detail & Related papers  (2025-04-05T09:32:03Z)
• PyEvalAI: AI-assisted evaluation of Jupyter Notebooks for immediate personalized feedback [43.56788158589046]
  PyEvalAI scores Jupyter notebooks using a combination of unit tests and a locally hosted language model to preserve privacy.<n>A case study demonstrates its effectiveness in improving feedback speed and grading efficiency for exercises in a university-level course on numerics.
  arXiv  Detail & Related papers  (2025-02-25T18:20:20Z)
• How Do Programming Students Use Generative AI? [7.863638253070439]
  We studied how programming students actually use generative AI tools like ChatGPT.<n>We observed two prevalent usage strategies: to seek knowledge about general concepts and to directly generate solutions.<n>Our findings indicate that concerns about potential decrease in programmers' agency and productivity with Generative AI are justified.
  arXiv  Detail & Related papers  (2025-01-17T10:25:41Z)
• Effects of a Prompt Engineering Intervention on Undergraduate Students' AI Self-Efficacy, AI Knowledge and Prompt Engineering Ability: A Mixed Methods Study [36.48421439947282]
  This study designed and implemented a prompt engineering intervention at a university in Hong Kong. It examined students' AI self-efficacy, AI knowledge, and proficiency in creating effective prompts.
  arXiv  Detail & Related papers  (2024-07-30T15:05:24Z)
• Desirable Characteristics for AI Teaching Assistants in Programming Education [2.9131215715703385]
  Digital teaching assistants have emerged as an appealing and scalable way to provide instant, equitable, round-the-clock support. Our results highlight that students value such tools for their ability to provide instant, engaging support. They also expressed a strong preference for features that enable them to retain autonomy in their learning journey.
  arXiv  Detail & Related papers  (2024-05-23T05:03:49Z)
• I would love this to be like an assistant, not the teacher: a voice of the customer perspective of what distance learning students want from an Artificial Intelligence Digital Assistant [0.0]
  This study examined the perceptions of ten online and distance learning students regarding the design of a hypothetical AI Digital Assistant (AIDA) All participants agreed on the usefulness of such an AI tool while studying and reported benefits from using it for real-time assistance and query resolution, support for academic tasks, personalisation and accessibility, together with emotional and social support. Students' concerns related to the ethical and social implications of implementing AIDA, data privacy and data use, operational challenges, academic integrity and misuse, and the future of education.
  arXiv  Detail & Related papers  (2024-02-16T08:10:41Z)
• Towards Goal-oriented Intelligent Tutoring Systems in Online Education [69.06930979754627]
  We propose a new task, named Goal-oriented Intelligent Tutoring Systems (GITS) GITS aims to enable the student's mastery of a designated concept by strategically planning a customized sequence of exercises and assessment. We propose a novel graph-based reinforcement learning framework, named Planning-Assessment-Interaction (PAI)
  arXiv  Detail & Related papers  (2023-12-03T12:37:16Z)
• Plagiarism and AI Assistance Misuse in Web Programming: Unfair Benefits and Characteristics [0.0]
  Plagiarized submissions are similar to the independent ones except in trivial aspects such as color and identifier names. Students believe AI assistance could be useful given proper acknowledgment of the use, although they are not convinced with readability and correctness of the solutions.
  arXiv  Detail & Related papers  (2023-10-31T00:51:14Z)
• MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems [74.73881579517055]
  We propose a framework to generate such dialogues by pairing human teachers with a Large Language Model prompted to represent common student errors. We describe how we use this framework to collect MathDial, a dataset of 3k one-to-one teacher-student tutoring dialogues.
  arXiv  Detail & Related papers  (2023-05-23T21:44:56Z)
• Smart tutor to provide feedback in programming courses [0.0]
  We present an AI based intelligent tutor that answers students programming questions. The tool has been tested by university students at the URJC along a whole course.
  arXiv  Detail & Related papers  (2023-01-24T11:00:06Z)
• Giving Feedback on Interactive Student Programs with Meta-Exploration [74.5597783609281]
  Developing interactive software, such as websites or games, is a particularly engaging way to learn computer science. Standard approaches require instructors to manually grade student-implemented interactive programs. Online platforms that serve millions, like Code.org, are unable to provide any feedback on assignments for implementing interactive programs.
  arXiv  Detail & Related papers  (2022-11-16T10:00:23Z)
• ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback [54.142719510638614]
  In this paper, we frame the problem of providing feedback as few-shot classification. A meta-learner adapts to give feedback to student code on a new programming question from just a few examples by instructors. Our approach was successfully deployed to deliver feedback to 16,000 student exam-solutions in a programming course offered by a tier 1 university.
  arXiv  Detail & Related papers  (2021-07-23T22:41:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.