Exploring Student Behaviors and Motivations using AI TAs with Optional Guardrails

• URL: http://arxiv.org/abs/2504.11146v1
• Date: Tue, 15 Apr 2025 12:48:22 GMT
• Title: Exploring Student Behaviors and Motivations using AI TAs with Optional Guardrails
• Authors: Amanpreet Kapoor, Marc Diaz, Stephen MacNeil, Leo Porter, Paul Denny,
• Abstract summary: "Guardrails" are features that provide scaffolded support rather than direct solutions.<n>"See Solution" feature disabled the guardrails and generated a verbatim response from the underlying model.<n>We found that 50% of the 885 students used the "See Solution" feature for at least one problem and 14% used it for all three problems.
• Score: 2.097564039749475
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: AI-powered chatbots and digital teaching assistants (AI TAs) are gaining popularity in programming education, offering students timely and personalized feedback. Despite their potential benefits, concerns about student over-reliance and academic misconduct have prompted the introduction of "guardrails" into AI TAs - features that provide scaffolded support rather than direct solutions. However, overly restrictive guardrails may lead students to bypass these tools and use unconstrained AI models, where interactions are not observable, thus limiting our understanding of students' help-seeking behaviors. To investigate this, we designed and deployed a novel AI TA tool with optional guardrails in one lab of a large introductory programming course. As students completed three code writing and debugging tasks, they had the option to receive guardrailed help or use a "See Solution" feature which disabled the guardrails and generated a verbatim response from the underlying model. We investigate students' motivations and use of this feature and examine the association between usage and their course performance. We found that 50% of the 885 students used the "See Solution" feature for at least one problem and 14% used it for all three problems. Additionally, low-performing students were more likely to use this feature and use it close to the deadline as they started assignments later. The predominant factors that motivated students to disable the guardrails were assistance in solving problems, time pressure, lack of self-regulation, and curiosity. Our work provides insights into students' solution-seeking motivations and behaviors, which has implications for the design of AI TAs that balance pedagogical goals with student preferences.

Related papers

• Improving Question Embeddings with Cognitiv Representation Optimization for Knowledge Tracing [77.14348157016518]
  The Knowledge Tracing (KT) aims to track changes in students' knowledge status and predict their future answers based on their historical answer records.<n>Current research on KT modeling focuses on predicting student' future performance based on existing, unupdated records of student learning interactions.<n>We propose a Cognitive Representation Optimization for Knowledge Tracing model, which utilizes a dynamic programming algorithm to optimize structure of cognitive representations.
  arXiv  Detail & Related papers  (2025-04-05T09:32:03Z)
• PyEvalAI: AI-assisted evaluation of Jupyter Notebooks for immediate personalized feedback [43.56788158589046]
  PyEvalAI scores Jupyter notebooks using a combination of unit tests and a locally hosted language model to preserve privacy.<n>A case study demonstrates its effectiveness in improving feedback speed and grading efficiency for exercises in a university-level course on numerics.
  arXiv  Detail & Related papers  (2025-02-25T18:20:20Z)
• How Do Programming Students Use Generative AI? [7.863638253070439]
  We studied how programming students actually use generative AI tools like ChatGPT.<n>We observed two prevalent usage strategies: to seek knowledge about general concepts and to directly generate solutions.<n>Our findings indicate that concerns about potential decrease in programmers' agency and productivity with Generative AI are justified.
  arXiv  Detail & Related papers  (2025-01-17T10:25:41Z)
• Desirable Characteristics for AI Teaching Assistants in Programming Education [2.9131215715703385]
  Digital teaching assistants have emerged as an appealing and scalable way to provide instant, equitable, round-the-clock support. Our results highlight that students value such tools for their ability to provide instant, engaging support. They also expressed a strong preference for features that enable them to retain autonomy in their learning journey.
  arXiv  Detail & Related papers  (2024-05-23T05:03:49Z)
• I would love this to be like an assistant, not the teacher: a voice of the customer perspective of what distance learning students want from an Artificial Intelligence Digital Assistant [0.0]
  This study examined the perceptions of ten online and distance learning students regarding the design of a hypothetical AI Digital Assistant (AIDA) All participants agreed on the usefulness of such an AI tool while studying and reported benefits from using it for real-time assistance and query resolution, support for academic tasks, personalisation and accessibility, together with emotional and social support. Students' concerns related to the ethical and social implications of implementing AIDA, data privacy and data use, operational challenges, academic integrity and misuse, and the future of education.
  arXiv  Detail & Related papers  (2024-02-16T08:10:41Z)
• Towards Goal-oriented Intelligent Tutoring Systems in Online Education [69.06930979754627]
  We propose a new task, named Goal-oriented Intelligent Tutoring Systems (GITS) GITS aims to enable the student's mastery of a designated concept by strategically planning a customized sequence of exercises and assessment. We propose a novel graph-based reinforcement learning framework, named Planning-Assessment-Interaction (PAI)
  arXiv  Detail & Related papers  (2023-12-03T12:37:16Z)
• Plagiarism and AI Assistance Misuse in Web Programming: Unfair Benefits and Characteristics [0.0]
  Plagiarized submissions are similar to the independent ones except in trivial aspects such as color and identifier names. Students believe AI assistance could be useful given proper acknowledgment of the use, although they are not convinced with readability and correctness of the solutions.
  arXiv  Detail & Related papers  (2023-10-31T00:51:14Z)
• MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems [74.73881579517055]
  We propose a framework to generate such dialogues by pairing human teachers with a Large Language Model prompted to represent common student errors. We describe how we use this framework to collect MathDial, a dataset of 3k one-to-one teacher-student tutoring dialogues.
  arXiv  Detail & Related papers  (2023-05-23T21:44:56Z)
• Smart tutor to provide feedback in programming courses [0.0]
  We present an AI based intelligent tutor that answers students programming questions. The tool has been tested by university students at the URJC along a whole course.
  arXiv  Detail & Related papers  (2023-01-24T11:00:06Z)
• Giving Feedback on Interactive Student Programs with Meta-Exploration [74.5597783609281]
  Developing interactive software, such as websites or games, is a particularly engaging way to learn computer science. Standard approaches require instructors to manually grade student-implemented interactive programs. Online platforms that serve millions, like Code.org, are unable to provide any feedback on assignments for implementing interactive programs.
  arXiv  Detail & Related papers  (2022-11-16T10:00:23Z)
• ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback [54.142719510638614]
  In this paper, we frame the problem of providing feedback as few-shot classification. A meta-learner adapts to give feedback to student code on a new programming question from just a few examples by instructors. Our approach was successfully deployed to deliver feedback to 16,000 student exam-solutions in a programming course offered by a tier 1 university.
  arXiv  Detail & Related papers  (2021-07-23T22:41:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.