61A-Bot Report: AI Assistants in CS1 Save Students Homework Time and Reduce Demands on Staff. (Now What?)

• URL: http://arxiv.org/abs/2406.05600v2
• Date: Fri, 30 Aug 2024 20:05:36 GMT
• Title: 61A-Bot Report: AI Assistants in CS1 Save Students Homework Time and Reduce Demands on Staff. (Now What?)
• Authors: J. D. Zamfirescu-Pereira, Laryn Qi, Björn Hartmann, John DeNero, Narges Norouzi,
• Abstract summary: GPT-4-based interactive homework assistant ("61A Bot") for students in a large CS1 course. Over 2000 students made over 100,000 requests of our bot across two semesters. For students in the 50th-80th percentile, these reductions typically exceed 30 minutes per assignment.
• Score: 9.973179186668393
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Chatbot interfaces for LLMs enable students to get immediate, interactive help on homework assignments, but even a thoughtfully-designed bot may not serve all pedagogical goals. In this paper, we report on the development and deployment of a GPT-4-based interactive homework assistant ("61A Bot") for students in a large CS1 course; over 2000 students made over 100,000 requests of our bot across two semesters. Our assistant offers one-shot, contextual feedback, primarily through a low-friction "get feedback" prompt within the command-line "autograder" our students already run to test their code. Our Bot wraps student code in a custom prompt that supports our pedagogical goals and avoids providing solutions directly. We discuss our deployment and then analyze the impacts of our Bot on students, primarily through student-reported feedback and tracking of student homework progress. We find reductions in homework-related question rates in our course forum, as well as substantial reductions in homework completion time when our Bot is available. For students in the 50th-80th percentile, these reductions typically exceed 30 minutes per assignment, over 4 standard deviations faster than the mean in prior semesters. Finally, we conclude with a discussion of these observations, the potential impacts on student learning, as well as other potential costs and benefits of AI assistance in CS1.

Related papers

• Exploring Student Behaviors and Motivations using AI TAs with Optional Guardrails [2.097564039749475]
  "Guardrails" are features that provide scaffolded support rather than direct solutions. "See Solution" feature disabled the guardrails and generated a verbatim response from the underlying model. We found that 50% of the 885 students used the "See Solution" feature for at least one problem and 14% used it for all three problems.
  arXiv  Detail & Related papers  (2025-04-15T12:48:22Z)
• Multimodal Programming in Computer Science with Interactive Assistance Powered by Large Language Model [0.0]
  An interactive homework help system based on DeepSeek R1 is developed and first implemented for students enrolled in a large computer science beginning programming course. It wraps student work in a personalized prompt that advances our educational objectives without offering answers straight away.
  arXiv  Detail & Related papers  (2025-03-09T10:48:47Z)
• How Do Programming Students Use Generative AI? [7.863638253070439]
  We studied how programming students actually use generative AI tools like ChatGPT. We observed two prevalent usage strategies: to seek knowledge about general concepts and to directly generate solutions. Our findings indicate that concerns about potential decrease in programmers' agency and productivity with Generative AI are justified.
  arXiv  Detail & Related papers  (2025-01-17T10:25:41Z)
• GPT-4 as a Homework Tutor can Improve Student Engagement and Learning Outcomes [80.60912258178045]
  We developed a prompting strategy that enables GPT-4 to conduct interactive homework sessions for high-school students learning English as a second language. We carried out a Randomized Controlled Trial (RCT) in four high-school classes, replacing traditional homework with GPT-4 homework sessions for the treatment group. We observed significant improvements in learning outcomes, specifically a greater gain in grammar, and student engagement.
  arXiv  Detail & Related papers  (2024-09-24T11:22:55Z)
• Interactive Learning in Computer Science Education Supported by a Discord Chatbot [3.0294711465150006]
  The DiscordBot enables students to provide feedback on course activities through short surveys. It also supports attendance tracking and introduces lectures before they start. The data collected reveal that students can accurately perceive the activities' difficulty and expected results.
  arXiv  Detail & Related papers  (2024-07-27T14:22:40Z)
• Integrating AI Tutors in a Programming Course [0.0]
  RAGMan is an LLM-powered tutoring system that can support a variety of course-specific and homework-specific AI tutors. This paper describes the interactions the students had with the AI tutors, the students' feedback, and a comparative grade analysis.
  arXiv  Detail & Related papers  (2024-07-14T00:42:39Z)
• YODA: Teacher-Student Progressive Learning for Language Models [82.0172215948963]
  This paper introduces YODA, a teacher-student progressive learning framework. It emulates the teacher-student education process to improve the efficacy of model fine-tuning. Experiments show that training LLaMA2 with data from YODA improves SFT with significant performance gain.
  arXiv  Detail & Related papers  (2024-01-28T14:32:15Z)
• Using Assignment Incentives to Reduce Student Procrastination and Encourage Code Review Interactions [2.1684358357001465]
  This work presents an incentive system encouraging students to complete assignments many days before deadlines. Completed assignments are code reviewed by staff for correctness and providing feedback, which results in more student-instructor interactions. The incentives result in a change in student behavior with 45% of assignments completed early and 30% up to 4 days before the deadline.
  arXiv  Detail & Related papers  (2023-11-25T22:17:40Z)
• MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems [74.73881579517055]
  We propose a framework to generate such dialogues by pairing human teachers with a Large Language Model prompted to represent common student errors. We describe how we use this framework to collect MathDial, a dataset of 3k one-to-one teacher-student tutoring dialogues.
  arXiv  Detail & Related papers  (2023-05-23T21:44:56Z)
• Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning [54.636562516974884]
  In imitation and reinforcement learning, the cost of human supervision limits the amount of data that robots can be trained on. In this work, we propose MEDAL++, a novel design for self-improving robotic systems. The robot autonomously practices the task by learning to both do and undo the task, simultaneously inferring the reward function from the demonstrations.
  arXiv  Detail & Related papers  (2023-03-02T18:51:38Z)
• Giving Feedback on Interactive Student Programs with Meta-Exploration [74.5597783609281]
  Developing interactive software, such as websites or games, is a particularly engaging way to learn computer science. Standard approaches require instructors to manually grade student-implemented interactive programs. Online platforms that serve millions, like Code.org, are unable to provide any feedback on assignments for implementing interactive programs.
  arXiv  Detail & Related papers  (2022-11-16T10:00:23Z)
• From {Solution Synthesis} to {Student Attempt Synthesis} for Block-Based Visual Programming Tasks [20.64766977405438]
  We introduce a novel benchmark, StudentSyn, centered around the following challenge. For a given student, synthesize the student's attempt on a new target task after observing the student's attempt on a fixed reference task. This challenge is akin to that of program synthesis; however, instead of a solution (i.e., program an expert would write), the goal here is to synthesize a student attempt (i.e., program that a given student would write)
  arXiv  Detail & Related papers  (2022-05-03T01:32:47Z)
• TecCoBot: Technology-aided support for self-regulated learning [52.77024349608834]
  Self-study activities can increase the degree of activity and the contribution of self-study activities to the achievement of learning outcomes. Especially in times of a global pandemic, self-study activities are increasingly executed at home, where students already use technology-enhanced materials, processes, and digital platforms.
  arXiv  Detail & Related papers  (2021-11-23T13:50:21Z)
• CheerBots: Chatbots toward Empathy and Emotionusing Reinforcement Learning [60.348822346249854]
  This study presents a framework whereby several empathetic chatbots are based on understanding users' implied feelings and replying empathetically for multiple dialogue turns. We call these chatbots CheerBots. CheerBots can be retrieval-based or generative-based and were finetuned by deep reinforcement learning. To respond in an empathetic way, we develop a simulating agent, a Conceptual Human Model, as aids for CheerBots in training with considerations on changes in user's emotional states in the future to arouse sympathy.
  arXiv  Detail & Related papers  (2021-10-08T07:44:47Z)
• Using Machine Learning to Predict Engineering Technology Students' Success with Computer Aided Design [50.591267188664666]
  We show how data combined with machine learning techniques can predict how well a particular student will perform in a design task. We found that our models using early design sequence actions are particularly valuable for prediction. Further improvements to these models could lead to earlier predictions and thus provide students feedback sooner to enhance their learning.
  arXiv  Detail & Related papers  (2021-08-12T20:24:54Z)
• ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback [54.142719510638614]
  In this paper, we frame the problem of providing feedback as few-shot classification. A meta-learner adapts to give feedback to student code on a new programming question from just a few examples by instructors. Our approach was successfully deployed to deliver feedback to 16,000 student exam-solutions in a programming course offered by a tier 1 university.
  arXiv  Detail & Related papers  (2021-07-23T22:41:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.