61A-Bot: AI homework assistance in CS1 is fast and cheap -- but is it helpful?

• URL: http://arxiv.org/abs/2406.05600v1
• Date: Sun, 9 Jun 2024 00:23:20 GMT
• Title: 61A-Bot: AI homework assistance in CS1 is fast and cheap -- but is it helpful?
• Authors: J. D. Zamfirescu-Pereira, Laryn Qi, Björn Hartmann, John DeNero, Narges Norouzi,
• Abstract summary: This paper reports on the development and deployment of a GPT-4-based interactive homework assistant ("61A-Bot") for students in a large CS1 course. We find substantial reductions in homework completion time. These are most pronounced for students in the 50th-80th percentile, with reductions of over 30 minutes, over 4 standard deviations faster than the mean in prior semesters.
• Score: 9.973179186668393
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Chatbot interfaces for LLMs enable students to get immediate, interactive help on homework assignments, but even a thoughtfully-designed bot may not serve all pedagogical goals. In this paper, we report on the development and deployment of a GPT-4-based interactive homework assistant ("61A-Bot") for students in a large CS1 course; over 2000 students made over 100,000 requests of our bot across two semesters. Our assistant offers one-shot, contextual feedback, through both a "Get Help" button within a popular code editor, as well as a "get feedback" feature within our command-line autograder. These triggers wrap student code in a custom prompt that supports our pedagogical goals and avoids providing solutions directly. We discuss our development process and deployment, then analyze possible impacts of our Bot on students, primarily through student feedback and how long it takes students to complete homework problems. We ask: how does access to 61A-Bot impact homework completion time and subsequent course performance? In addition to reductions in homework-related question rates in our course forum, we find substantial reductions in homework completion time. These are most pronounced for students in the 50th-80th percentile, with reductions of over 30 minutes, over 4 standard deviations faster than the mean in prior semesters. However, it is not clear that these effects transfer to assignment contexts where the Bot is not available: we observe speedups in some contexts, no change in others, and some assignments later in the semester even show a slowdown instead. Though we have begun to disentangle these effects, further research is needed.

Related papers

• WIP: A Unit Testing Framework for Self-Guided Personalized Online Robotics Learning [3.613641107321095]
  This paper focuses on creating a system for unit testing while integrating it into the course workflow. In line with the framework's personalized student-centered approach, this method makes it easier for students to revise, and debug their programming work. The course workflow updated to include unit tests will strengthen the learning environment and make it more interactive so that students can learn how to program robots in a self-guided fashion.
  arXiv  Detail & Related papers  (2024-05-18T00:56:46Z)
• YODA: Teacher-Student Progressive Learning for Language Models [82.0172215948963]
  This paper introduces YODA, a teacher-student progressive learning framework. It emulates the teacher-student education process to improve the efficacy of model fine-tuning. Experiments show that training LLaMA2 with data from YODA improves SFT with significant performance gain.
  arXiv  Detail & Related papers  (2024-01-28T14:32:15Z)
• Using Assignment Incentives to Reduce Student Procrastination and Encourage Code Review Interactions [2.1684358357001465]
  This work presents an incentive system encouraging students to complete assignments many days before deadlines. Completed assignments are code reviewed by staff for correctness and providing feedback, which results in more student-instructor interactions. The incentives result in a change in student behavior with 45% of assignments completed early and 30% up to 4 days before the deadline.
  arXiv  Detail & Related papers  (2023-11-25T22:17:40Z)
• Does Starting Deep Learning Homework Earlier Improve Grades? [63.20583929886827]
  Students who start a homework assignment earlier and spend more time on it should receive better grades on the assignment. Existing literature on the impact of time spent on homework is not clear-cut and comes mostly from K-12 education. We develop a hierarchical Bayesian model to help make principled conclusions about the impact on student success.
  arXiv  Detail & Related papers  (2023-09-30T09:34:30Z)
• Smart tutor to provide feedback in programming courses [0.0]
  We present an AI based intelligent tutor that answers students programming questions. The tool has been tested by university students at the URJC along a whole course.
  arXiv  Detail & Related papers  (2023-01-24T11:00:06Z)
• TEMPERA: Test-Time Prompting via Reinforcement Learning [57.48657629588436]
  We propose Test-time Prompt Editing using Reinforcement learning (TEMPERA) In contrast to prior prompt generation methods, TEMPERA can efficiently leverage prior knowledge. Our method achieves 5.33x on average improvement in sample efficiency when compared to the traditional fine-tuning methods.
  arXiv  Detail & Related papers  (2022-11-21T22:38:20Z)
• Giving Feedback on Interactive Student Programs with Meta-Exploration [74.5597783609281]
  Developing interactive software, such as websites or games, is a particularly engaging way to learn computer science. Standard approaches require instructors to manually grade student-implemented interactive programs. Online platforms that serve millions, like Code.org, are unable to provide any feedback on assignments for implementing interactive programs.
  arXiv  Detail & Related papers  (2022-11-16T10:00:23Z)
• TecCoBot: Technology-aided support for self-regulated learning [52.77024349608834]
  Self-study activities can increase the degree of activity and the contribution of self-study activities to the achievement of learning outcomes. Especially in times of a global pandemic, self-study activities are increasingly executed at home, where students already use technology-enhanced materials, processes, and digital platforms.
  arXiv  Detail & Related papers  (2021-11-23T13:50:21Z)
• ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback [54.142719510638614]
  In this paper, we frame the problem of providing feedback as few-shot classification. A meta-learner adapts to give feedback to student code on a new programming question from just a few examples by instructors. Our approach was successfully deployed to deliver feedback to 16,000 student exam-solutions in a programming course offered by a tier 1 university.
  arXiv  Detail & Related papers  (2021-07-23T22:41:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.