CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes

• URL: http://arxiv.org/abs/2308.06921v1
• Date: Mon, 14 Aug 2023 03:52:24 GMT
• Title: CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes
• Authors: Mark Liffiton, Brad Sheese, Jaromir Savelka, Paul Denny
• Abstract summary: Large language models (LLMs) have emerged recently and show great promise for providing on-demand help at a large scale. We introduce CodeHelp, a novel LLM-powered tool designed with guardrails to provide on-demand assistance to programming students without directly revealing solutions. Our findings suggest that CodeHelp is well-received by students who especially value its availability and help with resolving errors, and that for instructors it is easy to deploy and complements, rather than replaces, the support that they provide to students.
• Score: 2.5949084781328744
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Computing educators face significant challenges in providing timely support to students, especially in large class settings. Large language models (LLMs) have emerged recently and show great promise for providing on-demand help at a large scale, but there are concerns that students may over-rely on the outputs produced by these models. In this paper, we introduce CodeHelp, a novel LLM-powered tool designed with guardrails to provide on-demand assistance to programming students without directly revealing solutions. We detail the design of the tool, which incorporates a number of useful features for instructors, and elaborate on the pipeline of prompting strategies we use to ensure generated outputs are suitable for students. To evaluate CodeHelp, we deployed it in a first-year computer and data science course with 52 students and collected student interactions over a 12-week period. We examine students' usage patterns and perceptions of the tool, and we report reflections from the course instructor and a series of recommendations for classroom use. Our findings suggest that CodeHelp is well-received by students who especially value its availability and help with resolving errors, and that for instructors it is easy to deploy and complements, rather than replaces, the support that they provide to students.

Related papers

• Learning to Ask: When LLMs Meet Unclear Instruction [49.256630152684764]
  Large language models (LLMs) can leverage external tools for addressing a range of tasks unattainable through language skills alone. We evaluate the performance of LLMs tool-use under imperfect instructions, analyze the error patterns, and build a challenging tool-use benchmark called Noisy ToolBench. We propose a novel framework, Ask-when-Needed (AwN), which prompts LLMs to ask questions to users whenever they encounter obstacles due to unclear instructions.
  arXiv  Detail & Related papers  (2024-08-31T23:06:12Z)
• Stepwise Verification and Remediation of Student Reasoning Errors with Large Language Model Tutors [78.53699244846285]
  Large language models (LLMs) present an opportunity to scale high-quality personalized education to all. LLMs struggle to precisely detect student's errors and tailor their feedback to these errors. Inspired by real-world teaching practice where teachers identify student errors and customize their response based on them, we focus on verifying student solutions.
  arXiv  Detail & Related papers  (2024-07-12T10:11:40Z)
• Desirable Characteristics for AI Teaching Assistants in Programming Education [2.9131215715703385]
  Digital teaching assistants have emerged as an appealing and scalable way to provide instant, equitable, round-the-clock support. Our results highlight that students value such tools for their ability to provide instant, engaging support. They also expressed a strong preference for features that enable them to retain autonomy in their learning journey.
  arXiv  Detail & Related papers  (2024-05-23T05:03:49Z)
• A Picture Is Worth a Thousand Words: Exploring Diagram and Video-Based OOP Exercises to Counter LLM Over-Reliance [2.1490831374964587]
  Large language models (LLMs) can effectively solve a range of more complex object-oriented programming (OOP) exercises with text-based specifications. This raises concerns about academic integrity, as students might use these models to complete assignments unethically. We propose an innovative approach to formulating OOP tasks using diagrams and videos, as a way to foster problem-solving and deter students from a copy-and-prompt approach in OOP courses.
  arXiv  Detail & Related papers  (2024-03-13T10:21:29Z)
• CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Update [69.59482029810198]
  CLOVA is a Closed-Loop Visual Assistant that operates within a framework encompassing inference, reflection, and learning phases. Results demonstrate that CLOVA surpasses existing tool-usage methods by 5% in visual question answering and multiple-image reasoning, by 10% in knowledge tagging, and by 20% in image editing.
  arXiv  Detail & Related papers  (2023-12-18T03:34:07Z)
• Patterns of Student Help-Seeking When Using a Large Language Model-Powered Programming Assistant [2.5949084781328744]
  This study examines students' use of an innovative tool that provides on-demand programming assistance without revealing solutions directly. We collected more than 2,500 queries submitted by students throughout the term. We found that most queries requested immediate help with programming assignments, whereas fewer requests asked for help on related concepts or for deepening conceptual understanding.
  arXiv  Detail & Related papers  (2023-10-25T20:36:05Z)
• Dcc --help: Generating Context-Aware Compiler Error Explanations with Large Language Models [53.04357141450459]
  dcc --help was deployed to our CS1 and CS2 courses, with 2,565 students using the tool over 64,000 times in ten weeks. We found that the LLM-generated explanations were conceptually accurate in 90% of compile-time and 75% of run-time cases, but often disregarded the instruction not to provide solutions in code.
  arXiv  Detail & Related papers  (2023-08-23T02:36:19Z)
• A large language model-assisted education tool to provide feedback on open-ended responses [2.624902795082451]
  We present a tool that uses large language models (LLMs), guided by instructor-defined criteria, to automate responses to open-ended questions. Our tool delivers rapid personalized feedback, enabling students to quickly test their knowledge and identify areas for improvement.
  arXiv  Detail & Related papers  (2023-07-25T19:49:55Z)
• MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems [74.73881579517055]
  We propose a framework to generate such dialogues by pairing human teachers with a Large Language Model prompted to represent common student errors. We describe how we use this framework to collect MathDial, a dataset of 3k one-to-one teacher-student tutoring dialogues.
  arXiv  Detail & Related papers  (2023-05-23T21:44:56Z)
• Demonstrating REACT: a Real-time Educational AI-powered Classroom Tool [0.9899017174990579]
  We present a new Real-time Educational AI-powered Classroom Tool that employs EDM techniques for supporting the decision-making process of educators. ReACT is a data-driven tool with a user-friendly graphical interface. It analyzes students' performance data and provides context-based alerts as well as recommendations to educators for course planning.
  arXiv  Detail & Related papers  (2021-07-30T03:09:59Z)
• ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback [54.142719510638614]
  In this paper, we frame the problem of providing feedback as few-shot classification. A meta-learner adapts to give feedback to student code on a new programming question from just a few examples by instructors. Our approach was successfully deployed to deliver feedback to 16,000 student exam-solutions in a programming course offered by a tier 1 university.
  arXiv  Detail & Related papers  (2021-07-23T22:41:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.