Patterns of Student Help-Seeking When Using a Large Language Model-Powered Programming Assistant

• URL: http://arxiv.org/abs/2310.16984v1
• Date: Wed, 25 Oct 2023 20:36:05 GMT
• Title: Patterns of Student Help-Seeking When Using a Large Language Model-Powered Programming Assistant
• Authors: Brad Sheese, Mark Liffiton, Jaromir Savelka, Paul Denny
• Abstract summary: This study examines students' use of an innovative tool that provides on-demand programming assistance without revealing solutions directly. We collected more than 2,500 queries submitted by students throughout the term. We found that most queries requested immediate help with programming assignments, whereas fewer requests asked for help on related concepts or for deepening conceptual understanding.
• Score: 2.5949084781328744
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Providing personalized assistance at scale is a long-standing challenge for computing educators, but a new generation of tools powered by large language models (LLMs) offers immense promise. Such tools can, in theory, provide on-demand help in large class settings and be configured with appropriate guardrails to prevent misuse and mitigate common concerns around learner over-reliance. However, the deployment of LLM-powered tools in authentic classroom settings is still rare, and very little is currently known about how students will use them in practice and what type of help they will seek. To address this, we examine students' use of an innovative LLM-powered tool that provides on-demand programming assistance without revealing solutions directly. We deployed the tool for 12 weeks in an introductory computer and data science course ($n = 52$), collecting more than 2,500 queries submitted by students throughout the term. We manually categorized all student queries based on the type of assistance sought, and we automatically analyzed several additional query characteristics. We found that most queries requested immediate help with programming assignments, whereas fewer requests asked for help on related concepts or for deepening conceptual understanding. Furthermore, students often provided minimal information to the tool, suggesting this is an area in which targeted instruction would be beneficial. We also found that students who achieved more success in the course tended to have used the tool more frequently overall. Lessons from this research can be leveraged by programming educators and institutions who plan to augment their teaching with emerging LLM-powered tools.

Related papers

• Learning to Ask: When LLMs Meet Unclear Instruction [49.256630152684764]
  Large language models (LLMs) can leverage external tools for addressing a range of tasks unattainable through language skills alone. We evaluate the performance of LLMs tool-use under imperfect instructions, analyze the error patterns, and build a challenging tool-use benchmark called Noisy ToolBench. We propose a novel framework, Ask-when-Needed (AwN), which prompts LLMs to ask questions to users whenever they encounter obstacles due to unclear instructions.
  arXiv  Detail & Related papers  (2024-08-31T23:06:12Z)
• What You Need is What You Get: Theory of Mind for an LLM-Based Code Understanding Assistant [0.0]
  A growing number of tools have used Large Language Models (LLMs) to support developers' code understanding. In this study, we designed an LLM-based conversational assistant that provides a personalized interaction based on inferred user mental state. Our results provide insights for researchers and tool builders who want to create or improve LLM-based conversational assistants to support novices in code understanding.
  arXiv  Detail & Related papers  (2024-08-08T14:08:15Z)
• Chain of Tools: Large Language Model is an Automatic Multi-tool Learner [54.992464510992605]
  Automatic Tool Chain (ATC) is a framework that enables the large language models (LLMs) to act as a multi-tool user. To scale up the scope of the tools, we next propose a black-box probing method. For a comprehensive evaluation, we build a challenging benchmark named ToolFlow.
  arXiv  Detail & Related papers  (2024-05-26T11:40:58Z)
• Towards Completeness-Oriented Tool Retrieval for Large Language Models [60.733557487886635]
  Real-world systems often incorporate a wide array of tools, making it impractical to input all tools into Large Language Models. Existing tool retrieval methods primarily focus on semantic matching between user queries and tool descriptions. We propose a novel modelagnostic COllaborative Learning-based Tool Retrieval approach, COLT, which captures not only the semantic similarities between user queries and tool descriptions but also takes into account the collaborative information of tools.
  arXiv  Detail & Related papers  (2024-05-25T06:41:23Z)
• Desirable Characteristics for AI Teaching Assistants in Programming Education [2.9131215715703385]
  Digital teaching assistants have emerged as an appealing and scalable way to provide instant, equitable, round-the-clock support. Our results highlight that students value such tools for their ability to provide instant, engaging support. They also expressed a strong preference for features that enable them to retain autonomy in their learning journey.
  arXiv  Detail & Related papers  (2024-05-23T05:03:49Z)
• CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Update [69.59482029810198]
  CLOVA is a Closed-Loop Visual Assistant that operates within a framework encompassing inference, reflection, and learning phases. Results demonstrate that CLOVA surpasses existing tool-usage methods by 5% in visual question answering and multiple-image reasoning, by 10% in knowledge tagging, and by 20% in image editing.
  arXiv  Detail & Related papers  (2023-12-18T03:34:07Z)
• Next-Step Hint Generation for Introductory Programming Using Large Language Models [0.8002196839441036]
  Large Language Models possess skills such as answering questions, writing essays or solving programming exercises. This work explores how LLMs can contribute to programming education by supporting students with automated next-step hints.
  arXiv  Detail & Related papers  (2023-12-03T17:51:07Z)
• Dcc --help: Generating Context-Aware Compiler Error Explanations with Large Language Models [53.04357141450459]
  dcc --help was deployed to our CS1 and CS2 courses, with 2,565 students using the tool over 64,000 times in ten weeks. We found that the LLM-generated explanations were conceptually accurate in 90% of compile-time and 75% of run-time cases, but often disregarded the instruction not to provide solutions in code.
  arXiv  Detail & Related papers  (2023-08-23T02:36:19Z)
• CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes [2.5949084781328744]
  Large language models (LLMs) have emerged recently and show great promise for providing on-demand help at a large scale. We introduce CodeHelp, a novel LLM-powered tool designed with guardrails to provide on-demand assistance to programming students without directly revealing solutions. Our findings suggest that CodeHelp is well-received by students who especially value its availability and help with resolving errors, and that for instructors it is easy to deploy and complements, rather than replaces, the support that they provide to students.
  arXiv  Detail & Related papers  (2023-08-14T03:52:24Z)
• A large language model-assisted education tool to provide feedback on open-ended responses [2.624902795082451]
  We present a tool that uses large language models (LLMs), guided by instructor-defined criteria, to automate responses to open-ended questions. Our tool delivers rapid personalized feedback, enabling students to quickly test their knowledge and identify areas for improvement.
  arXiv  Detail & Related papers  (2023-07-25T19:49:55Z)
• Large Language Models as Tool Makers [85.00361145117293]
  We introduce a closed-loop framework, referred to as LLMs A s Tool Makers (LATM), where LLMs create their own reusable tools for problem-solving. Our approach consists of two phases: 1) tool making: an LLM acts as the tool maker that crafts tools for a set of tasks. 2) tool using: another LLM acts as the tool user, which applies the tool built by the tool maker for problem-solving.
  arXiv  Detail & Related papers  (2023-05-26T17:50:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.