Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT

• URL: http://arxiv.org/abs/2405.19132v1
• Date: Wed, 29 May 2024 14:38:32 GMT
• Title: Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT
• Authors: Andreas Scholl, Daniel Schiffner, Natalie Kiesler,
• Abstract summary: Students at a large German university were motivated to solve programming exercises with the assistance of ChatGPT. The data was analyzed w.r.t. the prompts, frequencies, the chats' progress, contents, and other use pattern.
• Score: 0.12289361708127876
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large Language Models (LLMs) have taken the world by storm, and students are assumed to use related tools at a great scale. In this research paper we aim to gain an understanding of how introductory programming students chat with LLMs and related tools, e.g., ChatGPT-3.5. To address this goal, computing students at a large German university were motivated to solve programming exercises with the assistance of ChatGPT as part of their weekly introductory course exercises. Then students (n=213) submitted their chat protocols (with 2335 prompts in sum) as data basis for this analysis. The data was analyzed w.r.t. the prompts, frequencies, the chats' progress, contents, and other use pattern, which revealed a great variety of interactions, both potentially supportive and concerning. Learning about students' interactions with ChatGPT will help inform and align teaching practices and instructions for future introductory programming courses in higher education.

Related papers

• How Novice Programmers Use and Experience ChatGPT when Solving Programming Exercises in an Introductory Course [0.0]
  This research paper contributes to the computing education research community's understanding of Generative AI (GenAI) in the context of introductory programming. This study is guided by the following research questions:. What do students report on their use pattern of ChatGPT in the context of introductory programming exercises? How do students perceive ChatGPT in the context of introductory programming exercises?
  arXiv  Detail & Related papers  (2024-07-30T12:55:42Z)
• Enhancing Programming Education with ChatGPT: A Case Study on Student Perceptions and Interactions in a Python Course [7.182952031323369]
  This paper explores ChatGPT's impact on learning in a Python programming course tailored for first-year students over eight weeks. By analyzing responses from surveys, open-ended questions, and student-ChatGPT dialog data, we aim to provide a comprehensive view of ChatGPT's utility. Our study uncovers a generally positive reception toward ChatGPT and offers insights into its role in enhancing the programming education experience.
  arXiv  Detail & Related papers  (2024-03-20T15:47:28Z)
• Integrating ChatGPT in a Computer Science Course: Students Perceptions and Suggestions [0.0]
  This experience report explores students' perceptions and suggestions for integrating ChatGPT in a computer science course. Findings show the importance of carefully balancing using ChatGPT in computer science courses.
  arXiv  Detail & Related papers  (2023-12-22T10:48:34Z)
• Exploring ChatGPT's Capabilities on Vulnerability Management [56.4403395100589]
  We explore ChatGPT's capabilities on 6 tasks involving the complete vulnerability management process with a large-scale dataset containing 70,346 samples. One notable example is ChatGPT's proficiency in tasks like generating titles for software bug reports. Our findings reveal the difficulties encountered by ChatGPT and shed light on promising future directions.
  arXiv  Detail & Related papers  (2023-11-11T11:01:13Z)
• Exploring the Potential of Large Language Models to Generate Formative Programming Feedback [0.5371337604556311]
  We explore the potential of large language models (LLMs) for computing educators and learners. To achieve these goals, we used students' programming sequences from a dataset gathered within a CS1 course as input for ChatGPT. Results show that ChatGPT performs reasonably well for some of the introductory programming tasks and student errors. However, educators should provide guidance on how to use the provided feedback, as it can contain misleading information for novices.
  arXiv  Detail & Related papers  (2023-08-31T15:22:11Z)
• ChatCoT: Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Models [125.7209927536255]
  We propose ChatCoT, a tool-augmented chain-of-thought reasoning framework for chat-based LLMs. In ChatCoT, we model the chain-of-thought (CoT) reasoning as multi-turn conversations, to utilize tools in a more natural way through chatting. Our approach can effectively leverage the multi-turn conversation ability of chat-based LLMs, and integrate the thought chain following and tools manipulation in a unified way.
  arXiv  Detail & Related papers  (2023-05-23T17:54:33Z)
• Can ChatGPT Pass An Introductory Level Functional Language Programming Course? [2.3456295046913405]
  This paper aims to explore how well ChatGPT can perform in an introductory-level functional language programming course. Our comprehensive evaluation provides valuable insights into ChatGPT's impact from both student and instructor perspectives.
  arXiv  Detail & Related papers  (2023-04-29T20:30:32Z)
• ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning [70.57126720079971]
  Large language models (LLMs) have emerged as the most important breakthroughs in natural language processing (NLP) This paper evaluates ChatGPT on 7 different tasks, covering 37 diverse languages with high, medium, low, and extremely low resources. Compared to the performance of previous models, our extensive experimental results demonstrate a worse performance of ChatGPT for different NLP tasks and languages.
  arXiv  Detail & Related papers  (2023-04-12T05:08:52Z)
• A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity [79.12003701981092]
  We carry out an extensive technical evaluation of ChatGPT using 23 data sets covering 8 different common NLP application tasks. We evaluate the multitask, multilingual and multi-modal aspects of ChatGPT based on these data sets and a newly designed multimodal dataset. ChatGPT is 63.41% accurate on average in 10 different reasoning categories under logical reasoning, non-textual reasoning, and commonsense reasoning.
  arXiv  Detail & Related papers  (2023-02-08T12:35:34Z)
• Is ChatGPT a General-Purpose Natural Language Processing Task Solver? [113.22611481694825]
  Large language models (LLMs) have demonstrated the ability to perform a variety of natural language processing (NLP) tasks zero-shot. Recently, the debut of ChatGPT has drawn a great deal of attention from the natural language processing (NLP) community. It is not yet known whether ChatGPT can serve as a generalist model that can perform many NLP tasks zero-shot.
  arXiv  Detail & Related papers  (2023-02-08T09:44:51Z)
• A Categorical Archive of ChatGPT Failures [47.64219291655723]
  ChatGPT, developed by OpenAI, has been trained using massive amounts of data and simulates human conversation. It has garnered significant attention due to its ability to effectively answer a broad range of human inquiries. However, a comprehensive analysis of ChatGPT's failures is lacking, which is the focus of this study.
  arXiv  Detail & Related papers  (2023-02-06T04:21:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.