Benefits and Risks of Using ChatGPT4 as a Teaching Assistant for Computer Science Students

• URL: http://arxiv.org/abs/2411.16690v1
• Date: Fri, 08 Nov 2024 17:11:10 GMT
• Title: Benefits and Risks of Using ChatGPT4 as a Teaching Assistant for Computer Science Students
• Authors: Yaiza Aragonés-Soria, Julia Kotovich, Chitsutha Soomlek, Manuel Oriol,
• Abstract summary: ChatGPT3.5 shocked the software engineering community by its ability to generate answers to specialized questions about coding. This article evaluates this possibility at three levels: fundamental Computer Science knowledge (basic algorithms and data structures), core competency (design patterns), and advanced knowledge (quantum computing) The main result is that the performances of ChatGPT3.5 degrades drastically as the specialization of the domain increases.
• Score: 0.8437187555622164
• License:
• Abstract: Upon release, ChatGPT3.5 shocked the software engineering community by its ability to generate answers to specialized questions about coding. Immediately, many educators wondered if it was possible to use the chatbot as a support tool that helps students answer their programming questions. This article evaluates this possibility at three levels: fundamental Computer Science knowledge (basic algorithms and data structures), core competency (design patterns), and advanced knowledge (quantum computing). In each case, we ask normalized questions several times to ChatGPT3.5, then look at the correctness of answers, and finally check if this creates issues. The main result is that the performances of ChatGPT3.5 degrades drastically as the specialization of the domain increases: for basic algorithms it returns answers that are almost always correct, for design patterns the generated code contains many code smells and is generally of low quality, but it is still sometimes able to fix it (if asked), and for quantum computing it is often blatantly wrong.

Related papers

• Benchmarking ChatGPT on Algorithmic Reasoning [58.50071292008407]
  We evaluate ChatGPT's ability to solve algorithm problems from the CLRS benchmark suite that is designed for GNNs. We find that ChatGPT outperforms specialist GNN models, using Python to successfully solve these problems.
  arXiv  Detail & Related papers  (2024-04-04T13:39:06Z)
• ChatGPT-4 with Code Interpreter can be used to solve introductory college-level vector calculus and electromagnetism problems [0.0]
  We evaluated ChatGPT 3.5, 4, and 4 with Code Interpreter on a set of college-level engineering-math and electromagnetism problems. ChatGPT-4 with Code Interpreter was able to satisfactorily solve most problems we tested most of the time.
  arXiv  Detail & Related papers  (2023-09-16T05:19:39Z)
• Unmasking the giant: A comprehensive evaluation of ChatGPT's proficiency in coding algorithms and data structures [0.6990493129893112]
  We evaluate ChatGPT's ability to generate correct solutions to the problems fed to it, its code quality, and nature of run-time errors thrown by its code. We look into patterns in the test cases passed in order to gain some insights into how wrong ChatGPT code is in these kinds of situations.
  arXiv  Detail & Related papers  (2023-07-10T08:20:34Z)
• Chatbots put to the test in math and logic problems: A preliminary comparison and assessment of ChatGPT-3.5, ChatGPT-4, and Google Bard [68.8204255655161]
  We use 30 questions that are clear, without any ambiguities, fully described with plain text only, and have a unique, well defined correct answer. The answers are recorded and discussed, highlighting their strengths and weaknesses. It was found that ChatGPT-4 outperforms ChatGPT-3.5 in both sets of questions.
  arXiv  Detail & Related papers  (2023-05-30T11:18:05Z)
• Is ChatGPT the Ultimate Programming Assistant -- How far is it? [11.943927095071105]
  ChatGPT has received great attention: it can be used as a bot for discussing source code. We present an empirical study of ChatGPT's potential as a fully automated programming assistant.
  arXiv  Detail & Related papers  (2023-04-24T09:20:13Z)
• ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models [49.52083248451775]
  Large language models (LLMs) have made significant progress in NLP. We specifically focus on ChatGPT, a widely used and easily accessible LLM. We conduct a series of experiments on 11 datasets to evaluate ChatGPT's commonsense abilities.
  arXiv  Detail & Related papers  (2023-03-29T03:05:43Z)
• Analyzing ChatGPT's Aptitude in an Introductory Computer Engineering Course [6.531546527140474]
  ChatGPT is a tool that is able to generate plausible and human-sounding text answers to various questions. This work assesses ChatGPT's aptitude in answering quizzes, homework, exam, and laboratory questions in an introductory computer engineering course.
  arXiv  Detail & Related papers  (2023-03-13T16:22:43Z)
• Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Code [0.0]
  We analyzed effectiveness of three generative pre-trained transformer (GPT) models in answering multiple-choice question (MCQ) assessments. These findings can be leveraged by educators to adapt their instructional practices and assessments in programming courses.
  arXiv  Detail & Related papers  (2023-03-09T16:52:12Z)
• RealTime QA: What's the Answer Right Now? [137.04039209995932]
  We introduce REALTIME QA, a dynamic question answering (QA) platform that announces questions and evaluates systems on a regular basis. We build strong baseline models upon large pretrained language models, including GPT-3 and T5. GPT-3 tends to return outdated answers when retrieved documents do not provide sufficient information to find an answer.
  arXiv  Detail & Related papers  (2022-07-27T07:26:01Z)
• ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback [54.142719510638614]
  In this paper, we frame the problem of providing feedback as few-shot classification. A meta-learner adapts to give feedback to student code on a new programming question from just a few examples by instructors. Our approach was successfully deployed to deliver feedback to 16,000 student exam-solutions in a programming course offered by a tier 1 university.
  arXiv  Detail & Related papers  (2021-07-23T22:41:28Z)
• Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learning [56.771557756836906]
  We present a novel method that automatically learns a retrieval model alternately with the programmer from weak supervision. Our system leads to state-of-the-art performance on a large-scale task for complex question answering over knowledge bases.
  arXiv  Detail & Related papers  (2020-10-29T18:28:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.