ChatGPT: A Study on its Utility for Ubiquitous Software Engineering Tasks

• URL: http://arxiv.org/abs/2305.16837v1
• Date: Fri, 26 May 2023 11:29:06 GMT
• Title: ChatGPT: A Study on its Utility for Ubiquitous Software Engineering Tasks
• Authors: Giriprasad Sridhara and Ranjani H.G. and Sourav Mazumdar
• Abstract summary: ChatGPT (Chat Generative Pre-trained Transformer) launched by OpenAI on November 30, 2022. In this study, we explore how ChatGPT can be used to help with common software engineering tasks.
• Score: 2.084078990567849
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: ChatGPT (Chat Generative Pre-trained Transformer) is a chatbot launched by OpenAI on November 30, 2022. OpenAI's GPT-3 family of large language models serve as the foundation for ChatGPT. ChatGPT is fine-tuned with both supervised and reinforcement learning techniques and has received widespread attention for its articulate responses across diverse domains of knowledge. In this study, we explore how ChatGPT can be used to help with common software engineering tasks. Many of the ubiquitous tasks covering the breadth of software engineering such as ambiguity resolution in software requirements, method name suggestion, test case prioritization, code review, log summarization can potentially be performed using ChatGPT. In this study, we explore fifteen common software engineering tasks using ChatGPT. We juxtapose and analyze ChatGPT's answers with the respective state of the art outputs (where available) and/or human expert ground truth. Our experiments suggest that for many tasks, ChatGPT does perform credibly and the response from it is detailed and often better than the human expert output or the state of the art output. However, for a few other tasks, ChatGPT in its present form provides incorrect answers and hence is not suited for such tasks.

Related papers

• Exploring ChatGPT and its Impact on Society [7.652195319352287]
  ChatGPT is a large language model that can generate human-like responses in a conversational context. It has the potential to revolutionize various industries and transform the way we interact with technology. However, the use of ChatGPT has also raised several concerns, including ethical, social, and employment challenges.
  arXiv  Detail & Related papers  (2024-02-21T16:44:35Z)
• Exploring ChatGPT's Capabilities on Vulnerability Management [56.4403395100589]
  We explore ChatGPT's capabilities on 6 tasks involving the complete vulnerability management process with a large-scale dataset containing 70,346 samples. One notable example is ChatGPT's proficiency in tasks like generating titles for software bug reports. Our findings reveal the difficulties encountered by ChatGPT and shed light on promising future directions.
  arXiv  Detail & Related papers  (2023-11-11T11:01:13Z)
• Primacy Effect of ChatGPT [69.49920102917598]
  We study the primacy effect of ChatGPT: the tendency of selecting the labels at earlier positions as the answer. We hope that our experiments and analyses provide additional insights into building more reliable ChatGPT-based solutions.
  arXiv  Detail & Related papers  (2023-10-20T00:37:28Z)
• Is ChatGPT the Ultimate Programming Assistant -- How far is it? [11.943927095071105]
  ChatGPT has received great attention: it can be used as a bot for discussing source code. We present an empirical study of ChatGPT's potential as a fully automated programming assistant.
  arXiv  Detail & Related papers  (2023-04-24T09:20:13Z)
• ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning [70.57126720079971]
  Large language models (LLMs) have emerged as the most important breakthroughs in natural language processing (NLP) This paper evaluates ChatGPT on 7 different tasks, covering 37 diverse languages with high, medium, low, and extremely low resources. Compared to the performance of previous models, our extensive experimental results demonstrate a worse performance of ChatGPT for different NLP tasks and languages.
  arXiv  Detail & Related papers  (2023-04-12T05:08:52Z)
• ChatGPT-Crawler: Find out if ChatGPT really knows what it's talking about [15.19126287569545]
  This research examines the responses generated by ChatGPT from different Conversational QA corpora. The study employed BERT similarity scores to compare these responses with correct answers and obtain Natural Language Inference(NLI) labels. The study identified instances where ChatGPT provided incorrect answers to questions, providing insights into areas where the model may be prone to error.
  arXiv  Detail & Related papers  (2023-04-06T18:42:47Z)
• One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Era [95.2284704286191]
  GPT-4 (a.k.a. ChatGPT plus) is one small step for generative AI (GAI) but one giant leap for artificial general intelligence (AGI) Since its official release in November 2022, ChatGPT has quickly attracted numerous users with extensive media coverage. This work is the first to survey ChatGPT with a comprehensive review of its underlying technology, applications, and challenges.
  arXiv  Detail & Related papers  (2023-04-04T06:22:09Z)
• On the Educational Impact of ChatGPT: Is Artificial Intelligence Ready to Obtain a University Degree? [0.0]
  We evaluate the influence of ChatGPT on university education. We discuss how computer science higher education should adapt to tools like ChatGPT.
  arXiv  Detail & Related papers  (2023-03-20T14:27:37Z)
• Analyzing ChatGPT's Aptitude in an Introductory Computer Engineering Course [6.531546527140474]
  ChatGPT is a tool that is able to generate plausible and human-sounding text answers to various questions. This work assesses ChatGPT's aptitude in answering quizzes, homework, exam, and laboratory questions in an introductory computer engineering course.
  arXiv  Detail & Related papers  (2023-03-13T16:22:43Z)
• Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT [103.57103957631067]
  ChatGPT has attracted great attention, as it can generate fluent and high-quality responses to human inquiries. We evaluate ChatGPT's understanding ability by evaluating it on the most popular GLUE benchmark, and comparing it with 4 representative fine-tuned BERT-style models. We find that: 1) ChatGPT falls short in handling paraphrase and similarity tasks; 2) ChatGPT outperforms all BERT models on inference tasks by a large margin; 3) ChatGPT achieves comparable performance compared with BERT on sentiment analysis and question answering tasks.
  arXiv  Detail & Related papers  (2023-02-19T12:29:33Z)
• Is ChatGPT a General-Purpose Natural Language Processing Task Solver? [113.22611481694825]
  Large language models (LLMs) have demonstrated the ability to perform a variety of natural language processing (NLP) tasks zero-shot. Recently, the debut of ChatGPT has drawn a great deal of attention from the natural language processing (NLP) community. It is not yet known whether ChatGPT can serve as a generalist model that can perform many NLP tasks zero-shot.
  arXiv  Detail & Related papers  (2023-02-08T09:44:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.