Comparing Software Developers with ChatGPT: An Empirical Investigation
- URL: http://arxiv.org/abs/2305.11837v2
- Date: Thu, 25 May 2023 14:58:39 GMT
- Title: Comparing Software Developers with ChatGPT: An Empirical Investigation
- Authors: Nathalia Nascimento and Paulo Alencar and Donald Cowan
- Abstract summary: This paper conducts an empirical investigation, contrasting the performance of software engineers and AI systems, like ChatGPT, across different evaluation metrics.
The paper posits that a comprehensive comparison of software engineers and AI-based solutions, considering various evaluation criteria, is pivotal in fostering human-machine collaboration.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The advent of automation in particular Software Engineering (SE) tasks has
transitioned from theory to reality. Numerous scholarly articles have
documented the successful application of Artificial Intelligence to address
issues in areas such as project management, modeling, testing, and development.
A recent innovation is the introduction of ChatGPT, an ML-infused chatbot,
touted as a resource proficient in generating programming codes and formulating
software testing strategies for developers and testers respectively. Although
there is speculation that AI-based computation can increase productivity and
even substitute software engineers in software development, there is currently
a lack of empirical evidence to verify this. Moreover, despite the primary
focus on enhancing the accuracy of AI systems, non-functional requirements
including energy efficiency, vulnerability, fairness (i.e., human bias), and
safety frequently receive insufficient attention. This paper posits that a
comprehensive comparison of software engineers and AI-based solutions,
considering various evaluation criteria, is pivotal in fostering human-machine
collaboration, enhancing the reliability of AI-based methods, and understanding
task suitability for humans or AI. Furthermore, it facilitates the effective
implementation of cooperative work structures and human-in-the-loop processes.
This paper conducts an empirical investigation, contrasting the performance of
software engineers and AI systems, like ChatGPT, across different evaluation
metrics. The empirical study includes a case of assessing ChatGPT-generated
code versus code produced by developers and uploaded in Leetcode.
Related papers
- Lingma SWE-GPT: An Open Development-Process-Centric Language Model for Automated Software Improvement [62.94719119451089]
Lingma SWE-GPT series learns from and simulating real-world code submission activities.
Lingma SWE-GPT 72B resolves 30.20% of GitHub issues, marking a significant improvement in automatic issue resolution.
arXiv Detail & Related papers (2024-11-01T14:27:16Z) - The Role of Artificial Intelligence and Machine Learning in Software Testing [0.14896196009851972]
Artificial Intelligence (AI) and Machine Learning (ML) have significantly impacted various industries.
Software testing, a crucial part of the software development lifecycle (SDLC), ensures the quality and reliability of software products.
This paper explores the role of AI and ML in software testing by reviewing existing literature, analyzing current tools and techniques, and presenting case studies.
arXiv Detail & Related papers (2024-09-04T13:25:13Z) - Agent-Driven Automatic Software Improvement [55.2480439325792]
This research proposal aims to explore innovative solutions by focusing on the deployment of agents powered by Large Language Models (LLMs)
The iterative nature of agents, which allows for continuous learning and adaptation, can help surpass common challenges in code generation.
We aim to use the iterative feedback in these systems to further fine-tune the LLMs underlying the agents, becoming better aligned to the task of automated software improvement.
arXiv Detail & Related papers (2024-06-24T15:45:22Z) - Impact of the Availability of ChatGPT on Software Development: A Synthetic Difference in Differences Estimation using GitHub Data [49.1574468325115]
ChatGPT is an AI tool that enhances software production efficiency.
We estimate ChatGPT's effects on the number of git pushes, repositories, and unique developers per 100,000 people.
These results suggest that AI tools like ChatGPT can substantially boost developer productivity, though further analysis is needed to address potential downsides such as low quality code and privacy concerns.
arXiv Detail & Related papers (2024-06-16T19:11:15Z) - Developers' Perceptions on the Impact of ChatGPT in Software Development: A Survey [13.257222195239375]
We conducted a survey with 207 software developers to understand the impact of ChatGPT on software quality, productivity, and job satisfaction.
The study delves into developers' expectations regarding future adaptations of ChatGPT, concerns about potential job displacement, and perspectives on regulatory interventions.
arXiv Detail & Related papers (2024-05-20T17:31:16Z) - Using Machine Learning To Identify Software Weaknesses From Software
Requirement Specifications [49.1574468325115]
This research focuses on finding an efficient machine learning algorithm to identify software weaknesses from requirement specifications.
Keywords extracted using latent semantic analysis help map the CWE categories to PROMISE_exp. Naive Bayes, support vector machine (SVM), decision trees, neural network, and convolutional neural network (CNN) algorithms were tested.
arXiv Detail & Related papers (2023-08-10T13:19:10Z) - Genetic Micro-Programs for Automated Software Testing with Large Path
Coverage [0.0]
Existing software testing techniques focus on utilising search algorithms to discover input values that achieve high execution path coverage.
This paper outlines a novel genetic programming framework, where the evolved solutions are not input values, but micro-programs that can repeatedly generate input values.
We argue that our approach can be generalised such as to be applied to many different software systems, and is thus not specific to merely the particular software component on which it was trained.
arXiv Detail & Related papers (2023-02-14T18:47:21Z) - Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions [54.55334589363247]
We study whether conveying information about uncertainty enables programmers to more quickly and accurately produce code.
We find that highlighting tokens with the highest predicted likelihood of being edited leads to faster task completion and more targeted edits.
arXiv Detail & Related papers (2023-02-14T18:43:34Z) - Enabling Automated Machine Learning for Model-Driven AI Engineering [60.09869520679979]
We propose a novel approach to enable Model-Driven Software Engineering and Model-Driven AI Engineering.
In particular, we support Automated ML, thus assisting software engineers without deep AI knowledge in developing AI-intensive systems.
arXiv Detail & Related papers (2022-03-06T10:12:56Z) - Opening the Software Engineering Toolbox for the Assessment of
Trustworthy AI [17.910325223647362]
We argue for the application of software engineering and testing practices for the assessment of trustworthy AI.
We make the connection between the seven key requirements as defined by the European Commission's AI high-level expert group.
arXiv Detail & Related papers (2020-07-14T08:16:15Z) - Quality Management of Machine Learning Systems [0.0]
Artificial Intelligence (AI) has become a part of our daily lives due to major advances in Machine Learning (ML) techniques.
For business/mission-critical systems, serious concerns about reliability and maintainability of AI applications remain.
This paper presents a view of a holistic quality management framework for ML applications based on the current advances.
arXiv Detail & Related papers (2020-06-16T21:34:44Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.