Comparing Software Developers with ChatGPT: An Empirical Investigation
- URL: http://arxiv.org/abs/2305.11837v2
- Date: Thu, 25 May 2023 14:58:39 GMT
- Title: Comparing Software Developers with ChatGPT: An Empirical Investigation
- Authors: Nathalia Nascimento and Paulo Alencar and Donald Cowan
- Abstract summary: This paper conducts an empirical investigation, contrasting the performance of software engineers and AI systems, like ChatGPT, across different evaluation metrics.
The paper posits that a comprehensive comparison of software engineers and AI-based solutions, considering various evaluation criteria, is pivotal in fostering human-machine collaboration.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The advent of automation in particular Software Engineering (SE) tasks has
transitioned from theory to reality. Numerous scholarly articles have
documented the successful application of Artificial Intelligence to address
issues in areas such as project management, modeling, testing, and development.
A recent innovation is the introduction of ChatGPT, an ML-infused chatbot,
touted as a resource proficient in generating programming codes and formulating
software testing strategies for developers and testers respectively. Although
there is speculation that AI-based computation can increase productivity and
even substitute software engineers in software development, there is currently
a lack of empirical evidence to verify this. Moreover, despite the primary
focus on enhancing the accuracy of AI systems, non-functional requirements
including energy efficiency, vulnerability, fairness (i.e., human bias), and
safety frequently receive insufficient attention. This paper posits that a
comprehensive comparison of software engineers and AI-based solutions,
considering various evaluation criteria, is pivotal in fostering human-machine
collaboration, enhancing the reliability of AI-based methods, and understanding
task suitability for humans or AI. Furthermore, it facilitates the effective
implementation of cooperative work structures and human-in-the-loop processes.
This paper conducts an empirical investigation, contrasting the performance of
software engineers and AI systems, like ChatGPT, across different evaluation
metrics. The empirical study includes a case of assessing ChatGPT-generated
code versus code produced by developers and uploaded in Leetcode.
Related papers
- Agent-Driven Automatic Software Improvement [55.2480439325792]
This research proposal aims to explore innovative solutions by focusing on the deployment of agents powered by Large Language Models (LLMs)
The iterative nature of agents, which allows for continuous learning and adaptation, can help surpass common challenges in code generation.
We aim to use the iterative feedback in these systems to further fine-tune the LLMs underlying the agents, becoming better aligned to the task of automated software improvement.
arXiv Detail & Related papers (2024-06-24T15:45:22Z) - Impact of the Availability of ChatGPT on Software Development: A Synthetic Difference in Differences Estimation using GitHub Data [49.1574468325115]
ChatGPT is an AI tool that enhances software production efficiency.
We estimate ChatGPT's effects on the number of git pushes, repositories, and unique developers per 100,000 people.
These results suggest that AI tools like ChatGPT can substantially boost developer productivity, though further analysis is needed to address potential downsides such as low quality code and privacy concerns.
arXiv Detail & Related papers (2024-06-16T19:11:15Z) - Developers' Perceptions on the Impact of ChatGPT in Software Development: A Survey [13.257222195239375]
We conducted a survey with 207 software developers to understand the impact of ChatGPT on software quality, productivity, and job satisfaction.
The study delves into developers' expectations regarding future adaptations of ChatGPT, concerns about potential job displacement, and perspectives on regulatory interventions.
arXiv Detail & Related papers (2024-05-20T17:31:16Z) - Using Machine Learning To Identify Software Weaknesses From Software
Requirement Specifications [49.1574468325115]
This research focuses on finding an efficient machine learning algorithm to identify software weaknesses from requirement specifications.
Keywords extracted using latent semantic analysis help map the CWE categories to PROMISE_exp. Naive Bayes, support vector machine (SVM), decision trees, neural network, and convolutional neural network (CNN) algorithms were tested.
arXiv Detail & Related papers (2023-08-10T13:19:10Z) - Genetic Micro-Programs for Automated Software Testing with Large Path
Coverage [0.0]
Existing software testing techniques focus on utilising search algorithms to discover input values that achieve high execution path coverage.
This paper outlines a novel genetic programming framework, where the evolved solutions are not input values, but micro-programs that can repeatedly generate input values.
We argue that our approach can be generalised such as to be applied to many different software systems, and is thus not specific to merely the particular software component on which it was trained.
arXiv Detail & Related papers (2023-02-14T18:47:21Z) - Enabling Automated Machine Learning for Model-Driven AI Engineering [60.09869520679979]
We propose a novel approach to enable Model-Driven Software Engineering and Model-Driven AI Engineering.
In particular, we support Automated ML, thus assisting software engineers without deep AI knowledge in developing AI-intensive systems.
arXiv Detail & Related papers (2022-03-06T10:12:56Z) - Artificial Intelligence in Software Testing : Impact, Problems,
Challenges and Prospect [0.0]
The study aims to recognize and explain some of the biggest challenges software testers face while applying AI to testing.
The paper also proposes some key contributions of AI in the future to the domain of software testing.
arXiv Detail & Related papers (2022-01-14T10:21:51Z) - Empowered and Embedded: Ethics and Agile Processes [60.63670249088117]
We argue that ethical considerations need to be embedded into the (agile) software development process.
We put emphasis on the possibility to implement ethical deliberations in already existing and well established agile software development processes.
arXiv Detail & Related papers (2021-07-15T11:14:03Z) - Opening the Software Engineering Toolbox for the Assessment of
Trustworthy AI [17.910325223647362]
We argue for the application of software engineering and testing practices for the assessment of trustworthy AI.
We make the connection between the seven key requirements as defined by the European Commission's AI high-level expert group.
arXiv Detail & Related papers (2020-07-14T08:16:15Z) - Technology Readiness Levels for AI & ML [79.22051549519989]
Development of machine learning systems can be executed easily with modern tools, but the process is typically rushed and means-to-an-end.
Engineering systems follow well-defined processes and testing standards to streamline development for high-quality, reliable results.
We propose a proven systems engineering approach for machine learning development and deployment.
arXiv Detail & Related papers (2020-06-21T17:14:34Z) - Quality Management of Machine Learning Systems [0.0]
Artificial Intelligence (AI) has become a part of our daily lives due to major advances in Machine Learning (ML) techniques.
For business/mission-critical systems, serious concerns about reliability and maintainability of AI applications remain.
This paper presents a view of a holistic quality management framework for ML applications based on the current advances.
arXiv Detail & Related papers (2020-06-16T21:34:44Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.