Perception, performance, and detectability of conversational artificial
intelligence across 32 university courses
- URL: http://arxiv.org/abs/2305.13934v1
- Date: Sun, 7 May 2023 10:37:51 GMT
- Title: Perception, performance, and detectability of conversational artificial
intelligence across 32 university courses
- Authors: Hazem Ibrahim, Fengyuan Liu, Rohail Asim, Balaraju Battu, Sidahmed
Benabderrahmane, Bashar Alhafni, Wifag Adnan, Tuka Alhanai, Bedoor AlShebli,
Riyadh Baghdadi, Jocelyn J. B\'elanger, Elena Beretta, Kemal Celik, Moumena
Chaqfeh, Mohammed F. Daqaq, Zaynab El Bernoussi, Daryl Fougnie, Borja Garcia
de Soto, Alberto Gandolfi, Andras Gyorgy, Nizar Habash, J. Andrew Harris,
Aaron Kaufman, Lefteris Kirousis, Korhan Kocak, Kangsan Lee, Seungah S. Lee,
Samreen Malik, Michail Maniatakos, David Melcher, Azzam Mourad, Minsu Park,
Mahmoud Rasras, Alicja Reuben, Dania Zantout, Nancy W. Gleason, Kinga Makovi,
Talal Rahwan, Yasir Zaki
- Abstract summary: We compare the performance of ChatGPT against students on 32 university-level courses.
We find that ChatGPT's performance is comparable, if not superior, to that of students in many courses.
We find an emerging consensus among students to use the tool, and among educators to treat this as plagiarism.
- Score: 15.642614735026106
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The emergence of large language models has led to the development of powerful
tools such as ChatGPT that can produce text indistinguishable from
human-generated work. With the increasing accessibility of such technology,
students across the globe may utilize it to help with their school work -- a
possibility that has sparked discussions on the integrity of student
evaluations in the age of artificial intelligence (AI). To date, it is unclear
how such tools perform compared to students on university-level courses.
Further, students' perspectives regarding the use of such tools, and educators'
perspectives on treating their use as plagiarism, remain unknown. Here, we
compare the performance of ChatGPT against students on 32 university-level
courses. We also assess the degree to which its use can be detected by two
classifiers designed specifically for this purpose. Additionally, we conduct a
survey across five countries, as well as a more in-depth survey at the authors'
institution, to discern students' and educators' perceptions of ChatGPT's use.
We find that ChatGPT's performance is comparable, if not superior, to that of
students in many courses. Moreover, current AI-text classifiers cannot reliably
detect ChatGPT's use in school work, due to their propensity to classify
human-written answers as AI-generated, as well as the ease with which
AI-generated text can be edited to evade detection. Finally, we find an
emerging consensus among students to use the tool, and among educators to treat
this as plagiarism. Our findings offer insights that could guide policy
discussions addressing the integration of AI into educational frameworks.
Related papers
- How Well Can LLMs Echo Us? Evaluating AI Chatbots' Role-Play Ability with ECHO [55.25989137825992]
We introduce ECHO, an evaluative framework inspired by the Turing test.
This framework engages the acquaintances of the target individuals to distinguish between human and machine-generated responses.
We evaluate three role-playing LLMs using ECHO, with GPT-3.5 and GPT-4 serving as foundational models.
arXiv Detail & Related papers (2024-04-22T08:00:51Z) - Is ChatGPT Involved in Texts? Measure the Polish Ratio to Detect
ChatGPT-Generated Text [48.36706154871577]
We introduce a novel dataset termed HPPT (ChatGPT-polished academic abstracts)
It diverges from extant corpora by comprising pairs of human-written and ChatGPT-polished abstracts instead of purely ChatGPT-generated texts.
We also propose the "Polish Ratio" method, an innovative measure of the degree of modification made by ChatGPT compared to the original human-written text.
arXiv Detail & Related papers (2023-07-21T06:38:37Z) - Exploring User Perspectives on ChatGPT: Applications, Perceptions, and
Implications for AI-Integrated Education [40.38809129759498]
ChatGPT is most commonly used in the domains of higher education, K-12 education, and practical skills training.
On one hand, some users view it as a transformative tool capable of amplifying student self-efficacy and learning motivation.
On the other hand, there is a degree of apprehension among concerned users.
arXiv Detail & Related papers (2023-05-22T15:13:14Z) - ChatGPT in the Classroom: An Analysis of Its Strengths and Weaknesses
for Solving Undergraduate Computer Science Questions [5.962828109329824]
ChatGPT is an AI language model developed by OpenAI that can understand and generate human-like text.
There is concern that students may leverage ChatGPT to complete take-home assignments and exams and obtain favorable grades without genuinely acquiring knowledge.
arXiv Detail & Related papers (2023-04-28T17:26:32Z) - AI, write an essay for me: A large-scale comparison of human-written
versus ChatGPT-generated essays [66.36541161082856]
ChatGPT and similar generative AI models have attracted hundreds of millions of users.
This study compares human-written versus ChatGPT-generated argumentative student essays.
arXiv Detail & Related papers (2023-04-24T12:58:28Z) - Generative AI Perceptions: A Survey to Measure the Perceptions of
Faculty, Staff, and Students on Generative AI Tools in Academia [0.0]
ChatGPT is a natural language processing tool that can engage in human-like conversations.
This paper focuses on how ChatGPT is revolutionizing the realm of engineering education.
A survey was created to measure the effects of ChatGPT on students, faculty, and staff.
arXiv Detail & Related papers (2023-04-21T23:08:39Z) - To ChatGPT, or not to ChatGPT: That is the question! [78.407861566006]
This study provides a comprehensive and contemporary assessment of the most recent techniques in ChatGPT detection.
We have curated a benchmark dataset consisting of prompts from ChatGPT and humans, including diverse questions from medical, open Q&A, and finance domains.
Our evaluation results demonstrate that none of the existing methods can effectively detect ChatGPT-generated content.
arXiv Detail & Related papers (2023-04-04T03:04:28Z) - On the Educational Impact of ChatGPT: Is Artificial Intelligence Ready
to Obtain a University Degree? [0.0]
We evaluate the influence of ChatGPT on university education.
We discuss how computer science higher education should adapt to tools like ChatGPT.
arXiv Detail & Related papers (2023-03-20T14:27:37Z) - A Categorical Archive of ChatGPT Failures [47.64219291655723]
ChatGPT, developed by OpenAI, has been trained using massive amounts of data and simulates human conversation.
It has garnered significant attention due to its ability to effectively answer a broad range of human inquiries.
However, a comprehensive analysis of ChatGPT's failures is lacking, which is the focus of this study.
arXiv Detail & Related papers (2023-02-06T04:21:59Z) - ChatGPT: The End of Online Exam Integrity? [0.0]
This study evaluated the ability of ChatGPT, a recently developed artificial intelligence (AI) agent, to perform high-level cognitive tasks.
It raises concerns about the potential use of ChatGPT as a tool for academic misconduct in online exams.
arXiv Detail & Related papers (2022-12-19T08:15:16Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.