CHEAT: A Large-scale Dataset for Detecting ChatGPT-writtEn AbsTracts
- URL: http://arxiv.org/abs/2304.12008v2
- Date: Sat, 24 Feb 2024 05:52:28 GMT
- Title: CHEAT: A Large-scale Dataset for Detecting ChatGPT-writtEn AbsTracts
- Authors: Peipeng Yu, Jiahan Chen, Xuan Feng, Zhihua Xia
- Abstract summary: Malicious users could synthesize dummy academic content through ChatGPT.
We present a large-scale CHatGPT-writtEn AbsTract dataset (CHEAT) to support the development of detection algorithms.
- Score: 10.034193809833372
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The powerful ability of ChatGPT has caused widespread concern in the academic
community. Malicious users could synthesize dummy academic content through
ChatGPT, which is extremely harmful to academic rigor and originality. The need
to develop ChatGPT-written content detection algorithms call for large-scale
datasets. In this paper, we initially investigate the possible negative impact
of ChatGPT on academia,and present a large-scale CHatGPT-writtEn AbsTract
dataset (CHEAT) to support the development of detection algorithms. In
particular, the ChatGPT-written abstract dataset contains 35,304 synthetic
abstracts, with Generation, Polish, and Mix as prominent representatives. Based
on these data, we perform a thorough analysis of the existing text synthesis
detection algorithms. We show that ChatGPT-written abstracts are detectable,
while the detection difficulty increases with human involvement.Our dataset is
available in https://github.com/botianzhe/CHEAT.
Related papers
- Is ChatGPT the Future of Causal Text Mining? A Comprehensive Evaluation
and Analysis [8.031131164056347]
This study conducts comprehensive evaluations of ChatGPT's causal text mining capabilities.
We introduce a benchmark that extends beyond general English datasets.
We also provide an evaluation framework to ensure fair comparisons between ChatGPT and previous approaches.
arXiv Detail & Related papers (2024-02-22T12:19:04Z) - Exploring ChatGPT's Capabilities on Vulnerability Management [56.4403395100589]
We explore ChatGPT's capabilities on 6 tasks involving the complete vulnerability management process with a large-scale dataset containing 70,346 samples.
One notable example is ChatGPT's proficiency in tasks like generating titles for software bug reports.
Our findings reveal the difficulties encountered by ChatGPT and shed light on promising future directions.
arXiv Detail & Related papers (2023-11-11T11:01:13Z) - DEMASQ: Unmasking the ChatGPT Wordsmith [63.8746084667206]
We propose an effective ChatGPT detector named DEMASQ, which accurately identifies ChatGPT-generated content.
Our method addresses two critical factors: (i) the distinct biases in text composition observed in human- and machine-generated content and (ii) the alterations made by humans to evade previous detection methods.
arXiv Detail & Related papers (2023-11-08T21:13:05Z) - Fighting Fire with Fire: Can ChatGPT Detect AI-generated Text? [20.37071875344405]
We evaluate the zero-shot performance of ChatGPT in the task of human-written vs. AI-generated text detection.
We empirically investigate if ChatGPT is symmetrically effective in detecting AI-generated or human-written text.
arXiv Detail & Related papers (2023-08-02T17:11:37Z) - Is ChatGPT Involved in Texts? Measure the Polish Ratio to Detect
ChatGPT-Generated Text [48.36706154871577]
We introduce a novel dataset termed HPPT (ChatGPT-polished academic abstracts)
It diverges from extant corpora by comprising pairs of human-written and ChatGPT-polished abstracts instead of purely ChatGPT-generated texts.
We also propose the "Polish Ratio" method, an innovative measure of the degree of modification made by ChatGPT compared to the original human-written text.
arXiv Detail & Related papers (2023-07-21T06:38:37Z) - To ChatGPT, or not to ChatGPT: That is the question! [78.407861566006]
This study provides a comprehensive and contemporary assessment of the most recent techniques in ChatGPT detection.
We have curated a benchmark dataset consisting of prompts from ChatGPT and humans, including diverse questions from medical, open Q&A, and finance domains.
Our evaluation results demonstrate that none of the existing methods can effectively detect ChatGPT-generated content.
arXiv Detail & Related papers (2023-04-04T03:04:28Z) - Comparing Abstractive Summaries Generated by ChatGPT to Real Summaries
Through Blinded Reviewers and Text Classification Algorithms [0.8339831319589133]
ChatGPT, developed by OpenAI, is a recent addition to the family of language models.
We evaluate the performance of ChatGPT on Abstractive Summarization by the means of automated metrics and blinded human reviewers.
arXiv Detail & Related papers (2023-03-30T18:28:33Z) - Does Synthetic Data Generation of LLMs Help Clinical Text Mining? [51.205078179427645]
We investigate the potential of OpenAI's ChatGPT to aid in clinical text mining.
We propose a new training paradigm that involves generating a vast quantity of high-quality synthetic data.
Our method has resulted in significant improvements in the performance of downstream tasks.
arXiv Detail & Related papers (2023-03-08T03:56:31Z) - A Categorical Archive of ChatGPT Failures [47.64219291655723]
ChatGPT, developed by OpenAI, has been trained using massive amounts of data and simulates human conversation.
It has garnered significant attention due to its ability to effectively answer a broad range of human inquiries.
However, a comprehensive analysis of ChatGPT's failures is lacking, which is the focus of this study.
arXiv Detail & Related papers (2023-02-06T04:21:59Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.