Fighting Fire with Fire: Can ChatGPT Detect AI-generated Text?

• URL: http://arxiv.org/abs/2308.01284v2
• Date: Thu, 17 Aug 2023 22:34:38 GMT
• Title: Fighting Fire with Fire: Can ChatGPT Detect AI-generated Text?
• Authors: Amrita Bhattacharjee, Huan Liu
• Abstract summary: We evaluate the zero-shot performance of ChatGPT in the task of human-written vs. AI-generated text detection. We empirically investigate if ChatGPT is symmetrically effective in detecting AI-generated or human-written text.
• Score: 20.37071875344405
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) such as ChatGPT are increasingly being used for various use cases, including text content generation at scale. Although detection methods for such AI-generated text exist already, we investigate ChatGPT's performance as a detector on such AI-generated text, inspired by works that use ChatGPT as a data labeler or annotator. We evaluate the zero-shot performance of ChatGPT in the task of human-written vs. AI-generated text detection, and perform experiments on publicly available datasets. We empirically investigate if ChatGPT is symmetrically effective in detecting AI-generated or human-written text. Our findings provide insight on how ChatGPT and similar LLMs may be leveraged in automated detection pipelines by simply focusing on solving a specific aspect of the problem and deriving the rest from that solution. All code and data is available at https://github.com/AmritaBh/ChatGPT-as-Detector.

Related papers

• Spotting AI's Touch: Identifying LLM-Paraphrased Spans in Text [61.22649031769564]
  We propose a novel framework, paraphrased text span detection (PTD) PTD aims to identify paraphrased text spans within a text. We construct a dedicated dataset, PASTED, for paraphrased text span detection.
  arXiv  Detail & Related papers  (2024-05-21T11:22:27Z)
• Exploring ChatGPT's Capabilities on Vulnerability Management [56.4403395100589]
  We explore ChatGPT's capabilities on 6 tasks involving the complete vulnerability management process with a large-scale dataset containing 70,346 samples. One notable example is ChatGPT's proficiency in tasks like generating titles for software bug reports. Our findings reveal the difficulties encountered by ChatGPT and shed light on promising future directions.
  arXiv  Detail & Related papers  (2023-11-11T11:01:13Z)
• DetectGPT-SC: Improving Detection of Text Generated by Large Language Models through Self-Consistency with Masked Predictions [13.077729125193434]
  Existing detectors are built on the assumption that there is a distribution gap between human-generated and AI-generated texts. We find that large language models such as ChatGPT exhibit strong self-consistency in text generation and continuation. We propose a new method for AI-generated texts detection based on self-consistency with masked predictions.
  arXiv  Detail & Related papers  (2023-10-23T01:23:10Z)
• Is ChatGPT Involved in Texts? Measure the Polish Ratio to Detect ChatGPT-Generated Text [48.36706154871577]
  We introduce a novel dataset termed HPPT (ChatGPT-polished academic abstracts) It diverges from extant corpora by comprising pairs of human-written and ChatGPT-polished abstracts instead of purely ChatGPT-generated texts. We also propose the "Polish Ratio" method, an innovative measure of the degree of modification made by ChatGPT compared to the original human-written text.
  arXiv  Detail & Related papers  (2023-07-21T06:38:37Z)
• Is this Snippet Written by ChatGPT? An Empirical Study with a CodeBERT-Based Classifier [13.613735709997911]
  This paper presents an empirical study to investigate the feasibility of automated identification of AI-generated code snippets. We propose a novel approach called GPTSniffer, which builds on top of CodeBERT to detect source code written by AI. The results show that GPTSniffer can accurately classify whether code is human-written or AI-generated, and outperforms two baselines.
  arXiv  Detail & Related papers  (2023-07-18T16:01:15Z)
• Towards a Robust Detection of Language Model Generated Text: Is ChatGPT that Easy to Detect? [0.0]
  This paper proposes a methodology for developing and evaluating ChatGPT detectors for French text. The proposed method involves translating an English dataset into French and training a classifier on the translated data. Results show that the detectors can effectively detect ChatGPT-generated text, with a degree of robustness against basic attack techniques in in-domain settings.
  arXiv  Detail & Related papers  (2023-06-09T13:03:53Z)
• CHEAT: A Large-scale Dataset for Detecting ChatGPT-writtEn AbsTracts [10.034193809833372]
  Malicious users could synthesize dummy academic content through ChatGPT. We present a large-scale CHatGPT-writtEn AbsTract dataset (CHEAT) to support the development of detection algorithms.
  arXiv  Detail & Related papers  (2023-04-24T11:19:33Z)
• Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks [9.740764281808588]
  ChatGPT has the potential to reproduce human-generated label annotations in social computing tasks. We relabel five datasets covering stance detection (2x), sentiment analysis, hate speech, and bot detection. Our results highlight that ChatGPT does have the potential to handle these data annotation tasks, although a number of challenges remain.
  arXiv  Detail & Related papers  (2023-04-20T08:08:12Z)
• To ChatGPT, or not to ChatGPT: That is the question! [78.407861566006]
  This study provides a comprehensive and contemporary assessment of the most recent techniques in ChatGPT detection. We have curated a benchmark dataset consisting of prompts from ChatGPT and humans, including diverse questions from medical, open Q&A, and finance domains. Our evaluation results demonstrate that none of the existing methods can effectively detect ChatGPT-generated content.
  arXiv  Detail & Related papers  (2023-04-04T03:04:28Z)
• Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense [56.077252790310176]
  We present a paraphrase generation model (DIPPER) that can paraphrase paragraphs, condition on surrounding context, and control lexical diversity and content reordering. Using DIPPER to paraphrase text generated by three large language models (including GPT3.5-davinci-003) successfully evades several detectors, including watermarking. We introduce a simple defense that relies on retrieving semantically-similar generations and must be maintained by a language model API provider.
  arXiv  Detail & Related papers  (2023-03-23T16:29:27Z)
• Can AI-Generated Text be Reliably Detected? [54.670136179857344]
  Unregulated use of LLMs can potentially lead to malicious consequences such as plagiarism, generating fake news, spamming, etc. Recent works attempt to tackle this problem either using certain model signatures present in the generated text outputs or by applying watermarking techniques. In this paper, we show that these detectors are not reliable in practical scenarios.
  arXiv  Detail & Related papers  (2023-03-17T17:53:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.