Classification of Human- and AI-Generated Texts: Investigating Features for ChatGPT

• URL: http://arxiv.org/abs/2308.05341v1
• Date: Thu, 10 Aug 2023 05:09:42 GMT
• Title: Classification of Human- and AI-Generated Texts: Investigating Features for ChatGPT
• Authors: Lorenz Mindner, Tim Schlippe, Kristina Schaaff
• Abstract summary: We explore traditional and new features to detect text generated by AI from scratch and text rephrased by AI. For our experiments, we produced a new text corpus covering 10 school topics. Our best systems for classifying basic and advanced human-generated/AI-rephrased texts have F1-scores of more than 78%.
• Score: 0.25782420501870296
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, generative AIs like ChatGPT have become available to the wide public. These tools can for instance be used by students to generate essays or whole theses. But how does a teacher know whether a text is written by a student or an AI? In our work, we explore traditional and new features to (1) detect text generated by AI from scratch and (2) text rephrased by AI. Since we found that classification is more difficult when the AI has been instructed to create the text in a way that a human would not recognize that it was generated by an AI, we also investigate this more advanced case. For our experiments, we produced a new text corpus covering 10 school topics. Our best systems to classify basic and advanced human-generated/AI-generated texts have F1-scores of over 96%. Our best systems for classifying basic and advanced human-generated/AI-rephrased texts have F1-scores of more than 78%. The systems use a combination of perplexity, semantic, list lookup, error-based, readability, AI feedback, and text vector features. Our results show that the new features substantially help to improve the performance of many classifiers. Our best basic text rephrasing detection system even outperforms GPTZero by 183.8% relative in F1-score.

Related papers

• Human Bias in the Face of AI: The Role of Human Judgement in AI Generated Text Evaluation [48.70176791365903]
  This study explores how bias shapes the perception of AI versus human generated content. We investigated how human raters respond to labeled and unlabeled content.
  arXiv  Detail & Related papers  (2024-09-29T04:31:45Z)
• Raidar: geneRative AI Detection viA Rewriting [42.477151044325595]
  Large language models (LLMs) are more likely to modify human-written text than AI-generated text when tasked with rewriting. We introduce a method to detect AI-generated content by prompting LLMs to rewrite text and calculating the editing distance of the output. Our results illustrate the unique imprint of machine-generated text through the lens of the machines themselves.
  arXiv  Detail & Related papers  (2024-01-23T18:57:53Z)
• AI Content Self-Detection for Transformer-based Large Language Models [0.0]
  This paper introduces the idea of direct origin detection and evaluates whether generative AI systems can recognize their output and distinguish it from human-written texts. Google's Bard model exhibits the largest capability of self-detection with an accuracy of 94%, followed by OpenAI's ChatGPT with 83%.
  arXiv  Detail & Related papers  (2023-12-28T10:08:57Z)
• Classification of Human- and AI-Generated Texts for English, French, German, and Spanish [0.138120109831448]
  We analyze features to classify human- and AI-generated text for English, French, German and Spanish. For the detection of AI-generated text, the combination of all proposed features performs best. For the detection of AI-rephrased text, the systems with all features outperform systems with other features in many cases.
  arXiv  Detail & Related papers  (2023-12-08T07:42:06Z)
• UltraFeedback: Boosting Language Models with Scaled AI Feedback [99.4633351133207]
  We present textscUltraFeedback, a large-scale, high-quality, and diversified AI feedback dataset. Our work validates the effectiveness of scaled AI feedback data in constructing strong open-source chat language models.
  arXiv  Detail & Related papers  (2023-10-02T17:40:01Z)
• Distinguishing Human Generated Text From ChatGPT Generated Text Using Machine Learning [0.251657752676152]
  This paper presents a machine learning-based solution that can identify the ChatGPT delivered text from the human written text. We have tested the proposed model on a Kaggle dataset consisting of 10,000 texts out of which 5,204 texts were written by humans and collected from news and social media. On the corpus generated by GPT-3.5, the proposed algorithm presents an accuracy of 77%.
  arXiv  Detail & Related papers  (2023-05-26T09:27:43Z)
• AI, write an essay for me: A large-scale comparison of human-written versus ChatGPT-generated essays [66.36541161082856]
  ChatGPT and similar generative AI models have attracted hundreds of millions of users. This study compares human-written versus ChatGPT-generated argumentative student essays.
  arXiv  Detail & Related papers  (2023-04-24T12:58:28Z)
• Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense [56.077252790310176]
  We present a paraphrase generation model (DIPPER) that can paraphrase paragraphs, condition on surrounding context, and control lexical diversity and content reordering. Using DIPPER to paraphrase text generated by three large language models (including GPT3.5-davinci-003) successfully evades several detectors, including watermarking. We introduce a simple defense that relies on retrieving semantically-similar generations and must be maintained by a language model API provider.
  arXiv  Detail & Related papers  (2023-03-23T16:29:27Z)
• A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need? [112.12974778019304]
  generative AI (AIGC, a.k.a AI-generated content) has made headlines everywhere because of its ability to analyze and create text, images, and beyond. In the era of AI transitioning from pure analysis to creation, it is worth noting that ChatGPT, with its most recent language model GPT-4, is just a tool out of numerous AIGC tasks. This work focuses on the technological development of various AIGC tasks based on their output type, including text, images, videos, 3D content, etc.
  arXiv  Detail & Related papers  (2023-03-21T10:09:47Z)
• Exploring AI-Generated Text in Student Writing: How Does AI Help? [0.0]
  It remains unclear to what extent AI-generated text in these students' writing might lead to higher-quality writing. We explored 23 Hong Kong secondary school students' attempts to write stories comprising their own words and AI-generated text.
  arXiv  Detail & Related papers  (2023-03-10T14:36:47Z)
• An Exploration of Post-Editing Effectiveness in Text Summarization [58.99765574294715]
  "Post-editing" AI-generated text reduces human workload and improves the quality of AI output. We compare post-editing provided summaries with manual summarization for summary quality, human efficiency, and user experience. This study sheds valuable insights on when post-editing is useful for text summarization.
  arXiv  Detail & Related papers  (2022-06-13T18:00:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.