Artificial Intelligence versus Maya Angelou: Experimental evidence that people cannot differentiate AI-generated from human-written poetry

• URL: http://arxiv.org/abs/2005.09980v2
• Date: Tue, 8 Sep 2020 19:31:44 GMT
• Title: Artificial Intelligence versus Maya Angelou: Experimental evidence that people cannot differentiate AI-generated from human-written poetry
• Authors: Nils K\"obis, Luca Mossink
• Abstract summary: Natural language generation algorithms (NLG) have spurred much public attention and debate. One reason lies in the algorithms' purported ability to generate human-like text across various domains. We conducted two experiments assessing behavioral reactions to the state-of-the-art Natural Language Generation algorithm GPT-2. We discuss what these results convey about the performance of NLG algorithms to produce human-like text and propose methodologies to study such learning algorithms in human-agent experimental settings.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The release of openly available, robust natural language generation algorithms (NLG) has spurred much public attention and debate. One reason lies in the algorithms' purported ability to generate human-like text across various domains. Empirical evidence using incentivized tasks to assess whether people (a) can distinguish and (b) prefer algorithm-generated versus human-written text is lacking. We conducted two experiments assessing behavioral reactions to the state-of-the-art Natural Language Generation algorithm GPT-2 (Ntotal = 830). Using the identical starting lines of human poems, GPT-2 produced samples of poems. From these samples, either a random poem was chosen (Human-out-of-the-loop) or the best one was selected (Human-in-the-loop) and in turn matched with a human-written poem. In a new incentivized version of the Turing Test, participants failed to reliably detect the algorithmically-generated poems in the Human-in-the-loop treatment, yet succeeded in the Human-out-of-the-loop treatment. Further, people reveal a slight aversion to algorithm-generated poetry, independent on whether participants were informed about the algorithmic origin of the poem (Transparency) or not (Opacity). We discuss what these results convey about the performance of NLG algorithms to produce human-like text and propose methodologies to study such learning algorithms in human-agent experimental settings.

Related papers

• Human Bias in the Face of AI: The Role of Human Judgement in AI Generated Text Evaluation [48.70176791365903]
  This study explores how bias shapes the perception of AI versus human generated content. We investigated how human raters respond to labeled and unlabeled content.
  arXiv  Detail & Related papers  (2024-09-29T04:31:45Z)
• Sentiment analysis and random forest to classify LLM versus human source applied to Scientific Texts [0.0]
  It is proposed a new methodology to classify texts coming from an automatic text production engine or a human. Using four different sentiment lexicons, a number of new features where produced, and then fed to a machine learning random forest methodology, to train such a model. Results seem very convincing that this may be a promising research line to detect fraud, in such environments where human are supposed to be the source of texts.
  arXiv  Detail & Related papers  (2024-04-05T16:14:36Z)
• Forging the Forger: An Attempt to Improve Authorship Verification via Data Augmentation [52.72682366640554]
  Authorship Verification (AV) is a text classification task concerned with inferring whether a candidate text has been written by one specific author or by someone else. It has been shown that many AV systems are vulnerable to adversarial attacks, where a malicious author actively tries to fool the classifier by either concealing their writing style, or by imitating the style of another author.
  arXiv  Detail & Related papers  (2024-03-17T16:36:26Z)
• Evaluating the Efficacy of Hybrid Deep Learning Models in Distinguishing AI-Generated Text [0.0]
  My research investigates the use of cutting-edge hybrid deep learning models to accurately differentiate between AI-generated text and human writing. I applied a robust methodology, utilising a carefully selected dataset comprising AI and human texts from various sources, each tagged with instructions.
  arXiv  Detail & Related papers  (2023-11-27T06:26:53Z)
• Learning interactions to boost human creativity with bandits and GPT-4 [10.817205577415434]
  We employ a psychological task that demonstrates limits on human creativity, namely semantic feature generation. In experiments with humans and with a language AI (GPT-4) we contrast behavior in the standard task versus a variant in which participants can ask for algorithmically-generated hints. Humans and the AI show similar benefits from hints, and remarkably, bandits learning from AI responses prefer the same prompting strategy as those learning from human behavior.
  arXiv  Detail & Related papers  (2023-11-16T16:53:17Z)
• DEMASQ: Unmasking the ChatGPT Wordsmith [63.8746084667206]
  We propose an effective ChatGPT detector named DEMASQ, which accurately identifies ChatGPT-generated content. Our method addresses two critical factors: (i) the distinct biases in text composition observed in human- and machine-generated content and (ii) the alterations made by humans to evade previous detection methods.
  arXiv  Detail & Related papers  (2023-11-08T21:13:05Z)
• Towards Automatic Boundary Detection for Human-AI Collaborative Hybrid Essay in Education [10.606131520965604]
  This study investigates AI content detection in a rarely explored yet realistic setting. We first formalized the detection task as identifying the transition points between human-written content and AI-generated content. We then proposed a two-step approach where we separated AI-generated content from human-written content during the encoder training process.
  arXiv  Detail & Related papers  (2023-07-23T08:47:51Z)
• MAGE: Machine-generated Text Detection in the Wild [82.70561073277801]
  Large language models (LLMs) have achieved human-level text generation, emphasizing the need for effective AI-generated text detection. We build a comprehensive testbed by gathering texts from diverse human writings and texts generated by different LLMs. Despite challenges, the top-performing detector can identify 86.54% out-of-domain texts generated by a new LLM, indicating the feasibility for application scenarios.
  arXiv  Detail & Related papers  (2023-05-22T17:13:29Z)
• BACON: Deep-Learning Powered AI for Poetry Generation with Author Linguistic Style Transfer [91.3755431537592]
  This paper describes BACON, a prototype of an automatic poetry generator with author linguistic style transfer. It combines concepts and techniques from finite state machinery, probabilistic models, artificial neural networks and deep learning, to write original poetry with rich aesthetic-qualities in the style of any given author.
  arXiv  Detail & Related papers  (2021-12-14T00:08:36Z)
• A pattern recognition approach for distinguishing between prose and poetry [0.8971132850029492]
  We propose an automated method to distinguish between poetry and prose based solely on aural and rhythmic properties. The classification of the considered texts using the set of features extracted resulted in a best accuracy of 0.78, obtained with a neural network.
  arXiv  Detail & Related papers  (2021-07-18T18:44:17Z)
• Tortured phrases: A dubious writing style emerging in science. Evidence of critical issues affecting established journals [69.76097138157816]
  Probabilistic text generators have been used to produce fake scientific papers for more than a decade. Complex AI-powered generation techniques produce texts indistinguishable from that of humans. Some websites offer to rewrite texts for free, generating gobbledegook full of tortured phrases.
  arXiv  Detail & Related papers  (2021-07-12T20:47:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.