Can AI Write Classical Chinese Poetry like Humans? An Empirical Study Inspired by Turing Test

• URL: http://arxiv.org/abs/2401.04952v1
• Date: Wed, 10 Jan 2024 06:21:47 GMT
• Title: Can AI Write Classical Chinese Poetry like Humans? An Empirical Study Inspired by Turing Test
• Authors: Zekun Deng, Hao Yang, Jun Wang
• Abstract summary: We propose ProFTAP, a novel evaluation framework inspired by Turing test to assess AI's poetry writing capability. We find that recent large language models (LLMs) do indeed possess the ability to write classical Chinese poems nearly indistinguishable from those of humans.
• Score: 8.539465812580612
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Some argue that the essence of humanity, such as creativity and sentiment, can never be mimicked by machines. This paper casts doubt on this belief by studying a vital question: Can AI compose poetry as well as humans? To answer the question, we propose ProFTAP, a novel evaluation framework inspired by Turing test to assess AI's poetry writing capability. We apply it on current large language models (LLMs) and find that recent LLMs do indeed possess the ability to write classical Chinese poems nearly indistinguishable from those of humans. We also reveal that various open-source LLMs can outperform GPT-4 on this task.

Related papers

• AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text [53.15652021126663]
  We present CREATIVITY INDEX as the first step to quantify the linguistic creativity of a text. To compute CREATIVITY INDEX efficiently, we introduce DJ SEARCH, a novel dynamic programming algorithm. Experiments reveal that the CREATIVITY INDEX of professional human authors is on average 66.2% higher than that of LLMs.
  arXiv  Detail & Related papers  (2024-10-05T18:55:01Z)
• Pron vs Prompt: Can Large Language Models already Challenge a World-Class Fiction Author at Creative Text Writing? [0.8999666725996975]
  Large Language Models (LLMs) outperform average humans in a wide range of language-related tasks. We have carried out a contest between Patricio Pron and GPT-4, in the spirit of AI-human duels such as DeepBlue vs Kasparov and AlphaGo vs Lee Sidol. The results indicate that LLMs are still far from challenging a top human creative writer.
  arXiv  Detail & Related papers  (2024-07-01T09:28:58Z)
• Evaluating Diversity in Automatic Poetry Generation [25.53206868552533]
  We evaluate the diversity of automatically generated poetry along structural, lexical, semantic and stylistic dimensions. We find that current automatic poetry systems are considerably underdiverse along multiple dimensions. Our identified limitations may serve as the basis for more genuinely diverse future poetry generation models.
  arXiv  Detail & Related papers  (2024-06-21T16:03:21Z)
• How Well Can LLMs Echo Us? Evaluating AI Chatbots' Role-Play Ability with ECHO [55.25989137825992]
  We introduce ECHO, an evaluative framework inspired by the Turing test. This framework engages the acquaintances of the target individuals to distinguish between human and machine-generated responses. We evaluate three role-playing LLMs using ECHO, with GPT-3.5 and GPT-4 serving as foundational models.
  arXiv  Detail & Related papers  (2024-04-22T08:00:51Z)
• PaperCard for Reporting Machine Assistance in Academic Writing [48.33722012818687]
  ChatGPT, a question-answering system released by OpenAI in November 2022, has demonstrated a range of capabilities that could be utilised in producing academic papers. This raises critical questions surrounding the concept of authorship in academia. We propose a framework we name "PaperCard", a documentation for human authors to transparently declare the use of AI in their writing process.
  arXiv  Detail & Related papers  (2023-10-07T14:28:04Z)
• Borges and AI [14.879252696060302]
  Proponents and opponents grasp AI through the imagery popularised by science fiction. Will the machine become sentient and rebel against its creators? This exercise leads to a new perspective that illuminates the relation between language modelling and artificial intelligence.
  arXiv  Detail & Related papers  (2023-09-27T16:15:34Z)
• Understanding Natural Language Understanding Systems. A Critical Analysis [91.81211519327161]
  The development of machines that guillemotlefttalk like usguillemotright, also known as Natural Language Understanding (NLU) systems, is the Holy Grail of Artificial Intelligence (AI) But never has the trust that we can build guillemotlefttalking machinesguillemotright been stronger than the one engendered by the last generation of NLU systems. Are we at the dawn of a new era, in which the Grail is finally closer to us?
  arXiv  Detail & Related papers  (2023-03-01T08:32:55Z)
• Generation of Chinese classical poetry based on pre-trained model [1.6114012813668934]
  This paper mainly tries to use BART and other pre training models to generate metrical poetry text. It developed a set of AI poetry Turing problems, it was reviewed by a group of poets and poetry writing researchers. The model of poetry generation studied by the author generalizes works that cannot be distinguished from those of advanced scholars.
  arXiv  Detail & Related papers  (2022-11-04T16:05:31Z)
• BACON: Deep-Learning Powered AI for Poetry Generation with Author Linguistic Style Transfer [91.3755431537592]
  This paper describes BACON, a prototype of an automatic poetry generator with author linguistic style transfer. It combines concepts and techniques from finite state machinery, probabilistic models, artificial neural networks and deep learning, to write original poetry with rich aesthetic-qualities in the style of any given author.
  arXiv  Detail & Related papers  (2021-12-14T00:08:36Z)
• CCPM: A Chinese Classical Poetry Matching Dataset [50.90794811956129]
  We propose a novel task to assess a model's semantic understanding of poetry by poem matching. This task requires the model to select one line of Chinese classical poetry among four candidates according to the modern Chinese translation of a line of poetry. To construct this dataset, we first obtain a set of parallel data of Chinese classical poetry and modern Chinese translation.
  arXiv  Detail & Related papers  (2021-06-03T16:49:03Z)
• Artificial Intelligence versus Maya Angelou: Experimental evidence that people cannot differentiate AI-generated from human-written poetry [0.0]
  Natural language generation algorithms (NLG) have spurred much public attention and debate. One reason lies in the algorithms' purported ability to generate human-like text across various domains. We conducted two experiments assessing behavioral reactions to the state-of-the-art Natural Language Generation algorithm GPT-2. We discuss what these results convey about the performance of NLG algorithms to produce human-like text and propose methodologies to study such learning algorithms in human-agent experimental settings.
  arXiv  Detail & Related papers  (2020-05-20T11:52:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.