Can AI Write Classical Chinese Poetry like Humans? An Empirical Study Inspired by Turing Test

• URL: http://arxiv.org/abs/2401.04952v1
• Date: Wed, 10 Jan 2024 06:21:47 GMT
• Title: Can AI Write Classical Chinese Poetry like Humans? An Empirical Study Inspired by Turing Test
• Abstract summary: We propose ProFTAP, a novel evaluation framework inspired by Turing test to assess AI's poetry writing capability. We find that recent large language models (LLMs) do indeed possess the ability to write classical Chinese poems nearly indistinguishable from those of humans.
• Score: 8.539465812580612
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Some argue that the essence of humanity, such as creativity and sentiment, can never be mimicked by machines. This paper casts doubt on this belief by studying a vital question: Can AI compose poetry as well as humans? To answer the question, we propose ProFTAP, a novel evaluation framework inspired by Turing test to assess AI's poetry writing capability. We apply it on current large language models (LLMs) and find that recent LLMs do indeed possess the ability to write classical Chinese poems nearly indistinguishable from those of humans. We also reveal that various open-source LLMs can outperform GPT-4 on this task.

Related papers

• Jiuge-Tuiqiao: An Interpretable Human-AI System for Classical Chinese Poetry Refinement [48.293435958731614]
  Jiuge-Tuiqiao is an interactive human-AI collaborative system for classical Chinese poetry composition.<n>Users can lock characters or lines, receive real-time prosody feedback, and obtain interpretable refinement suggestions.<n>Preliminary experiments and user feedback suggest that Jiuge-Tuiqiao improves controllability, interpretability, and user engagement.
  arXiv  Detail & Related papers  (2026-08-24T11:05:09Z)
• Do Large Language Models Perform Well on Comprehending Poetic Logic in Modern Chinese Poetry? [47.71394430680399]
  Large Language Models (LLMs) have achieved significant progress across a wide range of natural language processing (NLP) tasks.<n>Modern Chinese poetry necessitate a distinct form of reasoning for effective comprehension.<n>We propose Peony, the first benchmark specifically designed for evaluating the poetic logic of modern Chinese poetry.
  arXiv  Detail & Related papers  (2026-08-22T07:52:00Z)
• Who Wrote This Line? Evaluating the Detection of LLM-Generated Classical Chinese Poetry [45.27531384029669]
  AI-generated literary creations have raised prominent issues of creative authenticity and ethics in literary world.<n>Previous works have made significant progress in detecting AI-generated text, but it has yet to address classical Chinese poetry.<n>We introduce ChangAn, a benchmark for detecting LLM-generated classical Chinese poetry.
  arXiv  Detail & Related papers  (2026-04-11T08:52:08Z)
• Beyond Detection: Rethinking Education in the Age of AI-writing [58.49695743664001]
  As generative AI tools like ChatGPT enter classrooms, workplaces and everyday thinking, writing is at risk of becoming a formality.<n>This paper explores what is lost when we let machines write for us, drawing on cognitive psychology, educational theory and real classroom practices.
  arXiv  Detail & Related papers  (2026-03-26T11:16:01Z)
• Human-AI Interaction Traces as Blackout Poetry: Reframing AI-Supported Writing as Found-Text Creativity [12.76364003775266]
  We argue for a different lens by exploring how human-AI interaction traces might function as expressive artifacts.<n>We frame AI-generated text as found material through which writers' acts of curation and reinterpretation become inscribed atop the AI's original output.
  arXiv  Detail & Related papers  (2026-03-10T16:04:53Z)
• The author is dead, but what if they never lived? A reception experiment on Czech AI- and human-authored poetry [0.0]
  We examine the perception of AI- and human-written Czech poetry.<n>We ask if Czech native speakers are able to identify it and how they judge it.
  arXiv  Detail & Related papers  (2025-11-26T17:53:59Z)
• Benchmarking the Detection of LLMs-Generated Modern Chinese Poetry [37.86155340473244]
  It is difficult to identify whether a poem originated from humans or AI.<n>This paper proposes a novel benchmark for detecting AI-generated modern Chinese poetry.
  arXiv  Detail & Related papers  (2025-09-01T17:01:45Z)
• AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text [53.15652021126663]
  We present CREATIVITY INDEX as the first step to quantify the linguistic creativity of a text. To compute CREATIVITY INDEX efficiently, we introduce DJ SEARCH, a novel dynamic programming algorithm. Experiments reveal that the CREATIVITY INDEX of professional human authors is on average 66.2% higher than that of LLMs.
  arXiv  Detail & Related papers  (2024-10-05T18:55:01Z)
• Pron vs Prompt: Can Large Language Models already Challenge a World-Class Fiction Author at Creative Text Writing? [0.8999666725996975]
  Large Language Models (LLMs) outperform average humans in a wide range of language-related tasks. We have carried out a contest between Patricio Pron and GPT-4, in the spirit of AI-human duels such as DeepBlue vs Kasparov and AlphaGo vs Lee Sidol. The results indicate that LLMs are still far from challenging a top human creative writer.
  arXiv  Detail & Related papers  (2024-07-01T09:28:58Z)
• Evaluating Diversity in Automatic Poetry Generation [25.53206868552533]
  We evaluate the diversity of automatically generated poetry along structural, lexical, semantic and stylistic dimensions. We find that current automatic poetry systems are considerably underdiverse along multiple dimensions. Our identified limitations may serve as the basis for more genuinely diverse future poetry generation models.
  arXiv  Detail & Related papers  (2024-06-21T16:03:21Z)
• How Well Can LLMs Echo Us? Evaluating AI Chatbots' Role-Play Ability with ECHO [55.25989137825992]
  We introduce ECHO, an evaluative framework inspired by the Turing test. This framework engages the acquaintances of the target individuals to distinguish between human and machine-generated responses. We evaluate three role-playing LLMs using ECHO, with GPT-3.5 and GPT-4 serving as foundational models.
  arXiv  Detail & Related papers  (2024-04-22T08:00:51Z)
• PaperCard for Reporting Machine Assistance in Academic Writing [48.33722012818687]
  ChatGPT, a question-answering system released by OpenAI in November 2022, has demonstrated a range of capabilities that could be utilised in producing academic papers. This raises critical questions surrounding the concept of authorship in academia. We propose a framework we name "PaperCard", a documentation for human authors to transparently declare the use of AI in their writing process.
  arXiv  Detail & Related papers  (2023-10-07T14:28:04Z)
• Borges and AI [14.879252696060302]
  Proponents and opponents grasp AI through the imagery popularised by science fiction. Will the machine become sentient and rebel against its creators? This exercise leads to a new perspective that illuminates the relation between language modelling and artificial intelligence.
  arXiv  Detail & Related papers  (2023-09-27T16:15:34Z)
• Understanding Natural Language Understanding Systems. A Critical Analysis [91.81211519327161]
  The development of machines that guillemotlefttalk like usguillemotright, also known as Natural Language Understanding (NLU) systems, is the Holy Grail of Artificial Intelligence (AI) But never has the trust that we can build guillemotlefttalking machinesguillemotright been stronger than the one engendered by the last generation of NLU systems. Are we at the dawn of a new era, in which the Grail is finally closer to us?
  arXiv  Detail & Related papers  (2023-03-01T08:32:55Z)
• Generation of Chinese classical poetry based on pre-trained model [1.6114012813668934]
  This paper mainly tries to use BART and other pre training models to generate metrical poetry text. It developed a set of AI poetry Turing problems, it was reviewed by a group of poets and poetry writing researchers. The model of poetry generation studied by the author generalizes works that cannot be distinguished from those of advanced scholars.
  arXiv  Detail & Related papers  (2022-11-04T16:05:31Z)
• BACON: Deep-Learning Powered AI for Poetry Generation with Author Linguistic Style Transfer [91.3755431537592]
  This paper describes BACON, a prototype of an automatic poetry generator with author linguistic style transfer. It combines concepts and techniques from finite state machinery, probabilistic models, artificial neural networks and deep learning, to write original poetry with rich aesthetic-qualities in the style of any given author.
  arXiv  Detail & Related papers  (2021-12-14T00:08:36Z)
• CCPM: A Chinese Classical Poetry Matching Dataset [50.90794811956129]
  We propose a novel task to assess a model's semantic understanding of poetry by poem matching. This task requires the model to select one line of Chinese classical poetry among four candidates according to the modern Chinese translation of a line of poetry. To construct this dataset, we first obtain a set of parallel data of Chinese classical poetry and modern Chinese translation.
  arXiv  Detail & Related papers  (2021-06-03T16:49:03Z)
• Artificial Intelligence versus Maya Angelou: Experimental evidence that people cannot differentiate AI-generated from human-written poetry [0.0]
  Natural language generation algorithms (NLG) have spurred much public attention and debate. One reason lies in the algorithms' purported ability to generate human-like text across various domains. We conducted two experiments assessing behavioral reactions to the state-of-the-art Natural Language Generation algorithm GPT-2. We discuss what these results convey about the performance of NLG algorithms to produce human-like text and propose methodologies to study such learning algorithms in human-agent experimental settings.
  arXiv  Detail & Related papers  (2020-05-20T11:52:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.