Benchmarking the Detection of LLMs-Generated Modern Chinese Poetry

• URL: http://arxiv.org/abs/2509.01620v1
• Date: Mon, 01 Sep 2025 17:01:45 GMT
• Title: Benchmarking the Detection of LLMs-Generated Modern Chinese Poetry
• Authors: Shanshan Wang, Junchao Wu, Fengying Ye, Jingming Yao, Lidia S. Chao, Derek F. Wong,
• Abstract summary: It is difficult to identify whether a poem originated from humans or AI.<n>This paper proposes a novel benchmark for detecting AI-generated modern Chinese poetry.
• Score: 37.86155340473244
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid development of advanced large language models (LLMs) has made AI-generated text indistinguishable from human-written text. Previous work on detecting AI-generated text has made effective progress, but has not involved modern Chinese poetry. Due to the distinctive characteristics of modern Chinese poetry, it is difficult to identify whether a poem originated from humans or AI. The proliferation of AI-generated modern Chinese poetry has significantly disrupted the poetry ecosystem. Based on the urgency of identifying AI-generated poetry in the real Chinese world, this paper proposes a novel benchmark for detecting LLMs-generated modern Chinese poetry. We first construct a high-quality dataset, which includes both 800 poems written by six professional poets and 41,600 poems generated by four mainstream LLMs. Subsequently, we conduct systematic performance assessments of six detectors on this dataset. Experimental results demonstrate that current detectors cannot be used as reliable tools to detect modern Chinese poems generated by LLMs. The most difficult poetic features to detect are intrinsic qualities, especially style. The detection results verify the effectiveness and necessity of our proposed benchmark. Our work lays a foundation for future detection of AI-generated poetry.

Related papers

• Picturized and Recited with Dialects: A Multimodal Chinese Representation Framework for Sentiment Analysis of Classical Chinese Poetry [7.374104697960381]
  We propose a dialect-enhanced multimodal framework for classical Chinese poetry sentiment analysis.<n>We extract sentence-level audio features from the poetry and incorporate audio from multiple dialects.<n>Our framework outperforms state-of-the-art methods on two public datasets.
  arXiv  Detail & Related papers  (2025-05-19T14:58:44Z)
• Understanding Literary Texts by LLMs: A Case Study of Ancient Chinese Poetry [9.970908656435066]
  In genres such as poetry, jokes, and short stories, numerous AI tools have emerged, offering refreshing new perspectives. evaluating literary works is often complex and hard to fully quantify, which directly hinders the further development of AI creation. This paper attempts to explore the mysteries of literary texts from the perspective of large language models.
  arXiv  Detail & Related papers  (2024-08-22T04:25:06Z)
• RADAR: Robust AI-Text Detection via Adversarial Learning [69.5883095262619]
  RADAR is based on adversarial training of a paraphraser and a detector. The paraphraser's goal is to generate realistic content to evade AI-text detection. RADAR uses the feedback from the detector to update the paraphraser, and vice versa.
  arXiv  Detail & Related papers  (2023-07-07T21:13:27Z)
• Red Teaming Language Model Detectors with Language Models [114.36392560711022]
  Large language models (LLMs) present significant safety and ethical risks if exploited by malicious users. Recent works have proposed algorithms to detect LLM-generated text and protect LLMs. We study two types of attack strategies: 1) replacing certain words in an LLM's output with their synonyms given the context; 2) automatically searching for an instructional prompt to alter the writing style of the generation.
  arXiv  Detail & Related papers  (2023-05-31T10:08:37Z)
• MAGE: Machine-generated Text Detection in the Wild [82.70561073277801]
  Large language models (LLMs) have achieved human-level text generation, emphasizing the need for effective AI-generated text detection. We build a comprehensive testbed by gathering texts from diverse human writings and texts generated by different LLMs. Despite challenges, the top-performing detector can identify 86.54% out-of-domain texts generated by a new LLM, indicating the feasibility for application scenarios.
  arXiv  Detail & Related papers  (2023-05-22T17:13:29Z)
• Can AI-Generated Text be Reliably Detected? [50.95804851595018]
  Large Language Models (LLMs) perform impressively well in various applications.<n>The potential for misuse of these models in activities such as plagiarism, generating fake news, and spamming has raised concern about their responsible use.<n>We stress-test the robustness of these AI text detectors in the presence of an attacker.
  arXiv  Detail & Related papers  (2023-03-17T17:53:19Z)
• Generation of Chinese classical poetry based on pre-trained model [1.6114012813668934]
  This paper mainly tries to use BART and other pre training models to generate metrical poetry text. It developed a set of AI poetry Turing problems, it was reviewed by a group of poets and poetry writing researchers. The model of poetry generation studied by the author generalizes works that cannot be distinguished from those of advanced scholars.
  arXiv  Detail & Related papers  (2022-11-04T16:05:31Z)
• Chinese Traditional Poetry Generating System Based on Deep Learning [0.0]
  This paper proposes an automatic generation method of Chinese traditional poetry based on deep learning technology. It extracts keywords from each poem and matches them with the previous text to make the poem conform to the theme. When a user inputs a paragraph of text, the machine obtains the theme and generates poem sentence by sentence.
  arXiv  Detail & Related papers  (2021-10-24T02:43:03Z)
• CCPM: A Chinese Classical Poetry Matching Dataset [50.90794811956129]
  We propose a novel task to assess a model's semantic understanding of poetry by poem matching. This task requires the model to select one line of Chinese classical poetry among four candidates according to the modern Chinese translation of a line of poetry. To construct this dataset, we first obtain a set of parallel data of Chinese classical poetry and modern Chinese translation.
  arXiv  Detail & Related papers  (2021-06-03T16:49:03Z)
• Generating Major Types of Chinese Classical Poetry in a Uniformed Framework [88.57587722069239]
  We propose a GPT-2 based framework for generating major types of Chinese classical poems. Preliminary results show this enhanced model can generate Chinese classical poems of major types with high quality in both form and content.
  arXiv  Detail & Related papers  (2020-03-13T14:16:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.