Who Wrote This Line? Evaluating the Detection of LLM-Generated Classical Chinese Poetry

• URL: http://arxiv.org/abs/2604.10101v2
• Date: Thu, 16 Apr 2026 07:39:05 GMT
• Title: Who Wrote This Line? Evaluating the Detection of LLM-Generated Classical Chinese Poetry
• Abstract summary: AI-generated literary creations have raised prominent issues of creative authenticity and ethics in literary world.<n>Previous works have made significant progress in detecting AI-generated text, but it has yet to address classical Chinese poetry.<n>We introduce ChangAn, a benchmark for detecting LLM-generated classical Chinese poetry.
• Score: 45.27531384029669
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid development of large language models (LLMs) has extended text generation tasks into the literary domain. However, AI-generated literary creations has raised increasingly prominent issues of creative authenticity and ethics in literary world, making the detection of LLM-generated literary texts essential and urgent. While previous works have made significant progress in detecting AI-generated text, it has yet to address classical Chinese poetry. Due to the unique linguistic features of classical Chinese poetry, such as strict metrical regularity, a shared system of poetic imagery, and flexible syntax, distinguishing whether a poem is authored by AI presents a substantial challenge. To address these issues, we introduce ChangAn, a benchmark for detecting LLM-generated classical Chinese poetry that containing total 30,664 poems, 10,276 are human-written poems and 20,388 poems are generated by four popular LLMs. Based on ChangAn, we conducted a systematic evaluation of 12 AI detectors, investigating their performance variations across different text granularities and generation strategies. Our findings highlight the limitations of current Chinese text detectors, which fail to serve as reliable tools for detecting LLM-generated classical Chinese poetry. These results validate the effectiveness and necessity of our proposed ChangAn benchmark. Our dataset and code are available at https://github.com/VelikayaScarlet/ChangAn.

Related papers

• Jiuge-Tuiqiao: An Interpretable Human-AI System for Classical Chinese Poetry Refinement [48.293435958731614]
  Jiuge-Tuiqiao is an interactive human-AI collaborative system for classical Chinese poetry composition.<n>Users can lock characters or lines, receive real-time prosody feedback, and obtain interpretable refinement suggestions.<n>Preliminary experiments and user feedback suggest that Jiuge-Tuiqiao improves controllability, interpretability, and user engagement.
  arXiv  Detail & Related papers  (2026-08-24T11:05:09Z)
• Do Large Language Models Perform Well on Comprehending Poetic Logic in Modern Chinese Poetry? [47.71394430680399]
  Large Language Models (LLMs) have achieved significant progress across a wide range of natural language processing (NLP) tasks.<n>Modern Chinese poetry necessitate a distinct form of reasoning for effective comprehension.<n>We propose Peony, the first benchmark specifically designed for evaluating the poetic logic of modern Chinese poetry.
  arXiv  Detail & Related papers  (2026-08-22T07:52:00Z)
• Seeing the Poem: Image-Semantic Detection of AI-Generated Modern Chinese Poetry with MLLMs [71.29726278473758]
  Previous detection studies have shown that LLMs cannot be effectively used as detectors, but these studies have not addressed modern Chinese poetry.<n>This paper evaluates and enhances the performance of LLMs as detectors for modern Chinese poetry, and proposes an image-semantic guided poetry detection method.
  arXiv  Detail & Related papers  (2026-05-21T15:57:04Z)
• Benchmarking the Detection of LLMs-Generated Modern Chinese Poetry [37.86155340473244]
  It is difficult to identify whether a poem originated from humans or AI.<n>This paper proposes a novel benchmark for detecting AI-generated modern Chinese poetry.
  arXiv  Detail & Related papers  (2025-09-01T17:01:45Z)
• Picturized and Recited with Dialects: A Multimodal Chinese Representation Framework for Sentiment Analysis of Classical Chinese Poetry [7.374104697960381]
  We propose a dialect-enhanced multimodal framework for classical Chinese poetry sentiment analysis.<n>We extract sentence-level audio features from the poetry and incorporate audio from multiple dialects.<n>Our framework outperforms state-of-the-art methods on two public datasets.
  arXiv  Detail & Related papers  (2025-05-19T14:58:44Z)
• Understanding Literary Texts by LLMs: A Case Study of Ancient Chinese Poetry [9.970908656435066]
  In genres such as poetry, jokes, and short stories, numerous AI tools have emerged, offering refreshing new perspectives. evaluating literary works is often complex and hard to fully quantify, which directly hinders the further development of AI creation. This paper attempts to explore the mysteries of literary texts from the perspective of large language models.
  arXiv  Detail & Related papers  (2024-08-22T04:25:06Z)
• Large Language Models for Classical Chinese Poetry Translation: Benchmarking, Evaluating, and Improving [43.148203559785095]
  Large language models (LLMs) with impressive multilingual capabilities may bring a ray of hope to achieve this extreme translation demand.<n>This paper first introduces a suitable benchmark (PoetMT) where each Chinese poetry has a recognized elegant translation.<n>We propose a new metric based on GPT-4 to evaluate the extent to which current LLMs can meet these demands.
  arXiv  Detail & Related papers  (2024-08-19T12:34:31Z)
• MAGE: Machine-generated Text Detection in the Wild [82.70561073277801]
  Large language models (LLMs) have achieved human-level text generation, emphasizing the need for effective AI-generated text detection. We build a comprehensive testbed by gathering texts from diverse human writings and texts generated by different LLMs. Despite challenges, the top-performing detector can identify 86.54% out-of-domain texts generated by a new LLM, indicating the feasibility for application scenarios.
  arXiv  Detail & Related papers  (2023-05-22T17:13:29Z)
• CCPM: A Chinese Classical Poetry Matching Dataset [50.90794811956129]
  We propose a novel task to assess a model's semantic understanding of poetry by poem matching. This task requires the model to select one line of Chinese classical poetry among four candidates according to the modern Chinese translation of a line of poetry. To construct this dataset, we first obtain a set of parallel data of Chinese classical poetry and modern Chinese translation.
  arXiv  Detail & Related papers  (2021-06-03T16:49:03Z)
• Generating Major Types of Chinese Classical Poetry in a Uniformed Framework [88.57587722069239]
  We propose a GPT-2 based framework for generating major types of Chinese classical poems. Preliminary results show this enhanced model can generate Chinese classical poems of major types with high quality in both form and content.
  arXiv  Detail & Related papers  (2020-03-13T14:16:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.