POEMetric: The Last Stanza of Humanity

• URL: http://arxiv.org/abs/2604.03695v1
• Date: Sat, 04 Apr 2026 11:48:55 GMT
• Title: POEMetric: The Last Stanza of Humanity
• Abstract summary: Large Language Models (LLMs) can compose poetry, but how far are they from human poets?<n>We introduce POEMetric, the first comprehensive framework for poetry evaluation, examining 1) basic instruction-following abilities in generating poems according to a certain form and theme, 2) advanced abilities of showing creativity, lexical diversity, and idiosyncrasy, evoking emotional resonance, and using imagery and literary devices, and 3) general appraisal of the overall poem quality and estimation of authorship.
• Score: 1.8453032748632798
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large Language Models (LLMs) can compose poetry, but how far are they from human poets? In this paper, we introduce POEMetric, the first comprehensive framework for poetry evaluation, examining 1) basic instruction-following abilities in generating poems according to a certain form and theme, 2) advanced abilities of showing creativity, lexical diversity, and idiosyncrasy, evoking emotional resonance, and using imagery and literary devices, and 3) general appraisal of the overall poem quality and estimation of authorship. We curated a human poem dataset - 203 English poems of 7 fixed forms annotated with meter, rhyme patterns and themes - and experimented with 30 LLMs for poetry generation based on the same forms and themes of the human data, totaling 6,090 LLM poems. Based on POEMetric, we assessed the performance of both human poets and LLMs through rule-based evaluation and LLM-as-a-judge, whose results were validated by human experts. Results show that, though the top model achieved high form accuracy (4.26 out of 5.00, with Gemini-2.5-Pro as a judge; same below) and theme alignment (4.99), all models failed to reach the same level of advanced abilities as human poets, who achieved unparalleled creativity (4.02), idiosyncrasy (3.95), emotional resonance (4.06), and skillful use of imagery (4.49) and literary devices (4.67). Humans also defeated the best-performing LLM in overall poem quality (4.22 vs. 3.20). As such, poetry generation remains a formidable challenge for LLMs. Data and codes are released at https://github.com/Bingru-Li/POEMetric.

Related papers

• Do Large Language Models Perform Well on Comprehending Poetic Logic in Modern Chinese Poetry? [47.71394430680399]
  Large Language Models (LLMs) have achieved significant progress across a wide range of natural language processing (NLP) tasks.<n>Modern Chinese poetry necessitate a distinct form of reasoning for effective comprehension.<n>We propose Peony, the first benchmark specifically designed for evaluating the poetic logic of modern Chinese poetry.
  arXiv  Detail & Related papers  (2026-08-22T07:52:00Z)
• Can ChatGPT Really Understand Modern Chinese Poetry? [41.340415709375186]
  ChatGPT has demonstrated remarkable capabilities on both poetry generation and translation.<n>Previous poetry-related work merely analyzed experimental outcomes without addressing fundamental issues of comprehension.<n>This paper introduces a comprehensive framework for evaluating ChatGPT's understanding of modern poetry.
  arXiv  Detail & Related papers  (2026-03-21T15:14:43Z)
• Creating a digital poet [8.593632029412502]
  We report a seven-month poetry workshop in which a large language model was shaped into a digital poet.<n>The model developed a distinctive style and a coherent corpus, supported by quantitative and qualitative analyses.<n>After the workshop, a commercial publisher released a poetry collection authored by the model.
  arXiv  Detail & Related papers  (2026-02-18T16:25:10Z)
• Does ChatGPT Have a Poetic Style? [0.6827423171182154]
  We prompt the GPT-3.5 and GPT-4 models to generate English-language poems in 24 different poetic forms and styles. We analyze the resulting 5.7k poems, comparing them to a sample of 3.7k poems from the Poetry Foundation and the Academy of American Poets. We find that the GPT models, especially GPT-4, can successfully produce poems in a range of both common and uncommon English-language forms.
  arXiv  Detail & Related papers  (2024-10-20T06:01:34Z)
• Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasets [3.0040661953201475]
  Large language models (LLMs) can now generate and recognize poetry. We develop a task to evaluate how well LLMs recognize one aspect of English-language poetry. We show that state-of-the-art LLMs can successfully identify both common and uncommon fixed poetic forms.
  arXiv  Detail & Related papers  (2024-06-27T05:36:53Z)
• Measuring Psychological Depth in Language Models [50.48914935872879]
  We introduce the Psychological Depth Scale (PDS), a novel framework rooted in literary theory that measures an LLM's ability to produce authentic and narratively complex stories. We empirically validate our framework by showing that humans can consistently evaluate stories based on PDS (0.72 Krippendorff's alpha) Surprisingly, GPT-4 stories either surpassed or were statistically indistinguishable from highly-rated human-written stories sourced from Reddit.
  arXiv  Detail & Related papers  (2024-06-18T14:51:54Z)
• Generation of Chinese classical poetry based on pre-trained model [1.6114012813668934]
  This paper mainly tries to use BART and other pre training models to generate metrical poetry text. It developed a set of AI poetry Turing problems, it was reviewed by a group of poets and poetry writing researchers. The model of poetry generation studied by the author generalizes works that cannot be distinguished from those of advanced scholars.
  arXiv  Detail & Related papers  (2022-11-04T16:05:31Z)
• PoeLM: A Meter- and Rhyme-Controllable Language Model for Unsupervised Poetry Generation [42.12348554537587]
  Formal verse poetry imposes strict constraints on the meter and rhyme scheme of poems. Most prior work on generating this type of poetry uses existing poems for supervision. We propose an unsupervised approach to generate poems following any given meter and rhyme scheme.
  arXiv  Detail & Related papers  (2022-05-24T17:09:55Z)
• CCPM: A Chinese Classical Poetry Matching Dataset [50.90794811956129]
  We propose a novel task to assess a model's semantic understanding of poetry by poem matching. This task requires the model to select one line of Chinese classical poetry among four candidates according to the modern Chinese translation of a line of poetry. To construct this dataset, we first obtain a set of parallel data of Chinese classical poetry and modern Chinese translation.
  arXiv  Detail & Related papers  (2021-06-03T16:49:03Z)
• MixPoet: Diverse Poetry Generation via Learning Controllable Mixed Latent Space [79.70053419040902]
  We propose MixPoet, a novel model that absorbs multiple factors to create various styles and promote diversity. Based on a semi-supervised variational autoencoder, our model disentangles the latent space into some subspaces, with each conditioned on one influence factor by adversarial training. Experiment results on Chinese poetry demonstrate that MixPoet improves both diversity and quality against three state-of-the-art models.
  arXiv  Detail & Related papers  (2020-03-13T03:31:29Z)
• Introducing Aspects of Creativity in Automatic Poetry Generation [2.792030485253753]
  Poetry Generation involves teaching systems to automatically generate text that resembles poetic work. A deep learning system can learn to generate poetry on its own by training on a corpus of poems and modeling the particular style of language. We propose taking an approach that fine-tunes GPT-2, a pre-trained language model, to our downstream task of poetry generation.
  arXiv  Detail & Related papers  (2020-02-06T20:44:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.