Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasets

• URL: http://arxiv.org/abs/2406.18906v3
• Date: Thu, 10 Oct 2024 19:35:56 GMT
• Title: Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasets
• Authors: Melanie Walsh, Anna Preus, Maria Antoniak,
• Abstract summary: Large language models (LLMs) can now generate and recognize poetry. We develop a task to evaluate how well LLMs recognize one aspect of English-language poetry. We show that state-of-the-art LLMs can successfully identify both common and uncommon fixed poetic forms.
• Score: 3.0040661953201475
• License:
• Abstract: Large language models (LLMs) can now generate and recognize poetry. But what do LLMs really know about poetry? We develop a task to evaluate how well LLMs recognize one aspect of English-language poetry--poetic form--which captures many different poetic features, including rhyme scheme, meter, and word or line repetition. By using a benchmark dataset of over 4.1k human expert-annotated poems, we show that state-of-the-art LLMs can successfully identify both common and uncommon fixed poetic forms--such as sonnets, sestinas, and pantoums--with surprisingly high accuracy. However, performance varies significantly by poetic form; the models struggle to identify unfixed poetic forms, especially those based on topic or visual features. We additionally measure how many poems from our benchmark dataset are present in popular pretraining datasets or memorized by GPT-4, finding that pretraining presence and memorization may improve performance on this task, but results are inconclusive. We release a benchmark evaluation dataset with 1.4k public domain poems and form annotations, results of memorization experiments and data audits, and code.

Related papers

• A Computational Approach to Style in American Poetry [19.41186389974801]
  We develop a method to assess the style of American poems and to visualize a collection of poems in relation to one another. qualitative poetry criticism helped guide our development of metrics that analyze various orthographic, syntactic, and phonemic features. Our method has potential applications to academic research of texts, to research of the intuitive personal response to poetry, and to making recommendations to readers based on their favorite poems.
  arXiv  Detail & Related papers  (2023-10-13T18:49:14Z)
• PoetryDiffusion: Towards Joint Semantic and Metrical Manipulation in Poetry Generation [58.36105306993046]
  Controllable text generation is a challenging and meaningful field in natural language generation (NLG) In this paper, we pioneer the use of the Diffusion model for generating sonnets and Chinese SongCi poetry. Our model outperforms existing models in automatic evaluation of semantic, metrical, and overall performance as well as human evaluation.
  arXiv  Detail & Related papers  (2023-06-14T11:57:31Z)
• PoeLM: A Meter- and Rhyme-Controllable Language Model for Unsupervised Poetry Generation [42.12348554537587]
  Formal verse poetry imposes strict constraints on the meter and rhyme scheme of poems. Most prior work on generating this type of poetry uses existing poems for supervision. We propose an unsupervised approach to generate poems following any given meter and rhyme scheme.
  arXiv  Detail & Related papers  (2022-05-24T17:09:55Z)
• Syllabic Quantity Patterns as Rhythmic Features for Latin Authorship Attribution [74.27826764855911]
  We employ syllabic quantity as a base for deriving rhythmic features for the task of computational authorship attribution of Latin prose texts. Our experiments, carried out on three different datasets, using two different machine learning methods, show that rhythmic features based on syllabic quantity are beneficial in discriminating among Latin prose authors.
  arXiv  Detail & Related papers  (2021-10-27T06:25:31Z)
• CCPM: A Chinese Classical Poetry Matching Dataset [50.90794811956129]
  We propose a novel task to assess a model's semantic understanding of poetry by poem matching. This task requires the model to select one line of Chinese classical poetry among four candidates according to the modern Chinese translation of a line of poetry. To construct this dataset, we first obtain a set of parallel data of Chinese classical poetry and modern Chinese translation.
  arXiv  Detail & Related papers  (2021-06-03T16:49:03Z)
• Acrostic Poem Generation [26.604889384391726]
  We propose a new task in the area of computational creativity: acrostic poem generation in English. Acrostic poems are poems that contain a hidden message; typically, the first letter of each line spells out a word or short phrase. Our experiments show that the acrostic poems generated by our baseline are received well by humans and do not lose much quality due to the additional constraints.
  arXiv  Detail & Related papers  (2020-10-05T18:00:15Z)
• Abstractive Summarization of Spoken and Written Instructions with BERT [66.14755043607776]
  We present the first application of the BERTSum model to conversational language. We generate abstractive summaries of narrated instructional videos across a wide variety of topics. We envision this integrated as a feature in intelligent virtual assistants, enabling them to summarize both written and spoken instructional content upon request.
  arXiv  Detail & Related papers  (2020-08-21T20:59:34Z)
• MixPoet: Diverse Poetry Generation via Learning Controllable Mixed Latent Space [79.70053419040902]
  We propose MixPoet, a novel model that absorbs multiple factors to create various styles and promote diversity. Based on a semi-supervised variational autoencoder, our model disentangles the latent space into some subspaces, with each conditioned on one influence factor by adversarial training. Experiment results on Chinese poetry demonstrate that MixPoet improves both diversity and quality against three state-of-the-art models.
  arXiv  Detail & Related papers  (2020-03-13T03:31:29Z)
• Introducing Aspects of Creativity in Automatic Poetry Generation [2.792030485253753]
  Poetry Generation involves teaching systems to automatically generate text that resembles poetic work. A deep learning system can learn to generate poetry on its own by training on a corpus of poems and modeling the particular style of language. We propose taking an approach that fine-tunes GPT-2, a pre-trained language model, to our downstream task of poetry generation.
  arXiv  Detail & Related papers  (2020-02-06T20:44:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.