CCiV: A Benchmark for Structure, Rhythm and Quality in LLM-Generated Chinese \textit{Ci} Poetry

• URL: http://arxiv.org/abs/2602.14081v1
• Date: Sun, 15 Feb 2026 10:19:02 GMT
• Title: CCiV: A Benchmark for Structure, Rhythm and Quality in LLM-Generated Chinese \textit{Ci} Poetry
• Abstract summary: We introduce textbfChinese textbfCipai textbfVariants (textbfCCiV), a benchmark designed to assess textitCi poetry.<n>Our evaluation of 17 LLMs on 30 textitCipai reveals two critical phenomena: models frequently generate valid but unexpected historical variants of a poetic form, and adherence to tonal patterns is substantially harder than structural rules.
• Score: 21.543122877673294
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The generation of classical Chinese \textit{Ci} poetry, a form demanding a sophisticated blend of structural rigidity, rhythmic harmony, and artistic quality, poses a significant challenge for large language models (LLMs). To systematically evaluate and advance this capability, we introduce \textbf{C}hinese \textbf{Ci}pai \textbf{V}ariants (\textbf{CCiV}), a benchmark designed to assess LLM-generated \textit{Ci} poetry across these three dimensions: structure, rhythm, and quality. Our evaluation of 17 LLMs on 30 \textit{Cipai} reveals two critical phenomena: models frequently generate valid but unexpected historical variants of a poetic form, and adherence to tonal patterns is substantially harder than structural rules. We further show that form-aware prompting can improve structural and tonal control for stronger models, while potentially degrading weaker ones. Finally, we observe weak and inconsistent alignment between formal correctness and literary quality in our sample. CCiV highlights the need for variant-aware evaluation and more holistic constrained creative generation methods.

Related papers

• Implicit Style Conditioning: A Structured Style-Rewrite Framework for Low-Resource Character Modeling [0.0]
  Small Language Models (SLMs) with highly stylized personas remains a challenge due to data scarcity and the complexity of style disentanglement.<n>We propose a Structured Style-Rewrite Framework that explicitly disentangles style into three interpretable dimensions.<n>By leveraging explicit reasoning traces during training as a strong inductive bias, our approach aligns the model's latent representations with structured style features.
  arXiv  Detail & Related papers  (2026-03-06T06:04:47Z)
• TextPecker: Rewarding Structural Anomaly Quantification for Enhancing Visual Text Rendering [76.53315206999231]
  TextPecker is a plug-and-play structural anomaly perceptive RL strategy.<n>It mitigates noisy reward signals and works with any textto-image generators.<n>It significantly yields average gains of 4% in structural fidelity and 8.7% in semantic alignment for Chinese text rendering.
  arXiv  Detail & Related papers  (2026-02-24T13:40:23Z)
• Capabilities and Evaluation Biases of Large Language Models in Classical Chinese Poetry Generation: A Case Study on Tang Poetry [4.720025219010595]
  Large Language Models (LLMs) are increasingly applied to creative domains, yet their performance in classical Chinese poetry generation and evaluation remains poorly understood.<n>We propose a three-step evaluation framework that combines computational metrics, LLM-as-a-judge assessment, and human expert validation.
  arXiv  Detail & Related papers  (2025-10-17T05:00:37Z)
• From Plain Text to Poetic Form: Generating Metrically-Constrained Sanskrit Verses [22.08984009109879]
  We introduce a dataset designed for translating English prose into structured Sanskrit verse.<n>We explore constrained decoding strategies and instruction-based fine-tuning tailored to metrical and semantic fidelity.
  arXiv  Detail & Related papers  (2025-06-01T03:35:46Z)
• Align to Structure: Aligning Large Language Models with Structural Information [26.960069076925386]
  We introduce Structural Alignment, a novel method that aligns large language models with human-like discourse structures to enhance long-form text generation.<n>We employ a dense reward scheme within a Proximal Policy Optimization framework, assigning fine-grained, token-level rewards based on the discourse distinctiveness relative to human writing.
  arXiv  Detail & Related papers  (2025-04-04T17:40:04Z)
• GPT Czech Poet: Generation of Czech Poetic Strophes with Language Models [0.4444634303550442]
  We introduce a new model for generating poetry in Czech language, based on fine-tuning a pre-trained Large Language Model. We demonstrate that guiding the generation process by explicitly specifying strophe parameters within the poem text strongly improves the effectiveness of the model.
  arXiv  Detail & Related papers  (2024-06-18T06:19:45Z)
• PoetryDiffusion: Towards Joint Semantic and Metrical Manipulation in Poetry Generation [58.36105306993046]
  Controllable text generation is a challenging and meaningful field in natural language generation (NLG) In this paper, we pioneer the use of the Diffusion model for generating sonnets and Chinese SongCi poetry. Our model outperforms existing models in automatic evaluation of semantic, metrical, and overall performance as well as human evaluation.
  arXiv  Detail & Related papers  (2023-06-14T11:57:31Z)
• Physics of Language Models: Part 1, Learning Hierarchical Language Structures [51.68385617116854]
  Transformer-based language models are effective but complex, and understanding their inner workings and reasoning mechanisms is a significant challenge.<n>We introduce a family of synthetic CFGs that produce hierarchical rules, capable of generating lengthy sentences.<n>We demonstrate that generative models like GPT can accurately learn and reason over CFG-defined hierarchies and generate sentences based on it.
  arXiv  Detail & Related papers  (2023-05-23T04:28:16Z)
• Model Criticism for Long-Form Text Generation [113.13900836015122]
  We apply a statistical tool, model criticism in latent space, to evaluate the high-level structure of generated text. We perform experiments on three representative aspects of high-level discourse -- coherence, coreference, and topicality. We find that transformer-based language models are able to capture topical structures but have a harder time maintaining structural coherence or modeling coreference.
  arXiv  Detail & Related papers  (2022-10-16T04:35:58Z)
• How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN [63.79300884115027]
  Current language models can generate high-quality text. Are they simply copying text they have seen before, or have they learned generalizable linguistic abstractions? We introduce RAVEN, a suite of analyses for assessing the novelty of generated text.
  arXiv  Detail & Related papers  (2021-11-18T04:07:09Z)
• SongNet: Rigid Formats Controlled Text Generation [51.428634666559724]
  We propose a simple and elegant framework named SongNet to tackle this problem. The backbone of the framework is a Transformer-based auto-regressive language model. A pre-training and fine-tuning framework is designed to further improve the generation quality.
  arXiv  Detail & Related papers  (2020-04-17T01:40:18Z)
• Generating Major Types of Chinese Classical Poetry in a Uniformed Framework [88.57587722069239]
  We propose a GPT-2 based framework for generating major types of Chinese classical poems. Preliminary results show this enhanced model can generate Chinese classical poems of major types with high quality in both form and content.
  arXiv  Detail & Related papers  (2020-03-13T14:16:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.