CharPoet: A Chinese Classical Poetry Generation System Based on Token-free LLM

• URL: http://arxiv.org/abs/2401.03512v3
• Date: Wed, 20 Mar 2024 07:39:48 GMT
• Title: CharPoet: A Chinese Classical Poetry Generation System Based on Token-free LLM
• Abstract summary: CharPoet is a Chinese classical poetry generation system based on token-free LLM. Our token-free architecture generates in a character-by-character manner, enabling precise control over the number of characters. CharPoet achieves format accuracy above 0.96, outperforming Jiuge-GPT-2 (0.91) and GPT-4 (0.38)
• Score: 7.6361655389908965
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automatic Chinese classical poetry generation has attracted much research interest, but achieving effective control over format and content simultaneously remains challenging. Traditional systems usually accept keywords as user inputs, resulting in limited control over content. Large language models (LLMs) improve content control by allowing unrestricted user instructions, but the token-by-token generation process frequently makes format errors. Motivated by this, we propose CharPoet, a Chinese classical poetry generation system based on token-free LLM, which provides effective control over both format and content. Our token-free architecture generates in a character-by-character manner, enabling precise control over the number of characters. Pruned from existing token-based LLMs, CharPoet inherits their pretrained capabilities and can generate poetry following instructions like "Write me a poem for my mother's birthday." CharPoet achieves format accuracy above 0.96, outperforming Jiuge-GPT-2 (0.91) and GPT-4 (0.38). In terms of content quality, CharPoet surpasses traditional systems including Jiuge, and is comparable to other LLMs. Our system is open source and available at https://modelscope.cn/models/CharPoet/CharPoet. A video demonstration of CharPoet is available at https://youtu.be/voZ25qEp3Dc.

Related papers

• TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment [46.819665083185136]
  Inefficient tokenization results in long token-ID sequences and will slow down the training and inference of Large Language Models.<n>We introduce a method named TokAlign++ to improve vocabulary adaptation performance by learning better token alignment lexicon.<n> Experimental results on 15 languages show that our method boosts the multilingual text compression rates and preserves most of the multilingual ability of vanilla models.
  arXiv  Detail & Related papers  (2026-05-13T12:23:24Z)
• Who Wrote This Line? Evaluating the Detection of LLM-Generated Classical Chinese Poetry [45.27531384029669]
  AI-generated literary creations have raised prominent issues of creative authenticity and ethics in literary world.<n>Previous works have made significant progress in detecting AI-generated text, but it has yet to address classical Chinese poetry.<n>We introduce ChangAn, a benchmark for detecting LLM-generated classical Chinese poetry.
  arXiv  Detail & Related papers  (2026-04-11T08:52:08Z)
• Speculating LLMs' Chinese Training Data Pollution from Their Tokens [59.325920108736454]
  Polluted Chinese (PoC) tokens are indicated contents like pornography or online gambling.<n>We give a formal definition and taxonomy of PoC tokens based on the GPT's vocabulary.<n>We build a PoC token detector via fine-tuning an LLM to label PoC tokens in vocabularies.
  arXiv  Detail & Related papers  (2025-08-25T08:08:51Z)
• Prompt-Based One-Shot Exact Length-Controlled Generation with LLMs [56.47577824219207]
  We present a prompt-based strategy that compels an off-the-shelf large language model to generate exactly a desired number of tokens.<n>The prompt appends countdown markers and explicit counting rules so that the model "writes while counting"<n>On MT-Bench-LI, strict length compliance with GPT-4.1 leaps from below 30% under naive prompts to above 95% with our countdown prompt.
  arXiv  Detail & Related papers  (2025-08-19T13:12:01Z)
• From Language Models over Tokens to Language Models over Characters [54.123846188068384]
  Modern language models are internally -- and mathematically -- distributions over $ittoken$ strings rather than $itcharacter$ strings.<n>This paper presents algorithms for converting token-level language models to character-level ones.
  arXiv  Detail & Related papers  (2024-12-04T21:19:20Z)
• Parrot: Multilingual Visual Instruction Tuning [66.65963606552839]
  Existing methods mainly focus on aligning vision encoders with Multimodal Large Language Models (MLLMs) We introduce Parrot, a novel method that utilizes textual guidance to drive visual token alignment at the language level. Our method not only demonstrates state-of-the-art performance on multilingual MMBench and MMMB, but also excels across a broad range of multimodal tasks.
  arXiv  Detail & Related papers  (2024-06-04T17:56:28Z)
• Introducing Syllable Tokenization for Low-resource Languages: A Case Study with Swahili [29.252250069388687]
  Tokenization allows for the words to be split based on characters or subwords, creating word embeddings that best represent the structure of the language. We propose a syllable tokenizer and adopt an experiment-centric approach to validate the proposed tokenizer based on the Swahili language.
  arXiv  Detail & Related papers  (2024-03-26T17:26:50Z)
• AlignedCoT: Prompting Large Language Models via Native-Speaking Demonstrations [52.43593893122206]
  Alignedcot is an in-context learning technique for invoking Large Language Models. It achieves consistent and correct step-wise prompts in zero-shot scenarios. We conduct experiments on mathematical reasoning and commonsense reasoning.
  arXiv  Detail & Related papers  (2023-11-22T17:24:21Z)
• ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Models [23.381986209234157]
  In this work, we investigate end-to-end poetry generation conditioned on styles such as rhyme, meter, and alliteration. We successfully pre-train ByGPT5, a new token-free decoder-only language model, and fine-tune it on a large custom corpus of English and German quatrains annotated with our styles. We show that ByGPT5 outperforms other models such as mT5, ByT5, GPT-2 and ChatGPT, while also being more parameter efficient and performing favorably compared to humans.
  arXiv  Detail & Related papers  (2022-12-20T17:49:49Z)
• Help me write a poem: Instruction Tuning as a Vehicle for Collaborative Poetry Writing [17.975181293647697]
  We present CoPoet, a collaborative poetry writing system. In contrast to auto-completing a user's text, CoPoet is controlled by user instructions that specify the attributes of the desired text. A study with 15 qualified crowdworkers shows that users successfully write poems with CoPoet on diverse topics ranging from Monarchy to Climate change.
  arXiv  Detail & Related papers  (2022-10-25T00:07:10Z)
• What do tokens know about their characters and how do they know it? [3.8254443661593633]
  We show that pre-trained language models that use subword tokenization schemes can succeed at a variety of language tasks that require character-level information. We show that these models robustly encode character-level information and, in general, larger models perform better at the task.
  arXiv  Detail & Related papers  (2022-06-06T13:27:26Z)
• Chinese Traditional Poetry Generating System Based on Deep Learning [0.0]
  This paper proposes an automatic generation method of Chinese traditional poetry based on deep learning technology. It extracts keywords from each poem and matches them with the previous text to make the poem conform to the theme. When a user inputs a paragraph of text, the machine obtains the theme and generates poem sentence by sentence.
  arXiv  Detail & Related papers  (2021-10-24T02:43:03Z)
• Charformer: Fast Character Transformers via Gradient-based Subword Tokenization [50.16128796194463]
  We propose a new model inductive bias that learns a subword tokenization end-to-end as part of the model. We introduce a soft gradient-based subword tokenization module (GBST) that automatically learns latent subword representations from characters. We additionally introduce Charformer, a deep Transformer model that integrates GBST and operates on the byte level.
  arXiv  Detail & Related papers  (2021-06-23T22:24:14Z)
• CCPM: A Chinese Classical Poetry Matching Dataset [50.90794811956129]
  We propose a novel task to assess a model's semantic understanding of poetry by poem matching. This task requires the model to select one line of Chinese classical poetry among four candidates according to the modern Chinese translation of a line of poetry. To construct this dataset, we first obtain a set of parallel data of Chinese classical poetry and modern Chinese translation.
  arXiv  Detail & Related papers  (2021-06-03T16:49:03Z)
• SHUOWEN-JIEZI: Linguistically Informed Tokenizers For Chinese Language Model Pretraining [48.880840711568425]
  We study the influences of three main factors on the Chinese tokenization for pretrained language models. We propose three kinds of tokenizers: SHUOWEN (meaning Talk Word), the pronunciation-based tokenizers; 2) JIEZI (meaning Solve Character), the glyph-based tokenizers. We find that SHUOWEN and JIEZI tokenizers can generally outperform conventional single-character tokenizers.
  arXiv  Detail & Related papers  (2021-06-01T11:20:02Z)
• Generating Major Types of Chinese Classical Poetry in a Uniformed Framework [88.57587722069239]
  We propose a GPT-2 based framework for generating major types of Chinese classical poems. Preliminary results show this enhanced model can generate Chinese classical poems of major types with high quality in both form and content.
  arXiv  Detail & Related papers  (2020-03-13T14:16:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.