Picturized and Recited with Dialects: A Multimodal Chinese Representation Framework for Sentiment Analysis of Classical Chinese Poetry

• URL: http://arxiv.org/abs/2505.13210v1
• Date: Mon, 19 May 2025 14:58:44 GMT
• Title: Picturized and Recited with Dialects: A Multimodal Chinese Representation Framework for Sentiment Analysis of Classical Chinese Poetry
• Authors: Xiaocong Du, Haoyu Pei, Haipeng Zhang,
• Abstract summary: We propose a dialect-enhanced multimodal framework for classical Chinese poetry sentiment analysis.<n>We extract sentence-level audio features from the poetry and incorporate audio from multiple dialects.<n>Our framework outperforms state-of-the-art methods on two public datasets.
• Score: 7.374104697960381
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Classical Chinese poetry is a vital and enduring part of Chinese literature, conveying profound emotional resonance. Existing studies analyze sentiment based on textual meanings, overlooking the unique rhythmic and visual features inherent in poetry,especially since it is often recited and accompanied by Chinese paintings. In this work, we propose a dialect-enhanced multimodal framework for classical Chinese poetry sentiment analysis. We extract sentence-level audio features from the poetry and incorporate audio from multiple dialects,which may retain regional ancient Chinese phonetic features, enriching the phonetic representation. Additionally, we generate sentence-level visual features, and the multimodal features are fused with textual features enhanced by LLM translation through multimodal contrastive representation learning. Our framework outperforms state-of-the-art methods on two public datasets, achieving at least 2.51% improvement in accuracy and 1.63% in macro F1. We open-source the code to facilitate research in this area and provide insights for general multimodal Chinese representation.

Related papers

• Zero-Shot Chinese Character Recognition with Hierarchical Multi-Granularity Image-Text Aligning [52.92837273570818]
  Chinese characters exhibit unique structures and compositional rules, allowing for the use of fine-grained semantic information in representation.<n>We propose a Hierarchical Multi-Granularity Image-Text Aligning (Hi-GITA) framework based on a contrastive paradigm.<n>Our proposed Hi-GITA outperforms existing zero-shot CCR methods.
  arXiv  Detail & Related papers  (2025-05-30T17:39:14Z)
• Multi-Modal Multi-Granularity Tokenizer for Chu Bamboo Slip Scripts [65.10991154918737]
  This study focuses on the Chu bamboo slip (CBS) script used during the Spring and Autumn and Warring States period (771-256 BCE) in Ancient China. Our tokenizer first adopts character detection to locate character boundaries, and then conducts character recognition at both the character and sub-character levels. To support the academic community, we have also assembled the first large-scale dataset of CBSs with over 100K annotated character image scans.
  arXiv  Detail & Related papers  (2024-09-02T07:42:55Z)
• Understanding Literary Texts by LLMs: A Case Study of Ancient Chinese Poetry [9.970908656435066]
  In genres such as poetry, jokes, and short stories, numerous AI tools have emerged, offering refreshing new perspectives. evaluating literary works is often complex and hard to fully quantify, which directly hinders the further development of AI creation. This paper attempts to explore the mysteries of literary texts from the perspective of large language models.
  arXiv  Detail & Related papers  (2024-08-22T04:25:06Z)
• Large Language Models Meet Text-Centric Multimodal Sentiment Analysis: A Survey [66.166184609616]
  ChatGPT has opened up immense potential for applying large language models (LLMs) to text-centric multimodal tasks. It is still unclear how existing LLMs can adapt better to text-centric multimodal sentiment analysis tasks.
  arXiv  Detail & Related papers  (2024-06-12T10:36:27Z)
• CMDAG: A Chinese Metaphor Dataset with Annotated Grounds as CoT for Boosting Metaphor Generation [35.14142183519002]
  This paper introduces a large-scale high quality annotated Chinese Metaphor Corpus, which comprises around 28K sentences. To ensure the accuracy and consistency of our annotations, we introduce a comprehensive set of guidelines. Breaking tradition, our approach to metaphor generation emphasizes grounds and their distinct features rather than the conventional combination of tenors and vehicles.
  arXiv  Detail & Related papers  (2024-02-20T17:00:41Z)
• A Computational Approach to Style in American Poetry [19.41186389974801]
  We develop a method to assess the style of American poems and to visualize a collection of poems in relation to one another. qualitative poetry criticism helped guide our development of metrics that analyze various orthographic, syntactic, and phonemic features. Our method has potential applications to academic research of texts, to research of the intuitive personal response to poetry, and to making recommendations to readers based on their favorite poems.
  arXiv  Detail & Related papers  (2023-10-13T18:49:14Z)
• Shuo Wen Jie Zi: Rethinking Dictionaries and Glyphs for Chinese Language Pre-training [50.100992353488174]
  We introduce CDBERT, a new learning paradigm that enhances the semantics understanding ability of the Chinese PLMs with dictionary knowledge and structure of Chinese characters. We name the two core modules of CDBERT as Shuowen and Jiezi, where Shuowen refers to the process of retrieving the most appropriate meaning from Chinese dictionaries. Our paradigm demonstrates consistent improvements on previous Chinese PLMs across all tasks.
  arXiv  Detail & Related papers  (2023-05-30T05:48:36Z)
• Chinese Traditional Poetry Generating System Based on Deep Learning [0.0]
  This paper proposes an automatic generation method of Chinese traditional poetry based on deep learning technology. It extracts keywords from each poem and matches them with the previous text to make the poem conform to the theme. When a user inputs a paragraph of text, the machine obtains the theme and generates poem sentence by sentence.
  arXiv  Detail & Related papers  (2021-10-24T02:43:03Z)
• CCPM: A Chinese Classical Poetry Matching Dataset [50.90794811956129]
  We propose a novel task to assess a model's semantic understanding of poetry by poem matching. This task requires the model to select one line of Chinese classical poetry among four candidates according to the modern Chinese translation of a line of poetry. To construct this dataset, we first obtain a set of parallel data of Chinese classical poetry and modern Chinese translation.
  arXiv  Detail & Related papers  (2021-06-03T16:49:03Z)
• Generating Major Types of Chinese Classical Poetry in a Uniformed Framework [88.57587722069239]
  We propose a GPT-2 based framework for generating major types of Chinese classical poems. Preliminary results show this enhanced model can generate Chinese classical poems of major types with high quality in both form and content.
  arXiv  Detail & Related papers  (2020-03-13T14:16:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.