Multi-Modal Multi-Granularity Tokenizer for Chu Bamboo Slip Scripts

• URL: http://arxiv.org/abs/2409.01011v1
• Date: Mon, 2 Sep 2024 07:42:55 GMT
• Title: Multi-Modal Multi-Granularity Tokenizer for Chu Bamboo Slip Scripts
• Authors: Yingfa Chen, Chenlong Hu, Cong Feng, Chenyang Song, Shi Yu, Xu Han, Zhiyuan Liu, Maosong Sun,
• Abstract summary: This study focuses on the Chu bamboo slip (CBS) script used during the Spring and Autumn and Warring States period (771-256 BCE) in Ancient China. Our tokenizer first adopts character detection to locate character boundaries, and then conducts character recognition at both the character and sub-character levels. To support the academic community, we have also assembled the first large-scale dataset of CBSs with over 100K annotated character image scans.
• Score: 65.10991154918737
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This study presents a multi-modal multi-granularity tokenizer specifically designed for analyzing ancient Chinese scripts, focusing on the Chu bamboo slip (CBS) script used during the Spring and Autumn and Warring States period (771-256 BCE) in Ancient China. Considering the complex hierarchical structure of ancient Chinese scripts, where a single character may be a combination of multiple sub-characters, our tokenizer first adopts character detection to locate character boundaries, and then conducts character recognition at both the character and sub-character levels. Moreover, to support the academic community, we have also assembled the first large-scale dataset of CBSs with over 100K annotated character image scans. On the part-of-speech tagging task built on our dataset, using our tokenizer gives a 5.5% relative improvement in F1-score compared to mainstream sub-word tokenizers. Our work not only aids in further investigations of the specific script but also has the potential to advance research on other forms of ancient Chinese scripts.

Related papers

• HierCode: A Lightweight Hierarchical Codebook for Zero-shot Chinese Text Recognition [47.86479271322264]
  We propose HierCode, a novel and lightweight codebook that exploits the innate hierarchical nature of Chinese characters. HierCode employs a multi-hot encoding strategy, leveraging hierarchical binary tree encoding and prototype learning to create distinctive, informative representations for each character. This approach not only facilitates zero-shot recognition of OOV characters by utilizing shared radicals and structures but also excels in line-level recognition tasks by computing similarity with visual features.
  arXiv  Detail & Related papers  (2024-03-20T17:20:48Z)
• Chinese Text Recognition with A Pre-Trained CLIP-Like Model Through Image-IDS Aligning [61.34060587461462]
  We propose a two-stage framework for Chinese Text Recognition (CTR) We pre-train a CLIP-like model through aligning printed character images and Ideographic Description Sequences (IDS) This pre-training stage simulates humans recognizing Chinese characters and obtains the canonical representation of each character. The learned representations are employed to supervise the CTR model, such that traditional single-character recognition can be improved to text-line recognition.
  arXiv  Detail & Related papers  (2023-09-03T05:33:16Z)
• DeepSolo++: Let Transformer Decoder with Explicit Points Solo for Multilingual Text Spotting [112.45423990924283]
  DeepSolo++ is a simple DETR-like baseline that lets a single decoder with explicit points solo for text detection, recognition, and script identification simultaneously. Our method not only performs well in English scenes but also masters the transcription with complex font structure and a thousand-level character classes, such as Chinese.
  arXiv  Detail & Related papers  (2023-05-31T15:44:00Z)
• Joint Chinese Word Segmentation and Span-based Constituency Parsing [11.080040070201608]
  This work proposes a method for joint Chinese word segmentation and Span-based Constituency Parsing by adding extra labels to individual Chinese characters on the parse trees. Through experiments, the proposed algorithm outperforms the recent models for joint segmentation and constituency parsing on CTB 5.1.
  arXiv  Detail & Related papers  (2022-11-03T08:19:00Z)
• GCDT: A Chinese RST Treebank for Multigenre and Multilingual Discourse Parsing [9.367612782346207]
  GCDT is the largest hierarchical discourse treebank for Mandarin Chinese in the framework of Rhetorical Structure Theory (RST) We report on this dataset's parsing experiments, including state-of-the-art (SOTA) scores for Chinese RST parsing and RST parsing on the English GUM dataset.
  arXiv  Detail & Related papers  (2022-10-19T10:27:41Z)
• ZiGAN: Fine-grained Chinese Calligraphy Font Generation via a Few-shot Style Transfer Approach [7.318027179922774]
  ZiGAN is a powerful end-to-end Chinese calligraphy font generation framework. It does not require any manual operation or redundant preprocessing to generate fine-grained target-style characters. Our method has a state-of-the-art generalization ability in few-shot Chinese character style transfer.
  arXiv  Detail & Related papers  (2021-08-08T09:50:20Z)
• SHUOWEN-JIEZI: Linguistically Informed Tokenizers For Chinese Language Model Pretraining [48.880840711568425]
  We study the influences of three main factors on the Chinese tokenization for pretrained language models. We propose three kinds of tokenizers: SHUOWEN (meaning Talk Word), the pronunciation-based tokenizers; 2) JIEZI (meaning Solve Character), the glyph-based tokenizers. We find that SHUOWEN and JIEZI tokenizers can generally outperform conventional single-character tokenizers.
  arXiv  Detail & Related papers  (2021-06-01T11:20:02Z)
• 2kenize: Tying Subword Sequences for Chinese Script Conversion [54.33749520569979]
  We propose a model that can disambiguate between mappings and convert between the two scripts. Our proposed method outperforms previous Chinese Character conversion approaches by 6 points in accuracy.
  arXiv  Detail & Related papers  (2020-05-07T10:53:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.