Unicode Normalization and Grapheme Parsing of Indic Languages

• URL: http://arxiv.org/abs/2306.01743v2
• Date: Mon, 27 May 2024 12:48:00 GMT
• Title: Unicode Normalization and Grapheme Parsing of Indic Languages
• Authors: Nazmuddoha Ansary, Quazi Adibur Rahman Adib, Tahsin Reasat, Asif Shahriyar Sushmit, Ahmed Imtiaz Humayun, Sazia Mehnaz, Kanij Fatema, Mohammad Mamun Or Rashid, Farig Sadeque,
• Abstract summary: Writing systems of Indic languages have orthographic syllables, also known as complex graphemes, as unique horizontal units. Our proposed normalizer is a more efficient and effective tool than the previously used Indic normalizer. We report the pipeline for the scripts of 7 languages in this work and develop the framework for the integration of more scripts.
• Score: 2.974799610163104
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Writing systems of Indic languages have orthographic syllables, also known as complex graphemes, as unique horizontal units. A prominent feature of these languages is these complex grapheme units that comprise consonants/consonant conjuncts, vowel diacritics, and consonant diacritics, which, together make a unique Language. Unicode-based writing schemes of these languages often disregard this feature of these languages and encode words as linear sequences of Unicode characters using an intricate scheme of connector characters and font interpreters. Due to this way of using a few dozen Unicode glyphs to write thousands of different unique glyphs (complex graphemes), there are serious ambiguities that lead to malformed words. In this paper, we are proposing two libraries: i) a normalizer for normalizing inconsistencies caused by a Unicode-based encoding scheme for Indic languages and ii) a grapheme parser for Abugida text. It deconstructs words into visually distinct orthographic syllables or complex graphemes and their constituents. Our proposed normalizer is a more efficient and effective tool than the previously used IndicNLP normalizer. Moreover, our parser and normalizer are also suitable tools for general Abugida text processing as they performed well in our robust word-based and NLP experiments. We report the pipeline for the scripts of 7 languages in this work and develop the framework for the integration of more scripts.

Related papers

• Egalitarian Language Representation in Language Models: It All Begins with Tokenizers [0.0]
  We show that not all tokenizers offer fair representation for complex script languages such as Tamil, Sinhala, and Hindi. We introduce an improvement to the Byte Pair algorithm by incorporating graphemes, which we term Grapheme Pair. Our experiments show that grapheme-based character extraction outperforms byte-level tokenizers for complex scripts.
  arXiv  Detail & Related papers  (2024-09-17T19:05:37Z)
• T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Text [59.57676466961787]
  We propose a novel dynamic vector quantization (DVA-VAE) model that can adjust the encoding length based on the information density in sign language. Experiments conducted on the PHOENIX14T dataset demonstrate the effectiveness of our proposed method. We propose a new large German sign language dataset, PHOENIX-News, which contains 486 hours of sign language videos, audio, and transcription texts.
  arXiv  Detail & Related papers  (2024-06-11T10:06:53Z)
• HierCode: A Lightweight Hierarchical Codebook for Zero-shot Chinese Text Recognition [47.86479271322264]
  We propose HierCode, a novel and lightweight codebook that exploits the innate hierarchical nature of Chinese characters. HierCode employs a multi-hot encoding strategy, leveraging hierarchical binary tree encoding and prototype learning to create distinctive, informative representations for each character. This approach not only facilitates zero-shot recognition of OOV characters by utilizing shared radicals and structures but also excels in line-level recognition tasks by computing similarity with visual features.
  arXiv  Detail & Related papers  (2024-03-20T17:20:48Z)
• MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling [70.34758460372629]
  We introduce a new paradigm that encodes the same information with segments of consistent size across diverse languages. MYTE produces shorter encodings for all 99 analyzed languages. This, in turn, improves multilingual LM performance and diminishes the perplexity gap throughout diverse languages.
  arXiv  Detail & Related papers  (2024-03-15T21:21:11Z)
• Machine Translation by Projecting Text into the Same Phonetic-Orthographic Space Using a Common Encoding [3.0422770070015295]
  We propose an approach based on common multilingual Latin-based encodings (WX notation) that take advantage of language similarity. We verify the proposed approach by demonstrating experiments on similar language pairs. We also get up to 1 BLEU points improvement on distant and zero-shot language pairs.
  arXiv  Detail & Related papers  (2023-05-21T06:46:33Z)
• Beyond Arabic: Software for Perso-Arabic Script Manipulation [67.31374614549237]
  We provide a set of finite-state transducer (FST) components and corresponding utilities for manipulating the writing systems of languages that use the Perso-Arabic script. The library also provides simple FST-based romanization and transliteration.
  arXiv  Detail & Related papers  (2023-01-26T20:37:03Z)
• Graphemic Normalization of the Perso-Arabic Script [47.429213930688086]
  This paper documents the challenges that Perso-Arabic presents beyond the best-documented languages. We focus on the situation in natural language processing (NLP), which is affected by multiple, often neglected, issues. We evaluate the effects of script normalization on eight languages from diverse language families in the Perso-Arabic script diaspora on machine translation and statistical language modeling tasks.
  arXiv  Detail & Related papers  (2022-10-21T21:59:44Z)
• edATLAS: An Efficient Disambiguation Algorithm for Texting in Languages with Abugida Scripts [0.0]
  Abugida refers to a phonogram writing system where each syllable is represented using a single consonant or typographic ligature. We propose a disambiguation algorithm and showcase its usefulness in two novel input methods for languages using the abugida writing system. We show an improvement in typing speed by 19.49%, 25.13%, and 14.89%, in Hindi, Bengali, and Thai, respectively, using Ambiguous Input.
  arXiv  Detail & Related papers  (2021-01-05T03:16:34Z)
• Automatic Extraction of Rules Governing Morphological Agreement [103.78033184221373]
  We develop an automated framework for extracting a first-pass grammatical specification from raw text. We focus on extracting rules describing agreement, a morphosyntactic phenomenon at the core of the grammars of many of the world's languages. We apply our framework to all languages included in the Universal Dependencies project, with promising results.
  arXiv  Detail & Related papers  (2020-10-02T18:31:45Z)
• A Large Multi-Target Dataset of Common Bengali Handwritten Graphemes [1.009810782568186]
  We propose a labeling scheme that makes segmentation in-side alpha-syllabary words linear. The dataset contains 411k curated samples of 1295 unique commonly used Bengali graphemes. The dataset is open-sourced as a part of a public Handwritten Grapheme Classification Challenge on Kaggle.
  arXiv  Detail & Related papers  (2020-10-01T01:51:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.