GlotScript: A Resource and Tool for Low Resource Writing System Identification

• URL: http://arxiv.org/abs/2309.13320v2
• Date: Wed, 27 Mar 2024 14:57:29 GMT
• Title: GlotScript: A Resource and Tool for Low Resource Writing System Identification
• Abstract summary: GlotScript is an open resource for low resource writing system identification. GlotScript-R provides attested writing systems for more than 7,000 languages. GlotScript-T is a writing system identification tool that covers all 161 Unicode 15.0 scripts.
• Score: 53.56700754408902
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We present GlotScript, an open resource and tool for low resource writing system identification. GlotScript-R is a resource that provides the attested writing systems for more than 7,000 languages. It is compiled by aggregating information from existing writing system resources. GlotScript-T is a writing system identification tool that covers all 161 Unicode 15.0 scripts. For an input text, it returns its script distribution where scripts are identified by ISO 15924 codes. We also present two use cases for GlotScript. First, we demonstrate that GlotScript can help cleaning multilingual corpora such as mC4 and OSCAR. Second, we analyze the tokenization of a number of language models such as GPT-4 using GlotScript and provide insights on the coverage of low resource scripts and languages by each language model. We hope that GlotScript will become a useful resource for work on low resource languages in the NLP community. GlotScript-R and GlotScript-T are available at https://github.com/cisnlp/GlotScript.

Related papers

• Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation [0.27998963147546146]
  We introduce PuMVR, a benchmark of 1,000 strictly parallel image-text instances across Punjabi's three active scripts.<n>Models frequently solve visual tasks in one script while failing identical tasks in another, with accuracy deltas reaching 16%.<n>Cross-script in-context transfer is highly brittle, exposing script-locked knowledge representation.
  arXiv  Detail & Related papers  (2026-06-15T18:25:23Z)
• GlotOCR Bench: OCR Models Still Struggle Beyond a Handful of Unicode Scripts [58.92151016423978]
  We introduce GlotOCR Bench, a comprehensive benchmark evaluating OCR across 100+ Unicode scripts.<n>Our benchmark comprises clean and degraded image variants rendered from real multilingual texts.<n>We evaluate a broad suite of open-weight and proprietary vision-language models and find that most perform well on fewer than ten scripts.
  arXiv  Detail & Related papers  (2026-04-14T17:12:41Z)
• Linear Script Representations in Speech Foundation Models Enable Zero-Shot Transliteration [70.84108518476744]
  We show that script is linearly encoded in the activation space of multilingual speech models, and that modifying activations at inference time enables direct control over output script.<n>We apply this approach to inducing post-hoc control over the script of speech recognition output, where we observe competitive performance across all model sizes of Whisper.
  arXiv  Detail & Related papers  (2026-01-06T10:45:04Z)
• LangSAMP: Language-Script Aware Multilingual Pretraining [48.16511046793275]
  We propose Language-Script Aware Multilingual Pretraining (LangSAMP)<n>LangSAMP incorporates both language and script embeddings to enhance representation learning.<n>We apply LangSAMP to the continual pretraining of XLM-R on a highly multilingual corpus covering more than 500 languages.
  arXiv  Detail & Related papers  (2024-09-26T18:29:10Z)
• Exploring the Role of Transliteration in In-Context Learning for Low-resource Languages Written in Non-Latin Scripts [50.40191599304911]
  We investigate whether transliteration is also effective in improving LLMs' performance for low-resource languages written in non-Latin scripts. We propose three prompt templates, where the target-language text is represented in (1) its original script, (2) Latin script, or (3) both. Our findings show that the effectiveness of transliteration varies by task type and model size.
  arXiv  Detail & Related papers  (2024-07-02T14:51:20Z)
• Script-Agnostic Language Identification [21.19710835737713]
  Many modern languages, such as Konkani, Kashmiri, Punjabi etc., are synchronically written in several scripts. We propose learning script-agnostic representations using several different experimental strategies. We find that word-level script randomization and exposure to a language written in multiple scripts is extremely valuable for downstream script-agnostic language identification.
  arXiv  Detail & Related papers  (2024-06-25T19:23:42Z)
• TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models [50.40191599304911]
  We propose TransliCo to fine-tune an mPLM by contrasting sentences in its training data and their transliterations in a unified script. We show that Furina outperforms the original Glot500-m on various zero-shot crosslingual transfer tasks.
  arXiv  Detail & Related papers  (2024-01-12T15:12:48Z)
• GlotLID: Language Identification for Low-Resource Languages [51.38634652914054]
  GlotLID-M is an LID model that satisfies the desiderata of wide coverage, reliability and efficiency. It identifies 1665 languages, a large increase in coverage compared to prior work.
  arXiv  Detail & Related papers  (2023-10-24T23:45:57Z)
• Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whisper [96.43501666278316]
  This paper proposes a powerful Visual Speech Recognition (VSR) method for multiple languages. We employ a Whisper model which can conduct both language identification and audio-based speech recognition. By comparing the performances of VSR models trained on automatic labels and the human-annotated labels, we show that we can achieve similar VSR performance to that of human-annotated labels.
  arXiv  Detail & Related papers  (2023-09-15T16:53:01Z)
• GAS-Net: Generative Artistic Style Neural Networks for Fonts [8.569974263629218]
  This project aims to develop a few-shot cross-lingual font generator based on AGIS-Net. Our approaches include redesigning the encoder and the loss function. We will validate our method on multiple languages and datasets mentioned.
  arXiv  Detail & Related papers  (2022-12-06T11:23:16Z)
• UNKs Everywhere: Adapting Multilingual Language Models to New Scripts [103.79021395138423]
  Massively multilingual language models such as multilingual BERT (mBERT) and XLM-R offer state-of-the-art cross-lingual transfer performance on a range of NLP tasks. Due to their limited capacity and large differences in pretraining data, there is a profound performance gap between resource-rich and resource-poor target languages. We propose novel data-efficient methods that enable quick and effective adaptation of pretrained multilingual models to such low-resource languages and unseen scripts.
  arXiv  Detail & Related papers  (2020-12-31T11:37:28Z)
• Handwritten Script Identification from Text Lines [38.1188690493442]
  We propose a robust method towards identifying scripts from handwritten documents at text line-level. The recognition is based upon features extracted using Chain Code Histogram (CCH) and Discrete Fourier Transform (DFT) The proposed method is experimented on 800 handwritten text lines written in seven Indic scripts namely, Gujarati, Kannada, Malayalam, Oriya, Tamil, Telugu, Urdu along with Roman script.
  arXiv  Detail & Related papers  (2020-09-16T02:43:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.