ParsiPy: NLP Toolkit for Historical Persian Texts in Python

• URL: http://arxiv.org/abs/2503.17810v1
• Date: Sat, 22 Mar 2025 16:21:29 GMT
• Title: ParsiPy: NLP Toolkit for Historical Persian Texts in Python
• Authors: Farhan Farsi, Parnian Fazel, Sepand Haghighi, Sadra Sabouri, Farzaneh Goshtasb, Nadia Hajipour, Ehsaneddin Asgari, Hossein Sameti,
• Abstract summary: This work introduces ParsiPy, an NLP toolkit to handle phonetic transcriptions and analyze ancient texts.<n>ParsiPy offers modules for tokenization, lemmatization, part-of-speech tagging, phoneme-to-transliteration conversion, and word embedding.
• Score: 1.637832760977605
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The study of historical languages presents unique challenges due to their complex orthographic systems, fragmentary textual evidence, and the absence of standardized digital representations of text in those languages. Tackling these challenges needs special NLP digital tools to handle phonetic transcriptions and analyze ancient texts. This work introduces ParsiPy, an NLP toolkit designed to facilitate the analysis of historical Persian languages by offering modules for tokenization, lemmatization, part-of-speech tagging, phoneme-to-transliteration conversion, and word embedding. We demonstrate the utility of our toolkit through the processing of Parsig (Middle Persian) texts, highlighting its potential for expanding computational methods in the study of historical languages. Through this work, we contribute to computational philology, offering tools that can be adapted for the broader study of ancient texts and their digital preservation.

Related papers

• Adapting Multilingual Embedding Models to Historical Luxembourgish [5.474797258314828]
  This study examines multilingual embeddings for cross-lingual semantic search in historical Luxembourgish.<n>We use GPT-4o for sentence segmentation and translation, generating 20,000 parallel training sentences per language pair.<n>We adapt several multilingual embedding models through contrastive learning or knowledge distillation and increase accuracy significantly for all models.
  arXiv  Detail & Related papers  (2025-02-11T20:35:29Z)
• LogogramNLP: Comparing Visual and Textual Representations of Ancient Logographic Writing Systems for NLP [30.804518354947565]
  A large portion of logographic data persists in a purely visual form due to the absence of transcription. This issue poses a bottleneck for researchers seeking to apply NLP toolkits to study ancient logographic languages. We introduce LogogramNLP, the first benchmark enabling NLP analysis of ancient logographic languages.
  arXiv  Detail & Related papers  (2024-08-08T17:58:06Z)
• T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Text [59.57676466961787]
  We propose a novel dynamic vector quantization (DVA-VAE) model that can adjust the encoding length based on the information density in sign language. Experiments conducted on the PHOENIX14T dataset demonstrate the effectiveness of our proposed method. We propose a new large German sign language dataset, PHOENIX-News, which contains 486 hours of sign language videos, audio, and transcription texts.
  arXiv  Detail & Related papers  (2024-06-11T10:06:53Z)
• GujiBERT and GujiGPT: Construction of Intelligent Information Processing Foundation Language Models for Ancient Texts [11.289265479095956]
  GujiBERT and GujiGPT language models are foundational models specifically designed for intelligent information processing of ancient texts. These models have been trained on an extensive dataset that encompasses both simplified and traditional Chinese characters. These models have exhibited exceptional performance across a range of validation tasks using publicly available datasets.
  arXiv  Detail & Related papers  (2023-07-11T15:44:01Z)
• Lexical Complexity Prediction: An Overview [13.224233182417636]
  The occurrence of unknown words in texts significantly hinders reading comprehension. computational modelling has been applied to identify complex words in texts and substitute them for simpler alternatives. We present an overview of computational approaches to lexical complexity prediction focusing on the work carried out on English data.
  arXiv  Detail & Related papers  (2023-03-08T19:35:08Z)
• Beyond Arabic: Software for Perso-Arabic Script Manipulation [67.31374614549237]
  We provide a set of finite-state transducer (FST) components and corresponding utilities for manipulating the writing systems of languages that use the Perso-Arabic script. The library also provides simple FST-based romanization and transliteration.
  arXiv  Detail & Related papers  (2023-01-26T20:37:03Z)
• textless-lib: a Library for Textless Spoken Language Processing [50.070693765984075]
  We introduce textless-lib, a PyTorch-based library aimed to facilitate research in this research area. We describe the building blocks that the library provides and demonstrate its usability.
  arXiv  Detail & Related papers  (2022-02-15T12:39:42Z)
• Strategies of Effective Digitization of Commentaries and Sub-commentaries: Towards the Construction of Textual History [26.355399011710944]
  We use the text of the K=a'sik=avrtti ( KV) as a sample text, and with the help of philologists, we digitize the commentaries available to us. We divide each commentary and sub-commentary into functional units and describe the methodology and motivation behind the functional unit division.
  arXiv  Detail & Related papers  (2022-01-05T16:43:43Z)
• Digital Editions as Distant Supervision for Layout Analysis of Printed Books [76.29918490722902]
  We describe methods for exploiting this semantic markup as distant supervision for training and evaluating layout analysis models. In experiments with several model architectures on the half-million pages of the Deutsches Textarchiv (DTA), we find a high correlation of these region-level evaluation methods with pixel-level and word-level metrics. We discuss the possibilities for improving accuracy with self-training and the ability of models trained on the DTA to generalize to other historical printed books.
  arXiv  Detail & Related papers  (2021-12-23T16:51:53Z)
• Latin writing styles analysis with Machine Learning: New approach to old questions [0.0]
  In the Middle Ages texts were learned by heart and spread using oral means of communication from generation to generation. Taking into account such a specific construction of literature composed in Latin, we can search for and indicate the probability patterns of familiar sources of specific narrative texts.
  arXiv  Detail & Related papers  (2021-09-01T20:21:45Z)
• Enabling Language Models to Fill in the Blanks [81.59381915581892]
  We present a simple approach for text infilling, the task of predicting missing spans of text at any position in a document. We train (or fine-tune) off-the-shelf language models on sequences containing the concatenation of artificially-masked text and the text which was masked. We show that this approach, which we call infilling by language modeling, can enable LMs to infill entire sentences effectively on three different domains: short stories, scientific abstracts, and lyrics.
  arXiv  Detail & Related papers  (2020-05-11T18:00:03Z)
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer [64.22926988297685]
  Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP) In this paper, we explore the landscape of introducing transfer learning techniques for NLP by a unified framework that converts all text-based language problems into a text-to-text format.
  arXiv  Detail & Related papers  (2019-10-23T17:37:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.