SiPaKosa: A Comprehensive Corpus of Canonical and Classical Buddhist Texts in Sinhala and Pali

• URL: http://arxiv.org/abs/2603.29221v1
• Date: Tue, 31 Mar 2026 03:36:46 GMT
• Title: SiPaKosa: A Comprehensive Corpus of Canonical and Classical Buddhist Texts in Sinhala and Pali
• Abstract summary: SiPaKosa is a comprehensive corpus of Sinhala and Pali doctrinal texts comprising approximately 786K sentences and 9.25M words.<n>The corpus was created through high-quality OCR using Google Document AI on historical manuscripts.<n>We evaluate the performance of language models using ten pretrained models, with perplexity scores ranging from 1.09 to 189.67 on our corpus.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: SiPaKosa is a comprehensive corpus of Sinhala and Pali doctrinal texts comprising approximately 786K sentences and 9.25M words, incorporating 16 copyright-cleared historical Buddhist documents alongside the complete web-scraped Tripitaka canonical texts. The corpus was created through high-quality OCR using Google Document AI on historical manuscripts, combined with systematic web scraping of canonical repositories, followed by rigorous quality control and metadata annotation. The corpus is organised into language-specific subcorpora: Sinhala and Mixed Sinhala-Pali. We evaluate the performance of language models using ten pretrained models, with perplexity scores ranging from 1.09 to 189.67 on our corpus. This analysis shows that proprietary models significantly outperform open-source alternatives by factors of three to six times. This corpus supports the pretraining of domain-adapted language models, facilitates historical language analysis, and aids in the development of information retrieval systems for Buddhist scholarship while preserving Sinhala cultural heritage.

Related papers

• MonkeyOCRv2: A Visual-Text Foundation Model for Document AI [71.06391669976786]
  MonkeyOCRv2 is a visual-text pretrained model for document AI.<n> MonkeyDoc v2 is the largest document-image pretraining corpus, comprising 113 million images spanning 17 languages.
  arXiv  Detail & Related papers  (2026-07-13T13:43:39Z)
• SiDiaC: Sinhala Diachronic Corpus [1.256381443503838]
  SiDiaC is the first comprehensive Sinhala Diachronic Corpus, covering a historical span from the 5th to the 20th century CE.<n>SiDiaC comprises 58k words across 46 literary works, annotated carefully based on the written date, after filtering based on availability, authorship, copyright compliance, and data attribution.
  arXiv  Detail & Related papers  (2025-09-22T15:37:51Z)
• Skywork: A More Open Bilingual Foundation Model [55.927396986873816]
  We present Skywork-13B, a family of large language models (LLMs) trained on a corpus of over 3.2 trillion tokens drawn from both English and Chinese texts. We show that our model not only excels on popular benchmarks, but also achieves emphstate of the art performance in Chinese language modeling on diverse domains.
  arXiv  Detail & Related papers  (2023-10-30T08:31:47Z)
• A Corpus for Sentence-level Subjectivity Detection on English News Articles [49.49218203204942]
  We use our guidelines to collect NewsSD-ENG, a corpus of 638 objective and 411 subjective sentences extracted from English news articles on controversial topics. Our corpus paves the way for subjectivity detection in English without relying on language-specific tools, such as lexicons or machine translation.
  arXiv  Detail & Related papers  (2023-05-29T11:54:50Z)
• The Open corpus of the Veps and Karelian languages: overview and applications [52.77024349608834]
  The Open Corpus of the Veps and Karelian Languages (VepKar) is an extension of the Veps created in 2009. The VepKar corpus comprises texts in Karelian and Veps, multifunctional dictionaries linked to them, and software with an advanced system of search. Future plans include developing a speech module for working with audio recordings and a syntactic tagging module using morphological analysis outputs.
  arXiv  Detail & Related papers  (2022-06-08T13:05:50Z)
• Lahjoita puhetta -- a large-scale corpus of spoken Finnish with some benchmarks [9.160401226886947]
  The Donate Speech campaign has so far succeeded in gathering approximately 3600 hours of ordinary, colloquial Finnish speech. The primary goals of the collection were to create a representative, large-scale resource to study spontaneous spoken Finnish and to accelerate the development of language technology and speech-based services. We present the collection process and the collected corpus, and showcase its versatility through multiple use cases.
  arXiv  Detail & Related papers  (2022-03-24T07:50:25Z)
• Models and Datasets for Cross-Lingual Summarisation [78.56238251185214]
  We present a cross-lingual summarisation corpus with long documents in a source language associated with multi-sentence summaries in a target language. The corpus covers twelve language pairs and directions for four European languages, namely Czech, English, French and German. We derive cross-lingual document-summary instances from Wikipedia by combining lead paragraphs and articles' bodies from language aligned Wikipedia titles.
  arXiv  Detail & Related papers  (2022-02-19T11:55:40Z)
• A Novel Corpus of Discourse Structure in Humans and Computers [55.74664144248097]
  We present a novel corpus of 445 human- and computer-generated documents, comprising about 27,000 clauses. The corpus covers both formal and informal discourse, and contains documents generated using fine-tuned GPT-2.
  arXiv  Detail & Related papers  (2021-11-10T20:56:08Z)
• HmBlogs: A big general Persian corpus [0.0]
  This paper introduces the hmBlogs corpus for Persian, as a low resource language. This corpus has been prepared based on a collection of nearly 20 million blog posts over a period of about 15 years from a space of Persian blogs. It can be claimed that this corpus is currently the largest Persian corpus that has been prepared independently for the Persian language.
  arXiv  Detail & Related papers  (2021-11-03T17:26:52Z)
• Validation and Normalization of DCS corpus using Sanskrit Heritage tools to build a tagged Gold Corpus [0.0]
  The Digital Corpus of Sanskrit records around 650,000 sentences along with their morphological and lexical tagging. The Sanskrit Heritage Engine's Reader produces all possible segmentations with morphological and lexical analyses.
  arXiv  Detail & Related papers  (2020-05-13T19:23:43Z)
• Know thy corpus! Robust methods for digital curation of Web corpora [0.0]
  This paper proposes a novel framework for digital curation of Web corpora. It provides robust estimation of their parameters, such as their composition and the lexicon.
  arXiv  Detail & Related papers  (2020-03-13T17:21:57Z)
• CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpus [57.641761472372814]
  CoVoST is a multilingual speech-to-text translation corpus from 11 languages into English. It diversified with over 11,000 speakers and over 60 accents. CoVoST is released under CC0 license and free to use.
  arXiv  Detail & Related papers  (2020-02-04T14:35:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.