Related papers: SiDiaC-v.2.0: Sinhala Diachronic Corpus Version 2.0

SiDiaC-v.2.0: Sinhala Diachronic Corpus Version 2.0

URL: http://arxiv.org/abs/2603.10861v1
Date: Wed, 11 Mar 2026 15:10:32 GMT
Title: SiDiaC-v.2.0: Sinhala Diachronic Corpus Version 2.0
Abstract summary: SiDiaC-v.2.0 is the largest comprehensive Sinhala Diachronic to date, covering a period from 1800 CE to 1955 CE.<n>The corpus consists of 244k words across 185 literary works that underwent thorough filtering, preprocessing, and copyright compliance checks.<n>A subset of 59 documents totalling 70k words was annotated based on their written dates.
Score: 0.9001507266214666
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: SiDiaC-v.2.0 is the largest comprehensive Sinhala Diachronic Corpus to date, covering a period from 1800 CE to 1955 CE in terms of publication dates, and a historical span from the 5th to the 20th century CE in terms of written dates. The corpus consists of 244k words across 185 literary works that underwent thorough filtering, preprocessing, and copyright compliance checks, followed by extensive post-processing. Additionally, a subset of 59 documents totalling 70k words was annotated based on their written dates. Texts from the National Library of Sri Lanka were selected from the SiDiaC-v.1.0 non-filtered list, which was digitised using Google Document AI OCR. This was followed by post-processing to correct formatting issues, address code-mixing, include special tokens, and fix malformed tokens. The construction of SiDiaC-v.2.0 was informed by practices from other corpora, such as FarPaHC, SiDiaC-v.1.0, and CCOHA. This was particularly relevant for syntactic annotation and text normalisation strategies, given the shared characteristics of low-resource language status between Faroese and the similar cleaning strategies utilised in CCOHA. This corpus is categorised into two layers based on genres: primary and secondary. The primary categorisation is binary, assigning each book to either Non-Fiction or Fiction. The secondary categorisation is more detailed, grouping texts under specific genres such as Religious, History, Poetry, Language, and Medical. Despite facing challenges due to limited resources, SiDiaC-v.2.0 serves as a comprehensive resource for Sinhala NLP, building upon the work previously done in SiDiaC-v.1.0.

Related papers

Context-Aware Distillation and Ablation for Text2DSL [78.29352305480285]
We replace prompt-only synthetic generation with context-aware distillation.<n>A teacher large language model operates under an explicitly defined structured context.<n>This scales the verified PolkitBench corpus from 4,204 to 10,073 pairs.
arXiv Detail & Related papers (2026-06-21T16:27:24Z)
Explainable Disentangled Representation Learning for Generalizable Authorship Attribution in the Era of Generative AI [70.05655131235999]
Explainable Authorship Variational Autoencoder (EAVAE) is a novel framework that explicitly disentangles style from content.<n>Disentanglement is enforced through a novel discriminator.<n>EAVAE excels in few-shot learning over the M4 dataset for AI-generated text detection.
arXiv Detail & Related papers (2026-04-23T05:34:20Z)
SiPaKosa: A Comprehensive Corpus of Canonical and Classical Buddhist Texts in Sinhala and Pali [0.0]
SiPaKosa is a comprehensive corpus of Sinhala and Pali doctrinal texts comprising approximately 786K sentences and 9.25M words.<n>The corpus was created through high-quality OCR using Google Document AI on historical manuscripts.<n>We evaluate the performance of language models using ten pretrained models, with perplexity scores ranging from 1.09 to 189.67 on our corpus.
arXiv Detail & Related papers (2026-03-31T03:36:46Z)
Cross-Lingual SynthDocs: A Large-Scale Synthetic Corpus for Any to Arabic OCR and Document Understanding [3.587092806938212]
Cross-Lingual SynthDocs is a large-scale synthetic corpus designed to address the scarcity of Arabic resources for Optical Character Recognition (OCR) and Document Understanding (DU)<n>The dataset comprises over 2.5 million samples, including 1.5 million textual data, 270K fully annotated tables, and hundred thousands of real data based charts.
arXiv Detail & Related papers (2025-11-01T04:54:58Z)
SiDiaC: Sinhala Diachronic Corpus [1.256381443503838]
SiDiaC is the first comprehensive Sinhala Diachronic Corpus, covering a historical span from the 5th to the 20th century CE.<n>SiDiaC comprises 58k words across 46 literary works, annotated carefully based on the written date, after filtering based on availability, authorship, copyright compliance, and data attribution.
arXiv Detail & Related papers (2025-09-22T15:37:51Z)
Scaling Rich Style-Prompted Text-to-Speech Datasets [54.50850884210504]
Paralinguistic Speech Captions (ParaSpeechCaps) is a large-scale dataset that annotates speech utterances with rich style captions.<n>ParaSpeechCaps covers a total of 59 style tags, including both speaker-level intrinsic tags and utterance-level situational tags.<n>It consists of 342 hours of human-labelled data and 2427 hours of automatically annotated data.
arXiv Detail & Related papers (2025-03-06T18:57:40Z)
Discourse Features Enhance Detection of Document-Level Machine-Generated Content [53.41994768824785]
Machine-generated content poses challenges such as academic plagiarism and the spread of misinformation.<n>Existing MGC detectors often focus solely on surface-level information, overlooking implicit and structural features.<n>We introduce novel methodologies and datasets to overcome these challenges.
arXiv Detail & Related papers (2024-12-17T08:47:41Z)
Making Old Kurdish Publications Processable by Augmenting Available Optical Character Recognition Engines [1.174020933567308]
Kurdish libraries have many historical publications that were printed back in the early days when printing devices were brought to Kurdistan. Current Optical Character Recognition (OCR) systems are unable to extract text from historical documents as they have many issues. In this study, we adopt an open-source OCR framework by Google, Tesseract version 5.0, that has been used to extract text for various languages.
arXiv Detail & Related papers (2024-04-09T08:08:03Z)
Transformer-based Model for Word Level Language Identification in Code-mixed Kannada-English Texts [55.41644538483948]
We propose the use of a Transformer based model for word-level language identification in code-mixed Kannada English texts. The proposed model on the CoLI-Kenglish dataset achieves a weighted F1-score of 0.84 and a macro F1-score of 0.61.
arXiv Detail & Related papers (2022-11-26T02:39:19Z)
Combining Morphological and Histogram based Text Line Segmentation in the OCR Context [0.0]
Algorithmic approach proposed by this paper has been designed for this exact purpose. The method was developed to be applied on a historic data collection that commonly features quality issues. Because of the promising segmentation results that are joined by low computational cost, the algorithm was incorporated into the OCR pipeline of the National Library of Luxembourg.
arXiv Detail & Related papers (2021-03-16T09:06:25Z)
Abstractive Summarization of Spoken and Written Instructions with BERT [66.14755043607776]
We present the first application of the BERTSum model to conversational language. We generate abstractive summaries of narrated instructional videos across a wide variety of topics. We envision this integrated as a feature in intelligent virtual assistants, enabling them to summarize both written and spoken instructional content upon request.
arXiv Detail & Related papers (2020-08-21T20:59:34Z)
Announcing CzEng 2.0 Parallel Corpus with over 2 Gigawords [12.561914014993675]
We present a new release of the Czech-English parallel corpus CzEng 2.0 consisting of over 2 billion words (2 "gigawords") in each language. The corpus contains document-level information and is filtered with several techniques to lower the amount of noise.
arXiv Detail & Related papers (2020-07-06T18:48:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.