SiDiaC: Sinhala Diachronic Corpus

• URL: http://arxiv.org/abs/2509.17912v1
• Date: Mon, 22 Sep 2025 15:37:51 GMT
• Title: SiDiaC: Sinhala Diachronic Corpus
• Abstract summary: SiDiaC is the first comprehensive Sinhala Diachronic Corpus, covering a historical span from the 5th to the 20th century CE.<n>SiDiaC comprises 58k words across 46 literary works, annotated carefully based on the written date, after filtering based on availability, authorship, copyright compliance, and data attribution.
• Score: 1.256381443503838
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: SiDiaC, the first comprehensive Sinhala Diachronic Corpus, covers a historical span from the 5th to the 20th century CE. SiDiaC comprises 58k words across 46 literary works, annotated carefully based on the written date, after filtering based on availability, authorship, copyright compliance, and data attribution. Texts from the National Library of Sri Lanka were digitised using Google Document AI OCR, followed by post-processing to correct formatting and modernise the orthography. The construction of SiDiaC was informed by practices from other corpora, such as FarPaHC, particularly in syntactic annotation and text normalisation strategies, due to the shared characteristics of low-resourced language status. This corpus is categorised based on genres into two layers: primary and secondary. Primary categorisation is binary, classifying each book into Non-Fiction or Fiction, while the secondary categorisation is more specific, grouping texts under Religious, History, Poetry, Language, and Medical genres. Despite challenges including limited access to rare texts and reliance on secondary date sources, SiDiaC serves as a foundational resource for Sinhala NLP, significantly extending the resources available for Sinhala, enabling diachronic studies in lexical change, neologism tracking, historical syntax, and corpus-based lexicography.

Related papers

• Does Syntax Matter? A Graph-Augmented Variational Topic Model for Computational Social Sciences [51.56484100374058]
  This paper introduces the Structural Contextual Probabilistic Topic Model ( SCPTM), an architecture that incorporates syntactic dependency relations into topic inference.<n>We evaluate seven topic modeling techniques across four corpora differing in register and discourse structure.<n>Results show that SCPTM's neural architecture yields substantial gains in document-topic alignment over generative baselines.
  arXiv  Detail & Related papers  (2026-09-07T17:39:06Z)
• Koshur Pixel: a large-scale synthetic ocr dataset for kashmiri [0.0]
  We introduce Koshur Pixel, the first large-scale synthetic OCR dataset for Kashmiri.<n>Koshur Pixel comprises 613,078 image-text pairs generated from the KS-PRET-5M corpus using the SynthOCR-Gen framework.
  arXiv  Detail & Related papers  (2026-06-22T10:42:36Z)
• Explainable Disentangled Representation Learning for Generalizable Authorship Attribution in the Era of Generative AI [70.05655131235999]
  Explainable Authorship Variational Autoencoder (EAVAE) is a novel framework that explicitly disentangles style from content.<n>Disentanglement is enforced through a novel discriminator.<n>EAVAE excels in few-shot learning over the M4 dataset for AI-generated text detection.
  arXiv  Detail & Related papers  (2026-04-23T05:34:20Z)
• SiPaKosa: A Comprehensive Corpus of Canonical and Classical Buddhist Texts in Sinhala and Pali [0.0]
  SiPaKosa is a comprehensive corpus of Sinhala and Pali doctrinal texts comprising approximately 786K sentences and 9.25M words.<n>The corpus was created through high-quality OCR using Google Document AI on historical manuscripts.<n>We evaluate the performance of language models using ten pretrained models, with perplexity scores ranging from 1.09 to 189.67 on our corpus.
  arXiv  Detail & Related papers  (2026-03-31T03:36:46Z)
• SiDiaC-v.2.0: Sinhala Diachronic Corpus Version 2.0 [0.9001507266214666]
  SiDiaC-v.2.0 is the largest comprehensive Sinhala Diachronic to date, covering a period from 1800 CE to 1955 CE.<n>The corpus consists of 244k words across 185 literary works that underwent thorough filtering, preprocessing, and copyright compliance checks.<n>A subset of 59 documents totalling 70k words was annotated based on their written dates.
  arXiv  Detail & Related papers  (2026-03-11T15:10:32Z)
• Loci Similes: A Benchmark for Extracting Intertextualities in Latin Literature [4.132158161225706]
  Loci Similes is a benchmark for Latin intertextuality detection comprising of a curated dataset of 172k text segments containing 545 expert-verified parallels linking Late Antique authors to a corpus of classical authors.<n>We establish baselines for retrieval and classification of intertextualities with state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2026-01-12T13:34:49Z)
• Multilingual corpora for the study of new concepts in the social sciences and humanities: [0.0]
  This article presents a hybrid methodology for building a multilingual corpus designed to support the study of emerging concepts in the humanities and social sciences.<n>The corpus relies on two complementary sources: (1) textual content automatically extracted from company websites, cleaned for French and English, and (2) annual reports collected and automatically filtered according to documentary criteria (year, format, duplication)<n>The processing pipeline includes automatic language detection, filtering of non-relevant content, extraction of relevant segments, and enrichment with structural metadata.
  arXiv  Detail & Related papers  (2025-12-08T10:04:50Z)
• ANUBHUTI: A Comprehensive Corpus For Sentiment Analysis In Bangla Regional Languages [0.5062312533373298]
  ANUBHUTI fills a critical gap in resources for sentiment analysis in low resource Bangla dialects.<n>The dataset features political and religious content, reflecting the contemporary socio political landscape of Bangladesh.<n>The dataset was further refined through systematic checks for missing data, anomalies, and inconsistencies.
  arXiv  Detail & Related papers  (2025-06-26T18:13:54Z)
• Named Entity Recognition in Historical Italian: The Case of Giacomo Leopardi's Zibaldone [4.795582035438343]
  There is an urgent need of computational techniques able to adapt to the challenges of historical texts.<n>The rise of large language models (LLMs) has revolutionized natural language processing.<n>No thorough evaluation has been proposed for Italian texts.
  arXiv  Detail & Related papers  (2025-05-26T15:16:48Z)
• A Comparative Analysis of Word Segmentation, Part-of-Speech Tagging, and Named Entity Recognition for Historical Chinese Sources, 1900-1950 [0.0]
  This paper compares large language models (LLMs) and traditional natural language processing (NLP) tools for performing word segmentation, part-of-speech (POS) tagging, and named entity recognition (NER)<n> Historical Chinese documents pose challenges for text analysis due to their logographic script, the absence of natural word boundaries, and significant linguistic changes.
  arXiv  Detail & Related papers  (2025-03-25T17:07:21Z)
• A Novel Cartography-Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpus [71.77214818319054]
  Natural language inference is a proxy for natural language understanding. There is no publicly available NLI corpus for the Romanian language. We introduce the first Romanian NLI corpus (RoNLI) comprising 58K training sentence pairs.
  arXiv  Detail & Related papers  (2024-05-20T08:41:15Z)
• Advancing Topic Segmentation and Outline Generation in Chinese Texts: The Paragraph-level Topic Representation, Corpus, and Benchmark [44.06803331843307]
  paragraph-level topic structure can grasp and understand the overall context of a document from a higher level. The lack of large-scale, high-quality Chinese paragraph-level topic structure corpora restrained research and applications. We propose a hierarchical paragraph-level topic structure representation with three layers to guide the corpus construction. We employ a two-stage man-machine collaborative annotation method to construct the largest Chinese paragraph-level Topic Structure corpus.
  arXiv  Detail & Related papers  (2023-05-24T06:43:23Z)
• O-Dang! The Ontology of Dangerous Speech Messages [53.15616413153125]
  We present O-Dang!: The Ontology of Dangerous Speech Messages, a systematic and interoperable Knowledge Graph (KG) O-Dang! is designed to gather and organize Italian datasets into a structured KG, according to the principles shared within the Linguistic Linked Open Data community. It provides a model for encoding both gold standard and single-annotator labels in the KG.
  arXiv  Detail & Related papers  (2022-07-13T11:50:05Z)
• Razmecheno: Named Entity Recognition from Digital Archive of Diaries "Prozhito" [1.4823641127537543]
  This paper aims to create a novel dataset "Razmecheno", gathered from the diary texts of the project "Prozhito" in Russian. Razmecheno comprises 1331 sentences and 14119 tokens, sampled from diaries, written during the Perestroika.
  arXiv  Detail & Related papers  (2022-01-24T23:06:01Z)
• TaxoCom: Topic Taxonomy Completion with Hierarchical Discovery of Novel Topic Clusters [57.59286394188025]
  We propose a novel framework for topic taxonomy completion, named TaxoCom. TaxoCom discovers novel sub-topic clusters of terms and documents. Our comprehensive experiments on two real-world datasets demonstrate that TaxoCom not only generates the high-quality topic taxonomy in terms of term coherency and topic coverage.
  arXiv  Detail & Related papers  (2022-01-18T07:07:38Z)
• SCROLLS: Standardized CompaRison Over Long Language Sequences [62.574959194373264]
  We introduce SCROLLS, a suite of tasks that require reasoning over long texts. SCROLLS contains summarization, question answering, and natural language inference tasks. We make all datasets available in a unified text-to-text format and host a live leaderboard to facilitate research on model architecture and pretraining methods.
  arXiv  Detail & Related papers  (2022-01-10T18:47:15Z)
• LexSubCon: Integrating Knowledge from Lexical Resources into Contextual Embeddings for Lexical Substitution [76.615287796753]
  We introduce LexSubCon, an end-to-end lexical substitution framework based on contextual embedding models. This is achieved by combining contextual information with knowledge from structured lexical resources. Our experiments show that LexSubCon outperforms previous state-of-the-art methods on LS07 and CoInCo benchmark datasets.
  arXiv  Detail & Related papers  (2021-07-11T21:25:56Z)
• Quasi Error-free Text Classification and Authorship Recognition in a large Corpus of English Literature based on a Novel Feature Set [0.0]
  We show that in the entire GLEC quasi error-free text classification and authorship recognition is possible with a method using the same set of five style and five content features. Our data pave the way for many future computational and empirical studies of literature or experiments in reading psychology.
  arXiv  Detail & Related papers  (2020-10-21T07:39:55Z)
• Abstractive Summarization of Spoken and Written Instructions with BERT [66.14755043607776]
  We present the first application of the BERTSum model to conversational language. We generate abstractive summaries of narrated instructional videos across a wide variety of topics. We envision this integrated as a feature in intelligent virtual assistants, enabling them to summarize both written and spoken instructional content upon request.
  arXiv  Detail & Related papers  (2020-08-21T20:59:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.