ks-pret-5m: a 5 million word, 12 million token kashmiri pretraining dataset

• URL: http://arxiv.org/abs/2604.11066v1
• Date: Mon, 13 Apr 2026 06:46:37 GMT
• Title: ks-pret-5m: a 5 million word, 12 million token kashmiri pretraining dataset
• Abstract summary: KS-PRET-5M is the largest publicly available pretraining dataset for the Kashmiri language.<n>It comprises 5,090,244 (5.09M) words, 27,692,959 (27.6M) characters, and a vocabulary of 295,433 (295.4K) unique word types.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We present KS-PRET-5M, the largest publicly available pretraining dataset for the Kashmiri language, comprising 5,090,244 (5.09M) words, 27,692,959 (27.6M) characters, and a vocabulary of 295,433 (295.4K) unique word types. We assembled the dataset from two source classes: digitized archival and literary material, encompassing literature, news, biographies, novels, poetry, religious scholarship, and academic writing, recovered from the proprietary InPage desktop-publishing format using the converter of Malik~\cite{malik2024inpage}, and Unicode-native text collected from Kashmiri-language web sources. All text was processed through an eleven-stage cleaning pipeline that achieves a mean Kashmiri script ratio of 0.9965, reducing Devanagari contamination to 146 characters across the full dataset. We tokenized the dataset empirically using google/muril-base-cased, yielding a subword ratio of 2.383 tokens per word and a total of approximately 12.13 million subword tokens, substantially higher than prior estimates derived from non-Kashmiri Perso-Arabic analogues. KS-PRET-5M is released as a single continuous text stream under CC~BY~4.0 to support language model pretraining, tokenizer training, and computational linguistic research for Kashmiri.

Related papers

• Koshur Pixel: a large-scale synthetic ocr dataset for kashmiri [0.0]
  We introduce Koshur Pixel, the first large-scale synthetic OCR dataset for Kashmiri.<n>Koshur Pixel comprises 613,078 image-text pairs generated from the KS-PRET-5M corpus using the SynthOCR-Gen framework.
  arXiv  Detail & Related papers  (2026-06-22T10:42:36Z)
• Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech [61.759910921200834]
  Cross-lingual sentence encoders typically cover only a few hundred languages.<n>We introduce OmniSONAR, a new family of omnilingual, cross-lingual and cross-language sentence embedding models.
  arXiv  Detail & Related papers  (2026-03-17T14:47:35Z)
• Chitrakshara: A Large Multilingual Multimodal Dataset for Indian languages [2.592717137976924]
  We introduce the Chitrakshara dataset series, covering 11 Indian languages sourced from Common Crawl.<n>It comprises (1) Chitrakshara-IL, a large-scale interleaved pretraining dataset with 193M images, 30B text tokens, and 50M multilingual documents, and (2) Chitrakshara-Cap, which includes 44M image-text pairs with 733M tokens.
  arXiv  Detail & Related papers  (2026-03-06T15:01:25Z)
• ks-lit-3m: A 3.1 million word kashmiri text dataset for large language model pretraining [0.0]
  This paper introduces KS-LIT-3M, a curated corpus of 3.1 million words (16.4 million characters) specifically designed for pretraining language models on Kashmiri.<n>The dataset is released under the CC-BY-4.0 license to facilitate research in Kashmiri natural language processing.
  arXiv  Detail & Related papers  (2026-01-03T06:43:26Z)
• InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning [58.7966588457529]
  InfiMM-WebMath-40B is a high-quality dataset of interleaved image-text documents. It comprises 24 million web pages, 85 million associated image URLs, and 40 billion text tokens, all meticulously extracted and filtered from CommonCrawl. Our evaluations on text-only benchmarks show that, despite utilizing only 40 billion tokens, our dataset significantly enhances the performance of our 1.3B model. Our models set a new state-of-the-art among open-source models on multi-modal math benchmarks such as MathVerse and We-Math.
  arXiv  Detail & Related papers  (2024-09-19T08:41:21Z)
• AceParse: A Comprehensive Dataset with Diverse Structured Texts for Academic Literature Parsing [82.33075210051129]
  We introduce AceParse, the first comprehensive dataset designed to support the parsing of structured texts.<n>Based on AceParse, we fine-tuned a multimodal model, named Ace, which accurately parses various structured texts.<n>This model outperforms the previous state-of-the-art by 4.1% in terms of F1 score and by 5% in Jaccard Similarity.
  arXiv  Detail & Related papers  (2024-09-16T06:06:34Z)
• Data Engineering for Scaling Language Models to 128K Context [98.41554785106902]
  We study the continual pretraining recipe for scaling language models' context lengths to 128K. We find that naively upsampling longer data on certain domains like books, a common practice of existing work, gives suboptimal performance. Our recipe outperforms strong open-source long-context models and closes the gap to frontier models like GPT-4 128K.
  arXiv  Detail & Related papers  (2024-02-15T18:19:16Z)
• MegaWika: Millions of reports and their sources across 50 diverse languages [74.3909725023673]
  MegaWika consists of 13 million Wikipedia articles in 50 diverse languages, along with their 71 million referenced source materials. We process this dataset for a myriad of applications, including translating non-English articles for cross-lingual applications. MegaWika is the largest resource for sentence-level report generation and the only report generation dataset that is multilingual.
  arXiv  Detail & Related papers  (2023-07-13T20:04:02Z)
• A Benchmark and Dataset for Post-OCR text correction in Sanskrit [23.45279030301887]
  Sanskrit is a classical language with about 30 million extant manuscripts fit for digitisation. We release a post-OCR text correction dataset containing around 218,000 sentences, with 1.5 million words, from 30 different books.
  arXiv  Detail & Related papers  (2022-11-15T08:32:18Z)
• ASR2K: Speech Recognition for Around 2000 Languages without Audio [100.41158814934802]
  We present a speech recognition pipeline that does not require any audio for the target language. Our pipeline consists of three components: acoustic, pronunciation, and language models. We build speech recognition for 1909 languages by combining it with Crubadan: a large endangered languages n-gram database.
  arXiv  Detail & Related papers  (2022-09-06T22:48:29Z)
• Towards Deployable OCR models for Indic languages [33.6638069612644]
  Modelling unsegmented sequences using Connectionist Temporal Classification (CTC) is the most commonly used approach for segmentation-free OCR.<n>We present a study of various neural network models that uses CTC for transcribing step-wise predictions in the neural network output to a Unicode sequence.<n>We also introduce a new public dataset called Mozhi for word and line recognition in Indian language.
  arXiv  Detail & Related papers  (2022-05-13T16:19:21Z)
• Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users [32.23606056944172]
  We introduce Aksharantar, the largest publicly available transliteration dataset for Indian languages created by mining from monolingual and parallel corpora. The dataset contains 26 million transliteration pairs for 21 Indic languages from 3 language families using 12 scripts. Aksharantar is 21 times larger than existing datasets and is the first publicly available dataset for 7 languages and 1 language family.
  arXiv  Detail & Related papers  (2022-05-06T05:13:12Z)
• Comprehensive Benchmark Datasets for Amharic Scene Text Detection and Recognition [56.048783994698425]
  Ethiopic/Amharic script is one of the oldest African writing systems, which serves at least 23 languages in East Africa. The Amharic writing system, Abugida, has 282 syllables, 15 punctuation marks, and 20 numerals. We presented the first comprehensive public datasets named HUST-ART, HUST-AST, ABE, and Tana for Amharic script detection and recognition in the natural scene.
  arXiv  Detail & Related papers  (2022-03-23T03:19:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.