PashtoCorp: A 1.25-Billion-Word Corpus, Evaluation Suite, and Reproducible Pipeline for Low-Resource Language Development

• URL: http://arxiv.org/abs/2603.16354v1
• Date: Tue, 17 Mar 2026 10:36:18 GMT
• Title: PashtoCorp: A 1.25-Billion-Word Corpus, Evaluation Suite, and Reproducible Pipeline for Low-Resource Language Development
• Abstract summary: PashtoCorp is a 1.25-billion-word corpus for Pashto, a language spoken by 60 million people that remains underrepresented in NLP.<n>At 1.25B words across 2.81 million documents, PashtoCorp is 40x larger than the OSCAR Pashto subset and 83x larger than the previously largest dedicated Pashto corpus.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present PashtoCorp, a 1.25-billion-word corpus for Pashto, a language spoken by 60 million people that remains severely underrepresented in NLP. The corpus is assembled from 39 sources spanning seven HuggingFace datasets and 32 purpose-built web scrapers, processed through a reproducible pipeline with Arabic-script tokenization, SHA-256 deduplication, and quality filtering. At 1.25B words across 2.81 million documents, PashtoCorp is 40x larger than the OSCAR Pashto subset and 83x larger than the previously largest dedicated Pashto corpus. Continued MLM pretraining of XLM-R-base on PashtoCorp reduces held-out perplexity by 25.1% (8.08->6.06). On WikiANN Pashto NER, the pretrained model improves entity F1 by 10% relative (19.0%->21.0%) and reduces training variance nearly 7x; the largest gain appears at 50 training sentences (+27%), with PashtoCorp covering 97.9% of WikiANN entity vocabulary. On Belebele Pashto reading comprehension, Gemma-3n achieves 64.6% accuracy, the first published LLM baseline for Pashto on this benchmark. A leave-one-out source ablation shows that Wikipedia (0.7% of documents) is the most critical source for NER: removing it alone reduces entity F1 by 47%. Corpus data, trained model, and code are available at https://huggingface.co/datasets/ihanif/pashto-corpus, https://huggingface.co/ihanif/xlmr-pashto, and https://github.com/ihanif/pashto-corpus.

Related papers

• Expanding the Lexicon of Ge'ez Based African Languages: A Comparative Study of Amharic and Tigrinya [6.396476220617138]
  We introduce a vocabulary-extended variant of XLM-R targeting the two highest-resource Ge'ez-script languages, Amharic and Tigrinya.<n>VexMLM achieves 87.0 EM /90.0 F1, versus 66.0 EM/78.0 F1 for XLM-R and 74.0 EM/ 78.0 F1 for Glot500.<n>On SA, VEXMLM reaches 80.0% accuracy versus 77.0% (XLM-R) and 46.0% (Glot500)
  arXiv  Detail & Related papers  (2026-07-16T17:08:28Z)
• Context-Aware Distillation and Ablation for Text2DSL [78.29352305480285]
  We replace prompt-only synthetic generation with context-aware distillation.<n>A teacher large language model operates under an explicitly defined structured context.<n>This scales the verified PolkitBench corpus from 4,204 to 10,073 pairs.
  arXiv  Detail & Related papers  (2026-06-21T16:27:24Z)
• IHUBERT: Vector-Based Semantic Deduplication and Domain-Balanced Pretraining for Persian Resources [1.022165439882148]
  Persian pretrained language models (PLMs) are still limited by the scarcity of large-scale, high-quality pretraining corpora.<n>We present IHUBERT, a monolingual Persian PLM trained from scratch with the RoBERTa-base encoder (125M parameters) on a 45 GB curated subset of the Sepahr-Danesh collection.<n>IHUBERT is evaluated on seven Persian NLU benchmarks covering NER, sentiment analysis, topic classification, NLI, extractive question answering, and relation extraction.
  arXiv  Detail & Related papers  (2026-06-18T11:10:22Z)
• Benchmarking Speech-to-Speech Translation Models [55.00303727199927]
  Speech-to-speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol.<n>We introduce a benchmarking framework integrating 46 metrics across eight dimensions.<n>We deploy it on 1,248 model-language configurations from FLEURS and CVSS.
  arXiv  Detail & Related papers  (2026-06-02T07:01:33Z)
• VectraYX-Nano: A 42M-Parameter Spanish Cybersecurity Language Model with Curriculum Learning and Native Tool Use [0.0]
  We present VectraYX-Nano, a 41.95M- parameter decoder-only language model trained from scratch in Spanish for cybersecurity.<n>A 170M-token Spanish corpus assembled by an eight-VM distributed pipeline at $25 USD of cloud compute and split into three curriculum phases (conversational 42M, cybersecurity 118M, offensive tooling 10M)<n> Curriculum with replay across the three phases yields a monotonic loss descent (9.80 -> 3.17 -> 3.00 -> 2.16); after SFT (loss 1.74) the v2 bootstrap-ablation reference attains a conversational gate of 0.775 +/-
  arXiv  Detail & Related papers  (2026-05-13T18:03:07Z)
• Delta-Based Neural Architecture Search: LLM Fine-Tuning via Code Diffs [48.83701310501069]
  Large language models (LLMs) show strong potential for neural architecture generation.<n>Existing approaches produce complete model implementations from scratch.<n>We propose Delta-Code Generation, where fine-tuned LLMs generate compact unified diffs.
  arXiv  Detail & Related papers  (2026-05-06T13:32:05Z)
• Pashto Common Voice: Building the First Open Speech Corpus for a 60-Million-Speaker Low-Resource Language [0.0]
  We present the Pashto Common Voice corpus -- the first large-scale, openly licensed speech resource for Pashto.<n>Through a community effort spanning 2022-2025, the corpus grew from 1.5 hours and 5 contributors to 147 total hours and 1,483 unique speakers.<n>Speaker participation increased approximately 108-fold between CV17 and CV18, coinciding with a VOA Pashto broadcast campaign.
  arXiv  Detail & Related papers  (2026-03-27T22:22:03Z)
• Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech [61.759910921200834]
  Cross-lingual sentence encoders typically cover only a few hundred languages.<n>We introduce OmniSONAR, a new family of omnilingual, cross-lingual and cross-language sentence embedding models.
  arXiv  Detail & Related papers  (2026-03-17T14:47:35Z)
• BLUFF: Benchmarking the Detection of False and Synthetic Content across 58 Low-Resource Languages [11.893370892009834]
  BLUFF is a benchmark for detecting false and synthetic content in 79 languages.<n>It covers both high-resource "big-head" (20) and low-resource "long-tail" (59) languages.<n>We present AXL-CoI, a novel multi-agentic framework for controlled fake/real news generation.
  arXiv  Detail & Related papers  (2026-02-28T12:58:31Z)
• Apertus: Democratizing Open and Compliant LLMs for Global Language Environments [163.70368742538187]
  Apertus is a fully open suite of large language models (LLMs) designed to address two systemic shortcomings in today's open model ecosystem.<n>Apertus models are pretrained exclusively on openly available data, respecting robots.txt exclusions and filtering for non-permissive, toxic, and personally identifiable content.<n>The Apertus models also expand multilingual coverage, training on 15T tokens from over 1800 languages, with 40% of pretraining data allocated to non-English content.
  arXiv  Detail & Related papers  (2025-09-17T17:59:21Z)
• Towards Robust Speech Representation Learning for Thousands of Languages [77.2890285555615]
  Self-supervised learning (SSL) has helped extend speech technologies to more languages by reducing the need for labeled data. We propose XEUS, a Cross-lingual for Universal Speech, trained on over 1 million hours of data across 4057 languages.
  arXiv  Detail & Related papers  (2024-06-30T21:40:26Z)
• Building a Large Japanese Web Corpus for Large Language Models [20.407125127485276]
  This study builds a large Japanese web corpus by extracting and refining text from the Common Crawl archive. This corpus consists of approximately 312.1 billion characters (approximately 173 million pages) To confirm the quality of the corpus, we performed continual pre-training on Llama 2 7B, 13B, 70B, Mistral 7B v0.1, and Mixtral 8x7B Instruct as base LLMs.
  arXiv  Detail & Related papers  (2024-04-27T00:02:45Z)
• Skywork: A More Open Bilingual Foundation Model [55.927396986873816]
  We present Skywork-13B, a family of large language models (LLMs) trained on a corpus of over 3.2 trillion tokens drawn from both English and Chinese texts. We show that our model not only excels on popular benchmarks, but also achieves emphstate of the art performance in Chinese language modeling on diverse domains.
  arXiv  Detail & Related papers  (2023-10-30T08:31:47Z)
• No Language Left Behind: Scaling Human-Centered Machine Translation [69.28110770760506]
  We create datasets and models aimed at narrowing the performance gap between low and high-resource languages. We propose multiple architectural and training improvements to counteract overfitting while training on thousands of tasks. Our model achieves an improvement of 44% BLEU relative to the previous state-of-the-art.
  arXiv  Detail & Related papers  (2022-07-11T07:33:36Z)
• Does Corpus Quality Really Matter for Low-Resource Languages? [27.315905109092466]
  The vast majority of non-English corpora are derived from automatically filtered versions of CommonCrawl. Taking Basque as a case study, we explore tailored crawling (manually identifying and scraping websites with high-quality content) as an alternative to filtering CommonCrawl. Our new corpus, called EusCrawl, is similar in size to the Basque portion of popular multilingual corpora like CC100 and mC4.
  arXiv  Detail & Related papers  (2022-03-15T17:40:27Z)
• Monolingual and Parallel Corpora for Kangri Low Resource Language [0.0]
  This paper presents the dataset of Himachali low resource endangered language, Kangri (ISO 639-3xnr) listed in the United Nations Educational, Scientific and Cultural Organization (UNESCO) The corpus contains 1,81,552 Monolingual and 27,362 Hindi-Kangri Parallel corpora.
  arXiv  Detail & Related papers  (2021-03-22T05:52:51Z)
• ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context [58.40112382877868]
  We propose a novel CNN-RNN-transducer architecture, which we call ContextNet. ContextNet features a fully convolutional encoder that incorporates global context information into convolution layers by adding squeeze-and-excitation modules. We demonstrate that ContextNet achieves a word error rate (WER) of 2.1%/4.6% without external language model (LM), 1.9%/4.1% with LM and 2.9%/7.0% with only 10M parameters on the clean/noisy LibriSpeech test sets.
  arXiv  Detail & Related papers  (2020-05-07T01:03:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.