Arkios: An Open Bilingual English-Nepali Language Model Trained From Scratch, with a Devanagari-Aware Tokenizer

• URL: http://arxiv.org/abs/2608.30092v2
• Date: Tue, 01 Sep 2026 02:17:13 GMT
• Title: Arkios: An Open Bilingual English-Nepali Language Model Trained From Scratch, with a Devanagari-Aware Tokenizer
• Abstract summary: We present Arkios, a dense transformer pretrained from scratch on 150B tokens of bilingual English-Nepali text.<n>Arkios exceeds three comparably sized open models (Pythia-1.4B, TinyLlama-1.1B, OLMo-1B) despite an order of magnitude fewer training tokens.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present Arkios, a 1.04B-parameter dense transformer pretrained from scratch on 150B tokens of bilingual English-Nepali text, using a custom single-file C/CUDA training stack and a Devanagari-aware byte-level BPE tokenizer built for this project. On ARC-Easy and ARC-Challenge, Arkios exceeds three comparably sized open models (Pythia-1.4B, TinyLlama-1.1B, OLMo-1B) despite an order of magnitude fewer training tokens, likely aided by a match between our educational-web-text pretraining data and ARC's grade-school-science format rather than a general capability advantage. We report full evaluation results under standard protocols, including a correction to an earlier partial-sample estimate, and findings specific to evaluating small models in a low-resource language: the standard multiple-choice-letter prompt format used by common evaluation harnesses places this model at chance on Nepali reading comprehension, and simultaneously at chance on English in the same format, which would lead a naive benchmark run to conclude the model has no Nepali ability when in fact it does. Concretely, both languages score at chance in the letter-choice format (0.240 Nepali, 0.236 English, against a chance baseline of 0.250), while scoring the answer text directly reveals genuine, English-favoring comprehension (0.306 Nepali, 0.387 English). We describe a manifest-conditioned tool-use contract introduced during instruction tuning, where tool calls are permitted only when a tool manifest is declared in context and suppressed otherwise, and report where that contract holds and where it does not. We release both the base and instruction-tuned model weights under Apache-2.0. The training code and a small privately-sourced portion of the Nepali pretraining corpus are not released; everything needed to reproduce the reported numbers from the released weights is included here.

Related papers

• Manacá-1B: An Open, Reproducible Brazilian-Portuguese Language Model and a Tokenizer-Aware, Paired Evaluation [0.0]
  We release Manac-1B, an open decoder-only model of 1.72 billion parameters trained from scratch for Brazilian Portuguese.<n>We evaluate the model against nine open baselines on four Portuguese benchmarks under a single harness.
  arXiv  Detail & Related papers  (2026-08-31T01:00:14Z)
• Neural Grammatical Error Correction for Romanian [1.2680800636608986]
  We introduce a first Grammatical Error Correction corpus for Romanian consisting of 10k pairs of sentences.<n>Multiple neural models were experimented, together with pretraining strategies, which proved effective for GEC in low-resource settings.
  arXiv  Detail & Related papers  (2026-04-26T09:42:49Z)
• Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi [9.65814816271915]
  LilMoo is a 0.6-billion- parameter Hindi language model trained entirely from scratch.<n>It is developed through a fully transparent and reproducible pipeline optimized for limited compute environments.<n>Across comprehensive evaluation suites, LilMoo consistently outperforms comparably sized multilingual baselines.
  arXiv  Detail & Related papers  (2026-03-03T20:31:25Z)
• Towards Nepali-language LLMs: Efficient GPT training with a Nepali BPE tokenizer [0.0]
  This study presents a GPT-2-based Nepali language model trained using several training strategies inspired by GPT-3.<n>The model achieved a training loss of 3.168177, a validation loss of 3.081982, and a final perplexity of 21.80, demonstrating its capability to generate coherent Nepali news-style text.
  arXiv  Detail & Related papers  (2025-12-16T16:53:11Z)
• mmBERT: A Modern Multilingual Encoder with Annealed Language Learning [57.58071656545661]
  mmBERT is an encoder-only language model pretrained on 3T tokens of multilingual text.<n>We add over 1700 low-resource languages to the data mix only during the decay phase.<n>We show that mmBERT significantly outperforms the previous generation of models on classification and retrieval tasks.
  arXiv  Detail & Related papers  (2025-09-08T17:08:42Z)
• Pretraining Language Models to Ponder in Continuous Space [50.52734567589996]
  We introduce this pondering process into language models by repeatedly invoking the forward process within a single token generation step.<n>We show that the model can learn to ponder in this way through self-supervised learning, without any human annotations.
  arXiv  Detail & Related papers  (2025-05-27T03:47:33Z)
• Development of Pre-Trained Transformer-based Models for the Nepali Language [0.0]
  The Nepali language, spoken by approximately 32 million people worldwide, remains significantly underrepresented in this domain.<n>We have collected 27.5 GB of Nepali text data, approximately 2.4x larger than any previously available Nepali language corpus.<n>Our models outperformed the existing best model by 2 points on Nep-gLUE benchmark, scoring 95.60 and also outperformed existing models on text generation tasks.
  arXiv  Detail & Related papers  (2024-11-24T06:38:24Z)
• Paloma: A Benchmark for Evaluating Language Model Fit [112.481957296585]
  Evaluations of language models (LMs) commonly report perplexity on monolithic data held out from training.<n>We introduce Perplexity Analysis for Language Model Assessment (Paloma), a benchmark to measure LM fit to 546 English and code domains.
  arXiv  Detail & Related papers  (2023-12-16T19:12:45Z)
• Skywork: A More Open Bilingual Foundation Model [55.927396986873816]
  We present Skywork-13B, a family of large language models (LLMs) trained on a corpus of over 3.2 trillion tokens drawn from both English and Chinese texts. We show that our model not only excels on popular benchmarks, but also achieves emphstate of the art performance in Chinese language modeling on diverse domains.
  arXiv  Detail & Related papers  (2023-10-30T08:31:47Z)
• Language Model Pre-Training with Sparse Latent Typing [66.75786739499604]
  We propose a new pre-training objective, Sparse Latent Typing, which enables the model to sparsely extract sentence-level keywords with diverse latent types. Experimental results show that our model is able to learn interpretable latent type categories in a self-supervised manner without using any external knowledge.
  arXiv  Detail & Related papers  (2022-10-23T00:37:08Z)
• AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages [75.08199398141744]
  We present AmericasNLI, an extension of XNLI (Conneau et al.), to 10 indigenous languages of the Americas. We conduct experiments with XLM-R, testing multiple zero-shot and translation-based approaches. We find that XLM-R's zero-shot performance is poor for all 10 languages, with an average performance of 38.62%.
  arXiv  Detail & Related papers  (2021-04-18T05:32:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.