Multilingual TinyStories: A Synthetic Combinatorial Corpus of Indic Children's Stories for Training Small Language Models

• URL: http://arxiv.org/abs/2603.14563v1
• Date: Sun, 15 Mar 2026 19:31:47 GMT
• Title: Multilingual TinyStories: A Synthetic Combinatorial Corpus of Indic Children's Stories for Training Small Language Models
• Abstract summary: The TinyStories dataset is a large-scale, synthetically generated collection of children's stories encompassing 17 Indian languages.<n>We compiled 132,942 stories and over 93.9 million tokens in our release, serving as a foundational resource for multilingual language modeling and transfer learning.
• Score: 0.0
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: The development of robust language models for low-resource languages is frequently bottlenecked by the scarcity of high-quality, coherent, and domain-appropriate training corpora. In this paper, we introduce the Multilingual TinyStories dataset, a large-scale, synthetically generated collection of children's stories encompassing 17 Indian languages. Designed specifically for the training and evaluation of Small Language Models (SLMs), the corpus provides simple, narrative-driven text strictly localized to native scripts. We detail our hybrid curation pipeline, which leverages the Sarvam-M language model and a novel combinatorial prompt engineering framework for native generation, coupled with the Google Translate API for large-scale cross-lingual expansion. Through strict programmatic filtering, we compiled 132,942 stories and over 93.9 million tokens in our release, serving as a foundational resource for multilingual language modeling and transfer learning in the Indic linguistic sphere.

Related papers

• MameLoshnLM: Yiddish Language Model and Evaluation Benchmark [62.936233688546984]
  We present MameLoshnLM, the first open-source 8B- parameter language model built specifically for Yiddish.<n>Despite Yiddish's rich textual tradition, its limited digital presence and the scarcity of reliable evaluation resources have constrained progress in Yiddish language modeling.<n>Our results provide both a foundation for Yiddish NLP and a practical template for language model development in historically rich but digitally underrepresented languages.
  arXiv  Detail & Related papers  (2026-08-06T10:24:08Z)
• Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi [9.65814816271915]
  LilMoo is a 0.6-billion- parameter Hindi language model trained entirely from scratch.<n>It is developed through a fully transparent and reproducible pipeline optimized for limited compute environments.<n>Across comprehensive evaluation suites, LilMoo consistently outperforms comparably sized multilingual baselines.
  arXiv  Detail & Related papers  (2026-03-03T20:31:25Z)
• Kakugo: Distillation of Low-Resource Languages into Small Language Models [18.888596863202377]
  Kakugo is a pipeline designed to train general-purpose Small Language Models (SLMs) for low-resource languages.<n>With a total generation and training cost of under $50 per language, Kakugo offers a method accessible for communities to develop language-specific AI.
  arXiv  Detail & Related papers  (2026-01-20T15:05:44Z)
• Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research [139.69207791947738]
  Dolma is a three-trillion-token English corpus built from a diverse mixture of web content, scientific papers, code, public-domain books, social media, and encyclopedic materials. We document Dolma, including its design principles, details about its construction, and a summary of its contents. We present analyses and experimental results on intermediate states of Dolma to share what we have learned about important data curation practices.
  arXiv  Detail & Related papers  (2024-01-31T20:29:50Z)
• Paramanu: A Family of Novel Efficient Generative Foundation Language Models for Indian Languages [3.9018931027384056]
  We present "Paramanu", a family of novel language models (LM) for Indian languages. It covers 10 languages (Assamese, Bangla, Hindi, Konkani, Maithili, Marathi, Odia, Sanskrit, Tamil, Telugu) across 5 scripts. The models are pretrained on a single GPU with context size of 1024 and vary in size from 13.29 million (M) to 367.5 M parameters.
  arXiv  Detail & Related papers  (2024-01-31T17:58:10Z)
• NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languages [54.808217147579036]
  We conduct a case study on Indonesian local languages. We compare the effectiveness of online scraping, human translation, and paragraph writing by native speakers in constructing datasets. Our findings demonstrate that datasets generated through paragraph writing by native speakers exhibit superior quality in terms of lexical diversity and cultural content.
  arXiv  Detail & Related papers  (2023-09-19T14:42:33Z)
• Romanization-based Large-scale Adaptation of Multilingual Language Models [124.57923286144515]
  Large multilingual pretrained language models (mPLMs) have become the de facto state of the art for cross-lingual transfer in NLP. We study and compare a plethora of data- and parameter-efficient strategies for adapting the mPLMs to romanized and non-romanized corpora of 14 diverse low-resource languages. Our results reveal that UROMAN-based transliteration can offer strong performance for many languages, with particular gains achieved in the most challenging setups.
  arXiv  Detail & Related papers  (2023-04-18T09:58:34Z)
• Multi-lingual Evaluation of Code Generation Models [82.7357812992118]
  We present new benchmarks on evaluation code generation models: MBXP and Multilingual HumanEval, and MathQA-X. These datasets cover over 10 programming languages. We are able to assess the performance of code generation models in a multi-lingual fashion.
  arXiv  Detail & Related papers  (2022-10-26T17:17:06Z)
• Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalan [0.05277024349608833]
  This work focuses on Catalan with the aim of exploring what extent a medium-sized monolingual language model is competitive with state-of-the-art large multilingual models. We build a clean, high-quality textual Catalan corpus (CaText), train a Transformer-based language model for Catalan (BERTa), and devise a thorough evaluation in a diversity of settings. The result is a new benchmark, the Catalan Language Understanding Benchmark (CLUB), which we publish as an open resource.
  arXiv  Detail & Related papers  (2021-07-16T13:52:01Z)
• UNKs Everywhere: Adapting Multilingual Language Models to New Scripts [103.79021395138423]
  Massively multilingual language models such as multilingual BERT (mBERT) and XLM-R offer state-of-the-art cross-lingual transfer performance on a range of NLP tasks. Due to their limited capacity and large differences in pretraining data, there is a profound performance gap between resource-rich and resource-poor target languages. We propose novel data-efficient methods that enable quick and effective adaptation of pretrained multilingual models to such low-resource languages and unseen scripts.
  arXiv  Detail & Related papers  (2020-12-31T11:37:28Z)
• Learning Contextualised Cross-lingual Word Embeddings and Alignments for Extremely Low-Resource Languages Using Parallel Corpora [63.5286019659504]
  We propose a new approach for learning contextualised cross-lingual word embeddings based on a small parallel corpus. Our method obtains word embeddings via an LSTM encoder-decoder model that simultaneously translates and reconstructs an input sentence.
  arXiv  Detail & Related papers  (2020-10-27T22:24:01Z)
• The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MT [0.0]
  This paper describes the development of a new benchmark for machine translation that provides training and test data for thousands of language pairs. The main goal is to trigger the development of open translation tools and models with a much broader coverage of the World's languages.
  arXiv  Detail & Related papers  (2020-10-13T13:12:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.