TildeOpen LLM: Leveraging Curriculum Learning to Achieve Equitable Language Representation

• URL: http://arxiv.org/abs/2603.08182v1
• Date: Mon, 09 Mar 2026 10:03:17 GMT
• Title: TildeOpen LLM: Leveraging Curriculum Learning to Achieve Equitable Language Representation
• Abstract summary: This paper presents TildeOpen LLM, an open-weight foundational model trained for 34 European languages.<n>To address the data imbalance, we combine dataset up tenfold with a curriculum-based training schedule that alternates between uniform and natural language distributions.<n>The model surpasses existing open-weight models in text generation and comprehension, particularly for Baltic, Finno-Ugric, and Slavic languages.
• Score: 0.24358758276023726
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models often underperform in many European languages due to the dominance of English and a few high-resource languages in training data. This paper presents TildeOpen LLM, a 30-billion-parameter open-weight foundational model trained for 34 European languages to promote linguistic equity and improve performance for low-resource languages. To address the data imbalance, we combine dataset upsampling with a curriculum-based training schedule that alternates between uniform and natural language distributions. The resulting model performs favorably compared to other multilingual LLMs despite being trained with significantly fewer computing resources. Evaluation across multiple multilingual benchmarks shows that TildeOpen surpasses existing open-weight models in text generation and comprehension, particularly for Baltic, Finno-Ugric, and Slavic languages. Human evaluations confirm an up to tenfold reduction in linguistic errors relative to leading baselines. The model and associated resources are fully open-weight and publicly available at huggingface.co/TildeAI/TildeOpen-30b. These outcomes demonstrate that careful data curation and balanced training strategies can substantially enhance multilingual model quality without increasing model size or training volume.

Related papers

• Merge and Conquer: Instructing Multilingual Models by Adding Target Language Weights [5.014897572890602]
  Large Language Models (LLMs) remain heavily centered on English, with limited performance in low-resource languages.<n>We show that model merging is a viable and efficient alternative to traditional adaptation methods for low-resource languages.
  arXiv  Detail & Related papers  (2026-03-30T10:46:50Z)
• MiniLingua: A Small Open-Source LLM for European Languages [47.78284815754002]
  MiniLingua is a multilingual open-source LLM of one billion parameters trained from scratch for 13 European languages.<n>We release model weights, tokenizer and source code used for data processing and model training.
  arXiv  Detail & Related papers  (2025-12-15T13:12:42Z)
• Enhancing Code Generation for Low-Resource Languages: No Silver Bullet [55.39571645315926]
  Large Language Models (LLMs) rely on large and diverse datasets to learn syntax, semantics, and usage patterns of programming languages.<n>For low-resource languages, the limited availability of such data hampers the models' ability to generalize effectively.<n>We present an empirical study investigating the effectiveness of several approaches for boosting LLMs' performance on low-resource languages.
  arXiv  Detail & Related papers  (2025-01-31T12:23:28Z)
• LDP: Generalizing to Multilingual Visual Information Extraction by Language Decoupled Pretraining [2.6638517946494535]
  We propose a multilingual training paradigm LDP (Language Decoupled Pre-training) for better utilization of monolingual pre-training data.<n>Our proposed model LDM is first pre-trained on the language-independent data, where the language knowledge is decoupled by a diffusion model, and then the LDM is fine-tuned on the downstream languages.
  arXiv  Detail & Related papers  (2024-12-19T07:31:40Z)
• Unlocking the Potential of Model Merging for Low-Resource Languages [66.7716891808697]
  Adapting large language models to new languages typically involves continual pre-training (CT) followed by supervised fine-tuning (SFT) We propose model merging as an alternative for low-resource languages, combining models with distinct capabilities into a single model without additional training. Experiments based on Llama-2-7B demonstrate that model merging effectively endows LLMs for low-resource languages with task-solving abilities, outperforming CT-then-SFT in scenarios with extremely scarce data.
  arXiv  Detail & Related papers  (2024-07-04T15:14:17Z)
• Open Generative Large Language Models for Galician [1.3049334790726996]
  Large language models (LLMs) have transformed natural language processing. Yet, their predominantly English-centric training has led to biases and performance disparities across languages. This imbalance marginalizes minoritized languages, making equitable access to NLP technologies more difficult for languages with lower resources, such as Galician. We present the first two generative LLMs focused on Galician to bridge this gap.
  arXiv  Detail & Related papers  (2024-06-19T23:49:56Z)
• Poro 34B and the Blessing of Multilinguality [3.270981284471548]
  Poro 34B is a 34 billion parameter model trained for 1 trillion tokens of Finnish, English, and programming languages.<n>We show that a multilingual training approach can produce a model that substantially advances over the capabilities of existing models for Finnish.
  arXiv  Detail & Related papers  (2024-04-02T11:34:12Z)
• PolyLM: An Open Source Polyglot Large Language Model [57.64420154135178]
  We present PolyLM, a multilingual large language model (LLMs) trained on 640 billion (B) tokens, avaliable in two model sizes: 1.7B and 13B. To enhance its multilingual capabilities, we 1) integrate bilingual data into training data; and 2) adopt a curriculum learning strategy that increases the proportion of non-English data from 30% in the first stage to 60% in the final stage during pre-training. Further, we propose a multilingual self-instruct method which automatically generates 132.7K diverse multilingual instructions for model fine-tuning.
  arXiv  Detail & Related papers  (2023-07-12T09:00:37Z)
• Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Prompts [75.33019401706188]
  Large language models (LLMs) are known to effectively perform tasks by simply observing few exemplars. We propose to assemble synthetic exemplars from a diverse set of high-resource languages to prompt the LLMs to translate from any language into English. Our unsupervised prompting method performs on par with supervised few-shot learning in LLMs of different sizes for translations between English and 13 Indic and 21 African low-resource languages.
  arXiv  Detail & Related papers  (2023-06-20T08:27:47Z)
• Generalizing Multimodal Pre-training into Multilingual via Language Acquisition [54.69707237195554]
  English-based Vision-Language Pre-training has achieved great success in various downstream tasks. Some efforts have been taken to generalize this success to non-English languages through Multilingual Vision-Language Pre-training. We propose a textbfMultitextbfLingual textbfAcquisition (MLA) framework that can easily generalize a monolingual Vision-Language Pre-training model into multilingual.
  arXiv  Detail & Related papers  (2022-05-29T08:53:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.