Llama-GENBA-10B: A Trilingual Large Language Model for German, English and Bavarian

• URL: http://arxiv.org/abs/2509.05668v1
• Date: Sat, 06 Sep 2025 10:12:52 GMT
• Title: Llama-GENBA-10B: A Trilingual Large Language Model for German, English and Bavarian
• Abstract summary: Llama-GENBA-10B is a trilingual foundation model addressing English-centric bias in large language models.<n>It is continuously pretrained on 164B tokens (82B English, 82B German, and 80M Bavarian) balancing resources while preventing English dominance.
• Score: 0.7039577044513267
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We present Llama-GENBA-10B, a trilingual foundation model addressing English-centric bias in large language models. Built on Llama 3.1-8B and scaled to 10B parameters, Llama-GENBA-10B is continuously pretrained on 164B tokens (82B English, 82B German, and 80M Bavarian), balancing resources while preventing English dominance. Targeted at the German NLP community, the model also promotes Bavarian as a low-resource language. Development tackled four challenges: (1) curating a multilingual corpus despite Bavarian scarcity, (2) creating a unified tokenizer for English, German, and Bavarian, (3) optimizing architecture and language-ratio hyperparameters for cross-lingual transfer, and (4) establishing the first standardized trilingual evaluation suite by translating German benchmarks into Bavarian. Evaluations show that Llama-GENBA-10B achieves strong cross-lingual performance, with the fine-tuned variant surpassing Apertus-8B-2509 and gemma-2-9b in Bavarian and establishing itself as the best model in its class for this language, while also outperforming EuroLLM in English and matching its results in German. Training on the Cerebras CS-2 demonstrated efficient large-scale multilingual pretraining with documented energy use, offering a blueprint for inclusive foundation models that integrate low-resource languages.

Related papers

• A Sovereign, Open-Source Foundation Model for German and English [50.08096276509294]
  Soofi S 30B-A3B is a sovereign, open-source Mamba Transformer foundation model for German and English.<n>Its hybrid design activates only 3B of 30B parameters per token and keeps the inference cache near-constant as context grows.
  arXiv  Detail & Related papers  (2026-07-10T13:51:41Z)
• Gamayun's Path to Multilingual Mastery: Cost-Efficient Training of a 1.5B-Parameter LLM [30.381516759139203]
  We present Gamayun, a multilingual language model trained entirely from scratch on 2.5T tokens.<n>Our model supports 12 languages, with special focus on Russian.<n>It matches or exceeds Qwen3 (36T tokens) on most tasks outside advanced STEM.
  arXiv  Detail & Related papers  (2025-12-25T08:52:23Z)
• Pipeline Analysis for Developing Instruct LLMs in Low-Resource Languages: A Case Study on Basque [2.867517731896504]
  Large language models (LLMs) are typically optimized for resource-rich languages like English, exacerbating the gap between high-resource and underrepresented languages.<n>This work presents a detailed analysis of strategies for developing a model capable of following instructions in a low-resource language, specifically Basque, by focusing on three key stages: pre-training, instruction tuning, and alignment with human preferences.
  arXiv  Detail & Related papers  (2024-12-18T15:05:59Z)
• BgGPT 1.0: Extending English-centric LLMs to other languages [12.867025651644692]
  We present BgGPT-Gemma-2-27B-Instruct and BgGPT-Gemma-2-9B-Instruct: continually pretrained and fine-tuned versions of Google's Gemma-2 models.<n>Our models demonstrate strong performance in Bulgarian language tasks, setting a new standard for language-specific AI models.
  arXiv  Detail & Related papers  (2024-12-14T16:49:52Z)
• Cross-lingual transfer of multilingual models on low resource African Languages [0.20793001310272596]
  monolingual models, trained on a single language, may better capture the nuances of the target language. AfriBERT achieved the highest cross-lingual accuracy of 88.3% after fine-tuning. BiGRU emerged as the best-performing neural model with 83.3% accuracy.
  arXiv  Detail & Related papers  (2024-09-17T08:05:40Z)
• Baichuan 2: Open Large-scale Language Models [51.34140526283222]
  We present Baichuan 2, a series of large-scale multilingual language models containing 7 billion and 13 billion parameters, trained from scratch, on 2.6 trillion tokens.<n>Baichuan 2 matches or outperforms other open-source models of similar size on public benchmarks like MMLU, CMMLU, GSM8K, and HumanEval.
  arXiv  Detail & Related papers  (2023-09-19T04:13:22Z)
• ML-SUPERB: Multilingual Speech Universal PERformance Benchmark [94.64616634862995]
  Speech processing Universal PERformance Benchmark (SUPERB) is a leaderboard to benchmark the performance of Self-Supervised Learning (SSL) models on various speech processing tasks.<n>This paper presents multilingual SUPERB, covering 143 languages (ranging from high-resource to endangered), and considering both automatic speech recognition and language identification.<n>Similar to the SUPERB benchmark, we find speech SSL models can significantly improve performance compared to FBANK features.
  arXiv  Detail & Related papers  (2023-05-18T00:01:27Z)
• BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting [50.24676567971536]
  The BLOOM model is a large publicly available multilingual language model, but its pretraining was limited to 46 languages. We apply existing language adaptation strategies to BLOOM and benchmark its zero-shot prompting performance on eight new languages. We conclude that with sufficient training data language adaptation can generalize well to diverse languages.
  arXiv  Detail & Related papers  (2022-12-19T15:24:45Z)
• Few-shot Learning with Multilingual Language Models [66.49496434282564]
  We train multilingual autoregressive language models on a balanced corpus covering a diverse set of languages. Our largest model sets new state of the art in few-shot learning in more than 20 representative languages. We present a detailed analysis of where the model succeeds and fails, showing in particular that it enables cross-lingual in-context learning.
  arXiv  Detail & Related papers  (2021-12-20T16:52:35Z)
• Zero-Shot Cross-Lingual Transfer in Legal Domain Using Transformer models [0.0]
  We study zero-shot cross-lingual transfer from English to French and German under Multi-Label Text Classification. We extend EURLEX57K dataset, the English dataset for topic classification of legal documents, with French and German official translation. We find that Language model finetuning of multi-lingual pre-trained model (M-DistilBERT, M-BERT) leads to 32.0-34.94%, 76.15-87.54% relative improvement on French and German test sets.
  arXiv  Detail & Related papers  (2021-11-28T16:25:04Z)
• XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale [48.0390317915984]
  XLS-R is a large-scale model for cross-lingual speech representation learning based on wav2vec 2.0. We train models with up to 2B parameters on nearly half a million hours of publicly available speech audio in 128 languages.
  arXiv  Detail & Related papers  (2021-11-17T18:49:42Z)
• Multilingual Speech Translation with Efficient Finetuning of Pretrained Models [82.22294901727933]
  A minimalistic LNA (LayerNorm and Attention) finetuning can achieve zero-shot crosslingual and cross-modality transfer ability. Our approach demonstrates strong zero-shot performance in a many-to-many multilingual model.
  arXiv  Detail & Related papers  (2020-10-24T08:15:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.