Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMs

• URL: http://arxiv.org/abs/2410.03730v2
• Date: Tue, 15 Oct 2024 17:09:40 GMT
• Title: Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMs
• Authors: Mehdi Ali, Michael Fromm, Klaudia Thellmann, Jan Ebert, Alexander Arno Weber, Richard Rutmann, Charvi Jain, Max Lübbering, Daniel Steinigen, Johannes Leveling, Katrin Klug, Jasper Schulze Buschhoff, Lena Jurkschat, Hammam Abdelwahab, Benny Jörg Stein, Karl-Heinz Sylla, Pavel Denisov, Nicolo' Brandizzi, Qasid Saleem, Anirban Bhowmick, Lennard Helmer, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Alex Jude, Lalith Manjunath, Samuel Weinbach, Carolin Penke, Oleg Filatov, Shima Asaadi, Fabio Barth, Rafet Sifa, Fabian Küch, Andreas Herten, René Jäkel, Georg Rehm, Stefan Kesselheim, Joachim Köhler, Nicolas Flores-Herr,
• Abstract summary: We present two multilingual LLMs designed to embrace Europe's linguistic diversity by supporting all 24 official languages of the European Union. We detail the models' development principles, i.e., data composition, tokenizer optimization, and training methodologies.
• Score: 29.595342315049106
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present two multilingual LLMs designed to embrace Europe's linguistic diversity by supporting all 24 official languages of the European Union. Trained on a dataset comprising around 60% non-English data and utilizing a custom multilingual tokenizer, our models address the limitations of existing LLMs that predominantly focus on English or a few high-resource languages. We detail the models' development principles, i.e., data composition, tokenizer optimization, and training methodologies. The models demonstrate competitive performance across multilingual benchmarks, as evidenced by their performance on European versions of ARC, HellaSwag, MMLU, and TruthfulQA.

Related papers

• Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following [51.18383180774354]
  We introduce Multi-IF, a new benchmark designed to assess Large Language Models' proficiency in following multi-turn and multilingual instructions. Our evaluation of 14 state-of-the-art LLMs on Multi-IF reveals that it presents a significantly more challenging task than existing benchmarks. languages with non-Latin scripts (Hindi, Russian, and Chinese) generally exhibit higher error rates, suggesting potential limitations in the models' multilingual capabilities.
  arXiv  Detail & Related papers  (2024-10-21T00:59:47Z)
• Towards Multilingual LLM Evaluation for European Languages [3.3917876450975317]
  We introduce a multilingual evaluation approach tailored for European languages. We employ translated versions of five widely-used benchmarks to assess the capabilities of 40 LLMs across 21 European languages.
  arXiv  Detail & Related papers  (2024-10-11T15:53:24Z)
• EuroLLM: Multilingual Language Models for Europe [76.89545643715368]
  We introduce the EuroLLM project, aimed at developing a suite of open-weight multilingual LLMs. We outline the progress made to date, detailing our data collection and filtering process. We report our performance on multilingual general benchmarks and machine translation.
  arXiv  Detail & Related papers  (2024-09-24T16:51:36Z)
• A Survey of Large Language Models for European Languages [4.328283741894074]
  Large Language Models (LLMs) have gained significant attention due to their high performance on a wide range of natural language tasks. We present an overview of LLM families, including LLaMA, PaLM, GPT, and MoE. We provide a comprehensive summary of common monolingual and multilingual datasets used for pretraining large language models.
  arXiv  Detail & Related papers  (2024-08-27T13:10:05Z)
• X-LLaVA: Optimizing Bilingual Large Vision-Language Alignment [4.571088742209442]
  We create a 91K English-Korean-Chinese multilingual, multimodal training dataset. We develop a bilingual multimodal model that exhibits excellent performance in both Korean and English.
  arXiv  Detail & Related papers  (2024-03-18T01:14:47Z)
• D\'olares or Dollars? Unraveling the Bilingual Prowess of Financial LLMs Between Spanish and English [67.48541936784501]
  Tois'on de Oro is the first framework that establishes instruction datasets, finetuned LLMs, and evaluation benchmark for financial LLMs in Spanish joint with English. We construct a rigorously curated bilingual instruction dataset including over 144K Spanish and English samples from 15 datasets covering 7 tasks. We evaluate our model and existing LLMs using FLARE-ES, the first comprehensive bilingual evaluation benchmark with 21 datasets covering 9 tasks.
  arXiv  Detail & Related papers  (2024-02-12T04:50:31Z)
• UltraLink: An Open-Source Knowledge-Enhanced Multilingual Supervised Fine-tuning Dataset [69.33424532827608]
  Open-source large language models (LLMs) have gained significant strength across diverse fields. In this work, we construct an open-source multilingual supervised fine-tuning dataset. The resulting UltraLink dataset comprises approximately 1 million samples across five languages.
  arXiv  Detail & Related papers  (2024-02-07T05:05:53Z)
• Extrapolating Large Language Models to Non-English by Aligning Languages [109.09051737966178]
  Existing large language models show disparate capability across different languages. In this paper, we empower pre-trained LLMs on non-English languages by building semantic alignment across languages.
  arXiv  Detail & Related papers  (2023-08-09T13:32:06Z)
• PolyLM: An Open Source Polyglot Large Language Model [57.64420154135178]
  We present PolyLM, a multilingual large language model (LLMs) trained on 640 billion (B) tokens, avaliable in two model sizes: 1.7B and 13B. To enhance its multilingual capabilities, we 1) integrate bilingual data into training data; and 2) adopt a curriculum learning strategy that increases the proportion of non-English data from 30% in the first stage to 60% in the final stage during pre-training. Further, we propose a multilingual self-instruct method which automatically generates 132.7K diverse multilingual instructions for model fine-tuning.
  arXiv  Detail & Related papers  (2023-07-12T09:00:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.