A Sovereign, Open-Source Foundation Model for German and English

• URL: http://arxiv.org/abs/2607.09424v2
• Date: Mon, 13 Jul 2026 08:51:55 GMT
• Title: A Sovereign, Open-Source Foundation Model for German and English
• Abstract summary: Soofi S 30B-A3B is a sovereign, open-source Mamba Transformer foundation model for German and English.<n>Its hybrid design activates only 3B of 30B parameters per token and keeps the inference cache near-constant as context grows.
• Score: 50.08096276509294
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present Soofi S 30B-A3B, a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba Transformer foundation model for German and English. Its hybrid design activates only 3B of 30B parameters per token and keeps the inference cache near-constant as context grows, giving it a decisive throughput advantage over dense models for long-context, high-concurrency deployment. Pretrained on roughly 27 trillion tokens with deliberately up-weighted German, Soofi S matches dense 14 to 27B models on aggregate English and German benchmarks while achieving the best code aggregates in both languages among 17 open base models, and outperforms every European sovereign baseline in our comparison, including ones far larger in active parameters. Among fully open models, Soofi S obtains the highest English and German evaluation scores, ahead of Olmo 3 32B and Apertus 70B. Soofi S was built end-to-end on the German Industrial AI Cloud, a sovereign HPC scale AI infrastructure operated by Deutsche Telekom in Munich. Soofi S will be released under highly permissive, open-access terms: weights, selected intermediate checkpoints, full per-source data accounting, hyperparameters, and training and evaluation code. Where source licenses permit, data-construction artifacts are released under permissive licenses; commercially licensed sources are documented with aggregate statistics and exact mixture accounting.

Related papers

• DenseOn with the LateOn: Fully Open Dense and Late-Interaction Models for Multilingual, Long-Context, and Code Search [2.0669978569487384]
  We show how English supervision transfers to multilingual retrieval through translate-train.<n>We first reconstruct and curate 665M English contrastive pre-training pairs from 1.4B pairs across 34 public sources.<n>We translate the validated English data into eight languages, yielding 2.8B pairs with cross-lingual samples.
  arXiv  Detail & Related papers  (2026-07-29T17:50:51Z)
• LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening [69.1037790901185]
  We present LLMEval-Logic, a Chinese logical reasoning benchmark built from realistic situational scenarios.<n>Its pipeline forward-authors and expert-audits natural-language items together with their reference formalizations, verifies annotated answers with Z3, constructs expert rubrics for natural-to-formal grading, and hardens selected items through a closed-loop adversarial workflow.
  arXiv  Detail & Related papers  (2026-05-19T09:40:29Z)
• EngGPT2: Sovereign, Efficient and Open Intelligence [0.0]
  EngGPT2 is built to be a Sovereign, Efficient and Open model.<n>It's trained on 2.5 trillion tokens - less than Qwen3's 36T or Llama3's 15T.<n>It delivers performance on key benchmarks, including MMLU-Pro, GSM8K, IFEval and HumanEval.
  arXiv  Detail & Related papers  (2026-03-17T12:08:35Z)
• The German Commons - 154 Billion Tokens of Openly Licensed Text for German Language Models [41.865590656976316]
  The German Commons is the largest collection of openly licensed German text to date.<n>It compiles data from 41 sources across seven domains, encompassing legal, scientific, cultural, political, news, economic, and web text.
  arXiv  Detail & Related papers  (2025-10-15T18:24:26Z)
• MobileLLM-R1: Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes [60.57770396565211]
  We show that strong reasoning abilities can emerge with far less data.<n>MobileLLM-R50M achieves an AIME score of 15.5, compared to just 0.6 for OLMo-2-1.48B and 0.3 for SmolLM-2-1.7B.
  arXiv  Detail & Related papers  (2025-09-29T15:43:59Z)
• Llama-GENBA-10B: A Trilingual Large Language Model for German, English and Bavarian [0.7039577044513267]
  Llama-GENBA-10B is a trilingual foundation model addressing English-centric bias in large language models.<n>It is continuously pretrained on 164B tokens (82B English, 82B German, and 80M Bavarian) balancing resources while preventing English dominance.
  arXiv  Detail & Related papers  (2025-09-06T10:12:52Z)
• YAYI 2: Multilingual Open-Source Large Language Models [53.92832054643197]
  We propose YAYI 2, including both base and chat models, with 30 billion parameters. YAYI 2 is pre-trained from scratch on a multilingual corpus which contains 2.65 trillion tokens filtered by our pre-training data processing pipeline. The base model is aligned with human values through supervised fine-tuning with millions of instructions and reinforcement learning from human feedback.
  arXiv  Detail & Related papers  (2023-12-22T17:34:47Z)
• BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation [38.017030073108735]
  We show that a tailored and suitable bilingual pre-trained language model (dubbed BiBERT) achieves state-of-the-art translation performance. Our best models achieve BLEU scores of 30.45 for En->De and 38.61 for De->En on the IWSLT'14 dataset, and 31.26 for En->De and 34.94 for De->En on the WMT'14 dataset.
  arXiv  Detail & Related papers  (2021-09-09T23:43:41Z)
• GottBERT: a pure German Language Model [0.0]
  No German single language RoBERTa model is yet published, which we introduce in this work (GottBERT) In an evaluation we compare its performance on the two Named Entity Recognition (NER) tasks Conll 2003 and GermEval 2014 as well as on the text classification tasks GermEval 2018 (fine and coarse) and GNAD with existing German single language BERT models and two multilingual ones. GottBERT was successfully pre-trained on a 256 core TPU pod using the RoBERTa BASE architecture.
  arXiv  Detail & Related papers  (2020-12-03T17:45:03Z)
• Incorporating BERT into Parallel Sequence Decoding with Adapters [82.65608966202396]
  We propose to take two different BERT models as the encoder and decoder respectively, and fine-tune them by introducing simple and lightweight adapter modules. We obtain a flexible and efficient model which is able to jointly leverage the information contained in the source-side and target-side BERT models. Our framework is based on a parallel sequence decoding algorithm named Mask-Predict considering the bi-directional and conditional independent nature of BERT.
  arXiv  Detail & Related papers  (2020-10-13T03:25:15Z)
• Nearest Neighbor Machine Translation [113.96357168879548]
  We introduce $k$-nearest-neighbor machine translation ($k$NN-MT) It predicts tokens with a nearest neighbor classifier over a large datastore of cached examples. It consistently improves performance across many settings.
  arXiv  Detail & Related papers  (2020-10-01T22:24:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.