ModernGBERT: German-only 1B Encoder Model Trained from Scratch

• URL: http://arxiv.org/abs/2505.13136v1
• Date: Mon, 19 May 2025 14:07:20 GMT
• Title: ModernGBERT: German-only 1B Encoder Model Trained from Scratch
• Authors: Anton Ehrmanntraut, Julia Wunderle, Jan Pfister, Fotis Jannidis, Andreas Hotho,
• Abstract summary: We introduce ModernGBERT (134M, 1B), a fully transparent family of German encoder models trained from scratch.<n>We also present LL"aMmlein2Vec (120M, 1B, 7B), a family of encoders derived from German decoder-only models via LLM2Vec.
• Score: 3.193989599110687
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Despite the prominence of decoder-only language models, encoders remain crucial for resource-constrained applications. We introduce ModernGBERT (134M, 1B), a fully transparent family of German encoder models trained from scratch, incorporating architectural innovations from ModernBERT. To evaluate the practical trade-offs of training encoders from scratch, we also present LL\"aMmlein2Vec (120M, 1B, 7B), a family of encoders derived from German decoder-only models via LLM2Vec. We benchmark all models on natural language understanding, text embedding, and long-context reasoning tasks, enabling a controlled comparison between dedicated encoders and converted decoders. Our results show that ModernGBERT 1B outperforms prior state-of-the-art German encoders as well as encoders adapted via LLM2Vec, with regard to performance and parameter-efficiency. All models, training data, checkpoints and code are publicly available, advancing the German NLP ecosystem with transparent, high-performance encoder models.

Related papers

• Encoder-Decoder Gemma: Improving the Quality-Efficiency Trade-Off via Adaptation [52.19855651708349]
  We study a novel problem: adapting decoder-only large language models to encoder-decoder models.<n>We argue that adaptation not only enables inheriting the capability of decoder-only LLMs but also reduces the demand for computation.<n>Under similar inference budget, encoder-decoder LLMs achieve comparable (often better) pretraining performance but substantially better finetuning performance than their decoder-only counterpart.
  arXiv  Detail & Related papers  (2025-04-08T17:13:41Z)
• Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation [40.72168378706009]
  We explore translation models that are universal, efficient, and easy to optimize.<n>We apply large language models (LLMs) to NMT encoding and leave the NMT decoder unchanged.<n>We construct a new dataset involving multiple tasks to assess how well the machine translation system generalizes.
  arXiv  Detail & Related papers  (2025-03-09T12:54:05Z)
• Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference [15.921261060193416]
  We introduce ModernBERT, bringing modern model optimizations to encoder-only models.<n>ModernBERT models exhibit state-of-the-art results on a large pool of evaluations.<n>ModernBERT is also the most speed and memory efficient encoder.
  arXiv  Detail & Related papers  (2024-12-18T09:39:44Z)
• Improving Language Transfer Capability of Decoder-only Architecture in Multilingual Neural Machine Translation [28.07831604833682]
  We investigate the issue of the decoder-only architecture to its lack of language transfer capability.<n>We propose dividing the decoding process into two stages so that target tokens are explicitly excluded in the first stage.<n>We impose contrastive learning on translation instructions, resulting in improved performance in zero-shot translation.
  arXiv  Detail & Related papers  (2024-12-03T02:52:14Z)
• Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks [4.851704512420683]
  We introduce a method for evaluating decoder models on NLU tasks and apply it to the languages Danish, Swedish, Norwegian, Icelandic, Faroese, German, Dutch, and English.<n>Our findings reveal that encoder models can achieve significantly better NLU performance than decoder models despite having orders of magnitude fewer parameters.
  arXiv  Detail & Related papers  (2024-06-19T11:50:09Z)
• BPDec: Unveiling the Potential of Masked Language Modeling Decoder in BERT pretraining [0.5919433278490629]
  BERT (Bidirectional Representations from Transformers) has revolutionized the field of natural language processing through its exceptional performance on numerous tasks.<n>DeBERTa introduced an enhanced decoder adapted for BERT's encoder model for pretraining, proving to be highly effective.<n>We argue that the design and research around enhanced masked language modeling decoders have been underappreciated.
  arXiv  Detail & Related papers  (2024-01-29T03:25:11Z)
• LegoNN: Building Modular Encoder-Decoder Models [117.47858131603112]
  State-of-the-art encoder-decoder models are constructed and trained end-to-end as an atomic unit. No component of the model can be (re-)used without the others, making it impossible to share parts. We describe LegoNN, a procedure for building encoder-decoder architectures in a way so that its parts can be applied to other tasks without the need for fine-tuning.
  arXiv  Detail & Related papers  (2022-06-07T14:08:07Z)
• Multilingual Neural Machine Translation with Deep Encoder and Multiple Shallow Decoders [77.2101943305862]
  We propose a deep encoder with multiple shallow decoders (DEMSD) where each shallow decoder is responsible for a disjoint subset of target languages. DEMSD model with 2-layer decoders is able to obtain a 1.8x speedup on average compared to a standard transformer model with no drop in translation quality.
  arXiv  Detail & Related papers  (2022-06-05T01:15:04Z)
• Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Data [145.95460945321253]
  We introduce two pre-training tasks for the encoder-decoder network using acoustic units, i.e., pseudo codes. The proposed Speech2C can relatively reduce the word error rate (WER) by 19.2% over the method without decoder pre-training.
  arXiv  Detail & Related papers  (2022-03-31T15:33:56Z)
• DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders [92.90543340071007]
  We introduce DeltaLM, a pretrained multilingual encoder-decoder model. Specifically, we augment the pretrained multilingual encoder with a decoder and pre-train it in a self-supervised way. Experiments show that DeltaLM outperforms various strong baselines on both natural language generation and translation tasks.
  arXiv  Detail & Related papers  (2021-06-25T16:12:10Z)
• Bi-Decoder Augmented Network for Neural Machine Translation [108.3931242633331]
  We propose a novel Bi-Decoder Augmented Network (BiDAN) for the neural machine translation task. Since each decoder transforms the representations of the input text into its corresponding language, jointly training with two target ends can make the shared encoder has the potential to produce a language-independent semantic space.
  arXiv  Detail & Related papers  (2020-01-14T02:05:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.