The German Commons - 154 Billion Tokens of Openly Licensed Text for German Language Models

• URL: http://arxiv.org/abs/2510.13996v1
• Date: Wed, 15 Oct 2025 18:24:26 GMT
• Title: The German Commons - 154 Billion Tokens of Openly Licensed Text for German Language Models
• Authors: Lukas Gienapp, Christopher Schröder, Stefan Schweter, Christopher Akiki, Ferdinand Schlatt, Arden Zimmermann, Phillipe Genêt, Martin Potthast,
• Abstract summary: The German Commons is the largest collection of openly licensed German text to date.<n>It compiles data from 41 sources across seven domains, encompassing legal, scientific, cultural, political, news, economic, and web text.
• Score: 41.865590656976316
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language model development relies on large-scale training corpora, yet most contain data of unclear licensing status, limiting the development of truly open models. This problem is exacerbated for non-English languages, where openly licensed text remains critically scarce. We introduce the German Commons, the largest collection of openly licensed German text to date. It compiles data from 41 sources across seven domains, encompassing legal, scientific, cultural, political, news, economic, and web text. Through systematic sourcing from established data providers with verifiable licensing, it yields 154.56 billion tokens of high-quality text for language model training. Our processing pipeline implements comprehensive quality filtering, deduplication, and text formatting fixes, ensuring consistent quality across heterogeneous text sources. All domain subsets feature licenses of at least CC-BY-SA 4.0 or equivalent, ensuring legal compliance for model training and redistribution. The German Commons therefore addresses the critical gap in openly licensed German pretraining data, and enables the development of truly open German language models. We also release code for corpus construction and data filtering tailored to German language text, rendering the German Commons fully reproducible and extensible.

Related papers

• LEMUR: A Corpus for Robust Fine-Tuning of Multilingual Law Embedding Models for Retrieval [18.46710400838861]
  Large language models (LLMs) are increasingly used to access legal information.<n>Yet, their deployment in multilingual legal settings is constrained by unreliable retrieval and the lack of domain-adapted, open-embedding models.<n>We introduce LEMUR, a large-scale multilingual corpus of EU environmental legislation constructed from 24,953 official EUR-Lex PDF documents covering 25 languages.
  arXiv  Detail & Related papers  (2026-02-10T09:20:24Z)
• Apertus: Democratizing Open and Compliant LLMs for Global Language Environments [163.70368742538187]
  Apertus is a fully open suite of large language models (LLMs) designed to address two systemic shortcomings in today's open model ecosystem.<n>Apertus models are pretrained exclusively on openly available data, respecting robots.txt exclusions and filtering for non-permissive, toxic, and personally identifiable content.<n>The Apertus models also expand multilingual coverage, training on 15T tokens from over 1800 languages, with 40% of pretraining data allocated to non-English content.
  arXiv  Detail & Related papers  (2025-09-17T17:59:21Z)
• Multilingual Language Model Pretraining using Machine-translated Data [33.373858866989536]
  We translate FineWeb-Edu, a high-quality English web dataset, into nine languages.<n>We show that TransWebLLM matches or outperforms state-of-the-art multilingual models trained using closed data.
  arXiv  Detail & Related papers  (2025-02-18T19:27:53Z)
• CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleaving [61.73180469072787]
  We focus on the problem of spoken translation (ST) of code-switched speech in Indian languages to English text. We present a new end-to-end model architecture COSTA that scaffolds on pretrained automatic speech recognition (ASR) and machine translation (MT) modules. COSTA significantly outperforms many competitive cascaded and end-to-end multimodal baselines by up to 3.5 BLEU points.
  arXiv  Detail & Related papers  (2024-06-16T16:10:51Z)
• NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languages [54.808217147579036]
  We conduct a case study on Indonesian local languages. We compare the effectiveness of online scraping, human translation, and paragraph writing by native speakers in constructing datasets. Our findings demonstrate that datasets generated through paragraph writing by native speakers exhibit superior quality in terms of lexical diversity and cultural content.
  arXiv  Detail & Related papers  (2023-09-19T14:42:33Z)
• SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore [159.21914121143885]
  We present SILO, a new language model that manages this risk-performance tradeoff during inference. SILO is built by (1) training a parametric LM on Open License Corpus (OLC), a new corpus we curate with 228B tokens of public domain and permissively licensed text. Access to the datastore greatly improves out of domain performance, closing 90% of the performance gap with an LM trained on the Pile.
  arXiv  Detail & Related papers  (2023-08-08T17:58:15Z)
• MultiLegalPile: A 689GB Multilingual Legal Corpus [20.492525119942677]
  We release MultiLegalPile, a 689GB corpus in 24 languages from 17 jurisdictions. We pretrain two RoBERTa models and one Longformer multilingually, and 24 monolingual models on each of the language-specific subsets and evaluate them on LEXTREME. Our multilingual models set a new SotA on LEXTREME and our English models on LexGLUE.
  arXiv  Detail & Related papers  (2023-06-03T10:10:38Z)
• MultiLegalSBD: A Multilingual Legal Sentence Boundary Detection Dataset [0.0]
  Sentence Boundary Detection (SBD) is one of the foundational building blocks of Natural Language Processing (NLP) We curated a diverse multilingual legal dataset consisting of over 130'000 annotated sentences in 6 languages. We trained and tested monolingual and multilingual models based on CRF, BiLSTM-CRF, and transformers, demonstrating state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-05-02T05:52:03Z)
• CoVoST 2 and Massively Multilingual Speech-to-Text Translation [24.904548615918355]
  CoVoST 2 is a large-scale multilingual speech translation corpus covering translations from 21 languages into English and from English into 15 languages. This represents the largest open dataset available to date from total volume and language coverage perspective.
  arXiv  Detail & Related papers  (2020-07-20T17:53:35Z)
• CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpus [57.641761472372814]
  CoVoST is a multilingual speech-to-text translation corpus from 11 languages into English. It diversified with over 11,000 speakers and over 60 accents. CoVoST is released under CC0 license and free to use.
  arXiv  Detail & Related papers  (2020-02-04T14:35:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.