KORMo: Korean Open Reasoning Model for Everyone

• URL: http://arxiv.org/abs/2510.09426v1
• Date: Fri, 10 Oct 2025 14:31:25 GMT
• Title: KORMo: Korean Open Reasoning Model for Everyone
• Authors: Minjun Kim, Hyeonseok Lim, Hangyeol Yoo, Inho Won, Seungwoo Song, Minkyung Cho, Junhun Yuk, Changsu Choi, Dongjae Shin, Huige Lee, Hoyun Song, Alice Oh, Kyungtae Lim,
• Abstract summary: This work presents the first large-scale investigation into constructing a fully open bilingual large language model (LLM) for a non-English language, specifically Korean, trained predominantly on synthetic data.<n>We demonstrate that synthetic data, when carefully curated with balanced linguistic coverage and diverse instruction styles, does not cause instability or degradation during large-scale pretraining.<n>Our experiments reveal two key findings: (1) synthetic data can reliably sustain long-horizon pretraining without model collapse, and (2) bilingual instruction tuning enables near-native reasoning and discourse coherence in Korean.
• Score: 24.596298830917394
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This work presents the first large-scale investigation into constructing a fully open bilingual large language model (LLM) for a non-English language, specifically Korean, trained predominantly on synthetic data. We introduce KORMo-10B, a 10.8B-parameter model trained from scratch on a Korean-English corpus in which 68.74% of the Korean portion is synthetic. Through systematic experimentation, we demonstrate that synthetic data, when carefully curated with balanced linguistic coverage and diverse instruction styles, does not cause instability or degradation during large-scale pretraining. Furthermore, the model achieves performance comparable to that of contemporary open-weight multilingual baselines across a wide range of reasoning, knowledge, and instruction-following benchmarks. Our experiments reveal two key findings: (1) synthetic data can reliably sustain long-horizon pretraining without model collapse, and (2) bilingual instruction tuning enables near-native reasoning and discourse coherence in Korean. By fully releasing all components including data, code, training recipes, and logs, this work establishes a transparent framework for developing synthetic data-driven fully open models (FOMs) in low-resource settings and sets a reproducible precedent for future multilingual LLM research.

Related papers

• Gaperon: A Peppered English-French Generative Language Model Suite [25.492050653893184]
  Gaperon is a fully open suite of French-English-coding language models.<n>We study how data filtering and contamination interact to shape both benchmark and generative performance.
  arXiv  Detail & Related papers  (2025-10-29T17:59:39Z)
• Synthetic bootstrapped pretraining [52.92577542049469]
  We introduce Synthetic Bootstrapped Pretraining (SBP), a language model (LM) pretraining procedure.<n>SBP first learns a model of relations between documents from the pretraining dataset and then leverages it to synthesize a vast new corpus for joint training.<n>We find SBP consistently improves upon a strong repetition baseline and delivers a significant fraction of performance improvement attainable by an oracle upper bound.
  arXiv  Detail & Related papers  (2025-09-17T22:28:27Z)
• Seed-X: Building Strong Multilingual Translation LLM with 7B Parameters [53.59868121093848]
  We introduce Seed-X, a family of open-source language models (LLMs) with 7B parameter size.<n>The base model is pre-trained on a diverse, high-quality dataset encompassing both monolingual and bilingual content across 28 languages.<n>The instruct model is then finetuned to translate by Chain-of-Thought (CoT) reasoning and further enhanced through reinforcement learning (RL) to achieve better generalization across diverse language pairs.
  arXiv  Detail & Related papers  (2025-07-18T03:19:43Z)
• LDP: Generalizing to Multilingual Visual Information Extraction by Language Decoupled Pretraining [2.6638517946494535]
  We propose a multilingual training paradigm LDP (Language Decoupled Pre-training) for better utilization of monolingual pre-training data.<n>Our proposed model LDM is first pre-trained on the language-independent data, where the language knowledge is decoupled by a diffusion model, and then the LDM is fine-tuned on the downstream languages.
  arXiv  Detail & Related papers  (2024-12-19T07:31:40Z)
• Does Incomplete Syntax Influence Korean Language Model? Focusing on Word Order and Case Markers [7.275938266030414]
  Syntactic elements, such as word order and case markers, are fundamental in natural language processing. This study explores whether Korean language models can accurately capture this flexibility.
  arXiv  Detail & Related papers  (2024-07-12T11:33:41Z)
• Fine-Tuned Language Models Generate Stable Inorganic Materials as Text [53.81190146434045]
  Fine-tuning large language models on text-encoded atomistic data is simple to implement yet reliable.<n>We show that our strongest model can generate materials predicted to be metastable at about twice the rate of CDVAE.<n>Because of text prompting's inherent flexibility, our models can simultaneously be used for unconditional generation of stable material.
  arXiv  Detail & Related papers  (2024-02-06T20:35:28Z)
• CroissantLLM: A Truly Bilingual French-English Language Model [42.03897426049679]
  We introduce CroissantLLM, a 1.3B language model pretrained on a set of 3T English and French tokens.<n>We pioneer the approach of training an intrinsically bilingual model with a 1:1 English-to-French pretraining data ratio.<n>To assess performance outside of English, we craft a novel benchmark, FrenchBench.
  arXiv  Detail & Related papers  (2024-02-01T17:17:55Z)
• YAYI 2: Multilingual Open-Source Large Language Models [53.92832054643197]
  We propose YAYI 2, including both base and chat models, with 30 billion parameters. YAYI 2 is pre-trained from scratch on a multilingual corpus which contains 2.65 trillion tokens filtered by our pre-training data processing pipeline. The base model is aligned with human values through supervised fine-tuning with millions of instructions and reinforcement learning from human feedback.
  arXiv  Detail & Related papers  (2023-12-22T17:34:47Z)
• Synthetic Pre-Training Tasks for Neural Machine Translation [16.6378815054841]
  Our goal is to understand the factors that contribute to the effectiveness of pre-training models when using synthetic resources. We propose several novel approaches to pre-training translation models that involve different levels of lexical and structural knowledge. Our experiments on multiple language pairs reveal that pre-training benefits can be realized even with high levels of obfuscation or purely synthetic parallel data.
  arXiv  Detail & Related papers  (2022-12-19T21:34:00Z)
• KLUE: Korean Language Understanding Evaluation [43.94952771238633]
  We introduce Korean Language Understanding Evaluation (KLUE) benchmark. KLUE is a collection of 8 Korean natural language understanding (NLU) tasks. We build all of the tasks from scratch from diverse source corpora while respecting copyrights.
  arXiv  Detail & Related papers  (2021-05-20T11:40:30Z)
• Beyond English-Centric Multilingual Machine Translation [74.21727842163068]
  We create a true Many-to-Many multilingual translation model that can translate directly between any pair of 100 languages. We build and open source a training dataset that covers thousands of language directions with supervised data, created through large-scale mining. Our focus on non-English-Centric models brings gains of more than 10 BLEU when directly translating between non-English directions while performing competitively to the best single systems of WMT.
  arXiv  Detail & Related papers  (2020-10-21T17:01:23Z)
• Mixed-Lingual Pre-training for Cross-lingual Summarization [54.4823498438831]
  Cross-lingual Summarization aims at producing a summary in the target language for an article in the source language. We propose a solution based on mixed-lingual pre-training that leverages both cross-lingual tasks like translation and monolingual tasks like masked language models. Our model achieves an improvement of 2.82 (English to Chinese) and 1.15 (Chinese to English) ROUGE-1 scores over state-of-the-art results.
  arXiv  Detail & Related papers  (2020-10-18T00:21:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.