English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training

• URL: http://arxiv.org/abs/2604.13286v1
• Date: Tue, 14 Apr 2026 20:26:34 GMT
• Title: English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training
• Abstract summary: Post-training pipelines remain predominantly English-centric, contributing to performance disparities across languages.<n>We present a systematic, controlled study of the interplay between training language coverage, model scale, and task domain.<n>We find that increasing language coverage during post-training is largely beneficial across tasks and model scales.
• Score: 7.141617235075284
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the widespread multilingual deployment of large language models, post-training pipelines remain predominantly English-centric, contributing to performance disparities across languages. We present a systematic, controlled study of the interplay between training language coverage, model scale, and task domain, based on 220 supervised fine-tuning runs on parallel translated multilingual data mixtures spanning mathematical reasoning and API calling tasks, with models up to 8B parameters. We find that increasing language coverage during post-training is largely beneficial across tasks and model scales, with low-resource languages benefiting the most and high-resource languages plateauing rather than degrading. Even minimal multilinguality helps: incorporating a single non-English language improves both English performance and cross-lingual generalization, making English-only post-training largely suboptimal. Moreover, at sufficient language diversity, zero-shot cross-lingual transfer can match or exceed the effects of direct language inclusion in a low-diversity setting, although gains remain limited for typologically distant, low-resource languages.

Related papers

• Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation [50.731838216268706]
  We study multilingual routing dynamics during continual pre-training of an English-centric MoE model on a multilingual corpus.<n>We find that continual multilingual pre-training leads to diffused, language-agnostic routing in early and middle layers.<n>We propose a parameter-efficient adaptation strategy that updates language-specific and shared experts in the final MoE layers.
  arXiv  Detail & Related papers  (2026-05-28T10:12:41Z)
• Revisiting Multilingual Data Mixtures in Language Model Pretraining [20.282622416939997]
  We study the impact of different multilingual data mixtures in pretraining large language models.<n>We find that combining English and multilingual data does not necessarily degrade the in-language performance of either group.<n>We do not observe a significant "curse of multilinguality" as the number of training languages increases.
  arXiv  Detail & Related papers  (2025-10-29T20:46:03Z)
• Lens: Rethinking Multilingual Enhancement for Large Language Models [70.85065197789639]
  We propose Lens, a novel approach to enhance multilingual capabilities in large language models (LLMs)<n>Lens operates on two subspaces: the language-agnostic subspace, where it aligns target languages with the central language to inherit strong semantic representations, and the language-specific subspace, where it separates target and central languages to preserve linguistic specificity.<n>Lens significantly improves multilingual performance while maintaining the model's English proficiency, achieving better results with less computational cost compared to existing post-training approaches.
  arXiv  Detail & Related papers  (2024-10-06T08:51:30Z)
• LlamaTurk: Adapting Open-Source Generative Large Language Models for Low-Resource Language [2.9914612342004503]
  This study explores an alternative solution by adapting large language models, primarily trained on English, to low-resource languages. We assess various strategies, including continual training, instruction fine-tuning, task-specific fine-tuning, and vocabulary extension. The results show that continual training improves language comprehension, as reflected in perplexity scores, and task-specific tuning generally enhances performance of downstream tasks.
  arXiv  Detail & Related papers  (2024-05-13T13:41:59Z)
• No Train but Gain: Language Arithmetic for training-free Language Adapters enhancement [59.37775534633868]
  We introduce a novel method called language arithmetic, which enables training-free post-processing. The effectiveness of the proposed solution is demonstrated on three downstream tasks in a MAD-X-based set of cross-lingual schemes.
  arXiv  Detail & Related papers  (2024-04-24T08:52:40Z)
• Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed? [40.13166574854085]
  We investigate the minimal amount of multilinguality required to elicit cross-lingual generalisation in English-centric large language models. We find that multilingual instruction tuning with as few as two to three languages is both necessary and sufficient to elicit effective cross-lingual generalisation.
  arXiv  Detail & Related papers  (2023-12-20T00:49:52Z)
• Soft Language Clustering for Multilingual Model Pre-training [57.18058739931463]
  We propose XLM-P, which contextually retrieves prompts as flexible guidance for encoding instances conditionally. Our XLM-P enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods.
  arXiv  Detail & Related papers  (2023-06-13T08:08:08Z)
• Language-Family Adapters for Low-Resource Multilingual Neural Machine Translation [129.99918589405675]
  Large multilingual models trained with self-supervision achieve state-of-the-art results in a wide range of natural language processing tasks. Multilingual fine-tuning improves performance on low-resource languages but requires modifying the entire model and can be prohibitively expensive. We propose training language-family adapters on top of mBART-50 to facilitate cross-lingual transfer.
  arXiv  Detail & Related papers  (2022-09-30T05:02:42Z)
• Language Chameleon: Transformation analysis between languages using Cross-lingual Post-training based on Pre-trained language models [4.731313022026271]
  In this study, we focus on a single low-resource language and perform extensive evaluation and probing experiments using cross-lingual post-training (XPT) Results show that XPT not only outperforms or performs on par with monolingual models trained with orders of magnitudes more data but also is highly efficient in the transfer process.
  arXiv  Detail & Related papers  (2022-09-14T05:20:52Z)
• Discovering Representation Sprachbund For Multilingual Pre-Training [139.05668687865688]
  We generate language representation from multilingual pre-trained models and conduct linguistic analysis. We cluster all the target languages into multiple groups and name each group as a representation sprachbund. Experiments are conducted on cross-lingual benchmarks and significant improvements are achieved compared to strong baselines.
  arXiv  Detail & Related papers  (2021-09-01T09:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.