Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs

• URL: http://arxiv.org/abs/2605.24681v1
• Date: Sat, 23 May 2026 17:35:43 GMT
• Title: Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs
• Abstract summary: Large Language Models (LLMs) have shown great promise in multilingual machine translation (MT)<n>We propose Mix-MoE, a mixed Mixture-of-Experts framework designed to train LLMs for multilingual MT.<n>Our framework operates in two distinct stages: (1) post-pretraining with MoE on monolingual corpora, and (2) post-pretraining with MoE on parallel corpora.
• Score: 58.06302309139978
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) have shown great promise in multilingual machine translation (MT), even with limited bilingual supervision. However, fine-tuning LLMs with parallel corpora presents major challenges, namely parameter interference. To address these issues, we propose Mix-MoE, a mixed Mixture-of-Experts framework designed to train LLMs for multilingual MT. Our framework operates in two distinct stages: (1) post-pretraining with MoE on monolingual corpora, and (2) post-pretraining with MoE on parallel corpora. Crucially, we divide the MoE layers into two specialized groups: Language Model Experts (LM Experts) and Machine Translation Experts (MT Experts). LM Experts are designed to capture and retain the monolingual knowledge learned by the pre-trained LLM. MT Experts, on the other hand, are specifically trained to acquire and store bilingual translation knowledge. Furthermore, to facilitate effective interaction between these specialized experts and leverage potential underlying structural patterns in text, we introduce a routing mechanism enhanced by Fourier Transform features derived from model representations. The experimental results demonstrate that Mix-MoE excels in multilingual MT, significantly outperforming existing baselines and showing notable progress in mitigating parameter interference.

Related papers

• Scaling Model and Data for Multilingual Machine Translation with Open Large Language Models [38.626788948945844]
  We present a study of open large language models (LLMs) for multilingual machine translation (MT) across a range of languages.<n>We develop MiLMMT-46, which achieves top-tier multilingual translation performance across 46 languages.
  arXiv  Detail & Related papers  (2026-02-12T13:56:02Z)
• MiLorE-SSL: Scaling Multilingual Capabilities in Self-Supervised Models without Forgetting [69.6938830307759]
  MiLorE-SSL is a lightweight framework that combines LoRA modules with a soft mixture-of-experts mechanism for efficient continual multilingual training.<n>LoRA provides efficient low-rank adaptation, while soft MoE promotes flexible expert sharing across languages, reducing cross-lingual interference.<n>Experiments on ML-SUPERB demonstrate that MiLorE-SSL achieves strong performance in new languages and improves the ability in existing ones with only 2.14% trainable parameters.
  arXiv  Detail & Related papers  (2026-01-28T06:48:52Z)
• Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steering [61.0787902713059]
  We propose a routing-guided steering method that adaptively guides routing behavior in middle layers toward shared experts associated with dominant languages at inference time.<n>Our code is available at http://conctsai.com/multilingualism-in-Mixture-of-Experts-LLMs.
  arXiv  Detail & Related papers  (2026-01-20T15:04:25Z)
• Multilingual Routing in Mixture-of-Experts [45.90403983668531]
  We analyze expert routing patterns using parallel multilingual datasets.<n>We find that MoE models route tokens in language-specific ways in the early and late decoder layers.<n>We introduce a method that steers the router by promoting middle-layer task experts frequently activated in English.
  arXiv  Detail & Related papers  (2025-10-06T11:09:20Z)
• MultiPL-MoE: Multi-Programming-Lingual Extension of Large Language Models through Hybrid Mixture-of-Experts [56.106778414865126]
  MultiPL-MoE is a hybrid mixture of experts and token-level experts.<n>The segment-level MoE incorporates two innovative designs to better capture the syntactic structure and contextual patterns of programming languages.
  arXiv  Detail & Related papers  (2025-08-22T06:24:52Z)
• MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models [57.091523832149655]
  We propose a mixture of multimodal experts (MoME) to mitigate task interference and obtain a generalist MLLM. Our MoME is composed of two key components, a mixture of vision experts (MoVE) and a mixture of language experts (MoLE)
  arXiv  Detail & Related papers  (2024-07-17T16:31:38Z)
• Multi-Head Mixture-of-Experts [100.60556163597946]
  We propose Multi-Head Mixture-of-Experts (MH-MoE), which employs a multi-head mechanism to split each token into multiple sub-tokens. MH-MoE is straightforward to implement and decouples from other SMoE optimization methods, making it easy to integrate with other SMoE models for enhanced performance.
  arXiv  Detail & Related papers  (2024-04-23T13:47:09Z)
• On-the-Fly Fusion of Large Language Models and Machine Translation [3.718665608549311]
  We propose the on-the-fly ensembling of a machine translation model with an LLM prompted on the same task and input. We find that a slightly weaker-at-translation LLM can improve translations of a NMT model, and ensembling with an LLM can produce better translations than ensembling two stronger MT models.
  arXiv  Detail & Related papers  (2023-11-14T16:49:33Z)
• Simultaneous Machine Translation with Large Language Models [51.470478122113356]
  We investigate the possibility of applying Large Language Models to SimulMT tasks. We conducted experiments using the textttLlama2-7b-chat model on nine different languages from the MUST-C dataset. The results show that LLM outperforms dedicated MT models in terms of BLEU and LAAL metrics.
  arXiv  Detail & Related papers  (2023-09-13T04:06:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.