Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation

• URL: http://arxiv.org/abs/2604.03592v1
• Date: Sat, 04 Apr 2026 04:56:35 GMT
• Title: Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation
• Abstract summary: Mixture-of-Experts (MoE) models exhibit striking performance disparities across languages.<n>We propose RISE (Routing Isolation-guided Subnetwork Enhancement), a framework that exploits routing isolation to identify and adapt language-specific expertworks.
• Score: 65.69228479949693
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mixture-of-Experts (MoE) models exhibit striking performance disparities across languages, yet the internal mechanisms driving these gaps remain poorly understood. In this work, we conduct a systematic analysis of expert routing patterns in MoE models, revealing a phenomenon we term Language Routing Isolation, in which high- and low-resource languages tend to activate largely disjoint expert sets. Through layer-stratified analysis, we further show that routing patterns exhibit a layer-wise convergence-divergence pattern across model depth. Building on these findings, we propose RISE (Routing Isolation-guided Subnetwork Enhancement), a framework that exploits routing isolation to identify and adapt language-specific expert subnetworks. RISE applies a tripartite selection strategy, using specificity scores to identify language-specific experts in shallow and deep layers and overlap scores to select universal experts in middle layers. By training only the selected subnetwork while freezing all other parameters, RISE substantially improves low-resource language performance while preserving capabilities in other languages. Experiments on 10 languages demonstrate that RISE achieves target-language F1 gains of up to 10.85% with minimal cross-lingual degradation.

Related papers

• A Declarative-Procedural Perspective on Expert Routing in Bilingual Mixture-of-Experts Language Models [0.0]
  We investigate whether Mixture-of-Experts (MoE) language models develop linguistically structured expert routing during bilingual language acquisition.<n>Inspired by the Declarative-Procedural framework, we analyze lexical, grammatical, and syntactic processing in a decoder-only English-German MoE Transformer trained under sequential language exposure.
  arXiv  Detail & Related papers  (2026-08-15T07:53:12Z)
• SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignment [74.27200765510635]
  We propose SARA (Semantically Anchored Routing Alignment), a framework designed to transfer specialized capabilities from high-resource languages to low-resource languages.<n>We conduct experiments on 2 LLMs across 5 low-resource languages and 3 benchmarks.
  arXiv  Detail & Related papers  (2026-06-24T13:36:46Z)
• Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation [50.731838216268706]
  We study multilingual routing dynamics during continual pre-training of an English-centric MoE model on a multilingual corpus.<n>We find that continual multilingual pre-training leads to diffused, language-agnostic routing in early and middle layers.<n>We propose a parameter-efficient adaptation strategy that updates language-specific and shared experts in the final MoE layers.
  arXiv  Detail & Related papers  (2026-05-28T10:12:41Z)
• Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steering [61.0787902713059]
  We propose a routing-guided steering method that adaptively guides routing behavior in middle layers toward shared experts associated with dominant languages at inference time.<n>Our code is available at http://conctsai.com/multilingualism-in-Mixture-of-Experts-LLMs.
  arXiv  Detail & Related papers  (2026-01-20T15:04:25Z)
• A Unified Framework for Emotion Recognition and Sentiment Analysis via Expert-Guided Multimodal Fusion with Large Language Models [16.195689085967004]
  We present EGMF, a unified framework combining expert-guided multimodal fusion with large language models.<n>Our approach features three specialized expert networks--a fine-grained local expert for subtle emotional nuances, a semantic correlation expert for cross-modal relationships, and a global context expert for long-range dependencies.
  arXiv  Detail & Related papers  (2026-01-12T14:21:32Z)
• Orchestrating Heterogeneous Experts: A Scalable MoE Framework with Anisotropy-Preserving Fusion [9.664206410314955]
  Cross-border e-commerce faces the dual challenge of extreme linguistic diversity and fine-grained semantic nuances.<n>Existing approaches typically rely on scaling up a single monolithic Large Language Model (LLM)<n>We propose a scalable Coarse-grained Mixture-of-Experts (MoE) framework that leverages the inherent complementarity of distinct open-source LLMs.
  arXiv  Detail & Related papers  (2025-11-18T07:13:37Z)
• Multilingual Routing in Mixture-of-Experts [45.90403983668531]
  We analyze expert routing patterns using parallel multilingual datasets.<n>We find that MoE models route tokens in language-specific ways in the early and late decoder layers.<n>We introduce a method that steers the router by promoting middle-layer task experts frequently activated in English.
  arXiv  Detail & Related papers  (2025-10-06T11:09:20Z)
• Efficient Multilingual ASR Finetuning via LoRA Language Experts [59.27778147311189]
  This paper proposes an efficient finetuning framework for customized multilingual ASR via prepared LoRA language experts based on Whisper.<n>Through LoRA expert fusion or knowledge distillation, our approach achieves better recognition performance on target languages than standard fine-tuning methods.<n> Experimental results demonstrate that the proposed models yield approximately 10% and 15% relative performance gains in language-aware and language-agnostic scenarios.
  arXiv  Detail & Related papers  (2025-06-11T07:06:27Z)
• Examining Modularity in Multilingual LMs via Language-Specialized Subnetworks [15.758213562260913]
  We investigate the degree to which language-wise modularity naturally arises within models with no special modularity interventions. We show that language-specializedworks do naturally arise, and that SFT, rather than always increasing modularity, can decrease language specialization of Attributionworks in favor of more cross-lingual sharing.
  arXiv  Detail & Related papers  (2023-11-14T16:11:23Z)
• High-resource Language-specific Training for Multilingual Neural Machine Translation [109.31892935605192]
  We propose the multilingual translation model with the high-resource language-specific training (HLT-MT) to alleviate the negative interference. Specifically, we first train the multilingual model only with the high-resource pairs and select the language-specific modules at the top of the decoder. HLT-MT is further trained on all available corpora to transfer knowledge from high-resource languages to low-resource languages.
  arXiv  Detail & Related papers  (2022-07-11T14:33:13Z)
• Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation [81.7786241489002]
  Massively multilingual models for neural machine translation (NMT) are theoretically attractive, but often underperform bilingual models and deliver poor zero-shot translations. We argue that multilingual NMT requires stronger modeling capacity to support language pairs with varying typological characteristics. We propose random online backtranslation to enforce the translation of unseen training language pairs.
  arXiv  Detail & Related papers  (2020-04-24T17:21:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.