Mixture of Experts for Low-Resource LLMs

• URL: http://arxiv.org/abs/2605.17598v1
• Date: Sun, 17 May 2026 18:50:50 GMT
• Title: Mixture of Experts for Low-Resource LLMs
• Abstract summary: Mixture-of-Experts (MoE) architectures enable efficient model scaling, yet expert routing behavior across underrepresented languages remains poorly understood.<n>We analyze routing dynamics in two architecturally distinct MoE models -- a pure Transformer (Qwen3-30B-A3B) and a hybrid Mamba-Transformer (Nemotron-3-Nano-30B-A3B) -- using Hebrew as a morphologically rich, low-resource testbed.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mixture-of-Experts (MoE) architectures enable efficient model scaling, yet expert routing behavior across underrepresented languages remains poorly understood. We analyze routing dynamics in two architecturally distinct MoE models -- a pure Transformer (Qwen3-30B-A3B) and a hybrid Mamba-Transformer (Nemotron-3-Nano-30B-A3B) -- using Hebrew as a morphologically rich, low-resource testbed. Both pre-trained models exhibit \emph{deep-layer routing collapse}: usage entropy drops sharply in final layers and tokens concentrate on a narrow expert subset, a pattern largely absent for English. Continual pre-training (CPT) on balanced bilingual data substantially corrects this imbalance, increasing entropy and shifting routing toward shared, language-agnostic experts; supervised fine-tuning (SFT) alone achieves less complete correction. Extending the analysis to Japanese reveals quantitatively consistent collapse signatures, providing cross-linguistic evidence that the phenomenon is a systematic consequence of pre-training underrepresentation rather than any language-intrinsic property. Routing improvements correlate with consistent downstream benchmark gains, positioning routing entropy and expert specialization as principled diagnostics for multilingual capacity in MoE systems.

Related papers

• LatentMT: Machine Translation with Latent Reasoning [30.307790715793942]
  We introduce LatentMT, the first systematic study of latent-reasoning LoopLMs for machine translation.<n>Across 32 translation directions, LatentMT achieves performance comparable to models three to five times larger.<n>LatentMT requires lower training and inference compute than much larger non-latent-reasoning models.
  arXiv  Detail & Related papers  (2026-07-21T01:38:23Z)
• Routing-Aligned Fine-Tuning for Multilingual Downstream Tasks in Mixture-of-Experts Models [22.38900139129832]
  Existing fine-tuning approaches treat MoE models as monolithic learners, ignoring the heterogeneous routing structure that develops during pretraining.<n>We propose RA-MoE (Routing-Aligned MoE Fine-Tuning), a three-stage framework that categorizes parallel task examples into a four-way taxonomy based on correctness in English and the target language.<n>Experiments across three MoE models, three tasks, and six target languages demonstrate that RA-MoE consistently outperforms standard SFT and strong baselines.
  arXiv  Detail & Related papers  (2026-05-27T11:01:25Z)
• Unveiling Language Routing Isolation in Multilingual MoE Models for Interpretable Subnetwork Adaptation [65.69228479949693]
  Mixture-of-Experts (MoE) models exhibit striking performance disparities across languages.<n>We propose RISE (Routing Isolation-guided Subnetwork Enhancement), a framework that exploits routing isolation to identify and adapt language-specific expertworks.
  arXiv  Detail & Related papers  (2026-04-04T04:56:35Z)
• Retrofitting Earth System Models with Cadence-Limited Neural Operator Updates [3.9578288463123]
  We introduce an operator-learning framework that maps instantaneous model states to bias-correction tendencies.<n>We train two years E3SM simulations nudged toward ERA5 reanalysis, and the operators generalize across height levels and seasons.<n>Our framework emphasizes long-term stability, portability, and cadence-limited updates, demonstrating the utility of expressive ML operators.
  arXiv  Detail & Related papers  (2025-12-02T23:44:49Z)
• Orchestrating Heterogeneous Experts: A Scalable MoE Framework with Anisotropy-Preserving Fusion [9.664206410314955]
  Cross-border e-commerce faces the dual challenge of extreme linguistic diversity and fine-grained semantic nuances.<n>Existing approaches typically rely on scaling up a single monolithic Large Language Model (LLM)<n>We propose a scalable Coarse-grained Mixture-of-Experts (MoE) framework that leverages the inherent complementarity of distinct open-source LLMs.
  arXiv  Detail & Related papers  (2025-11-18T07:13:37Z)
• Mixture of Experts Made Intrinsically Interpretable [34.36996159677674]
  We present textbfMoE-X, a Mixture-of-Experts (MoE) language model designed to be emphintrinsically interpretable.<n>Our approach is motivated by the observation that, in language models, wider networks with sparse activations are more likely to capture interpretable factors.<n>MoE-X achieves perplexity better than GPT-2, with interpretability surpassing even sparse autoencoder (SAE)-based approaches.
  arXiv  Detail & Related papers  (2025-03-05T17:40:54Z)
• Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learning [73.73967342609603]
  We introduce a predictor-corrector learning framework to minimize truncation errors. We also propose an exponential moving average-based coefficient learning method to strengthen our higher-order predictor. Our model surpasses a robust 3.8B DeepNet by an average of 2.9 SacreBLEU, using only 1/3 parameters.
  arXiv  Detail & Related papers  (2024-11-05T12:26:25Z)
• Adapted Multimodal BERT with Layer-wise Fusion for Sentiment Analysis [84.12658971655253]
  We propose Adapted Multimodal BERT, a BERT-based architecture for multimodal tasks. adapter adjusts the pretrained language model for the task at hand, while the fusion layers perform task-specific, layer-wise fusion of audio-visual information with textual BERT representations. In our ablations we see that this approach leads to efficient models, that can outperform their fine-tuned counterparts and are robust to input noise.
  arXiv  Detail & Related papers  (2022-12-01T17:31:42Z)
• Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models [68.9288651177564]
  We present a novel MoE architecture based on matrix product operators (MPO) from quantum many-body physics. With the decomposed MPO structure, we can reduce the parameters of the original MoE architecture. Experiments on the three well-known downstream natural language datasets based on GPT2 show improved performance and efficiency in increasing model capacity.
  arXiv  Detail & Related papers  (2022-03-02T13:44:49Z)
• Distributionally Robust Multilingual Machine Translation [94.51866646879337]
  We propose a new learning objective for Multilingual neural machine translation (MNMT) based on distributionally robust optimization. We show how to practically optimize this objective for large translation corpora using an iterated best response scheme. Our method consistently outperforms strong baseline methods in terms of average and per-language performance under both many-to-one and one-to-many translation settings.
  arXiv  Detail & Related papers  (2021-09-09T03:48:35Z)
• Uncertainty-Aware Balancing for Multilingual and Multi-Domain Neural Machine Translation Training [58.72619374790418]
  MultiUAT dynamically adjusts the training data usage based on the model's uncertainty. We analyze the cross-domain transfer and show the deficiency of static and similarity based methods.
  arXiv  Detail & Related papers  (2021-09-06T08:30:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.