EMO: Frustratingly Easy Progressive Training of Extendable MoE

• URL: http://arxiv.org/abs/2605.13247v2
• Date: Thu, 14 May 2026 02:03:27 GMT
• Title: EMO: Frustratingly Easy Progressive Training of Extendable MoE
• Abstract summary: We argue that current MoE training allocates too many experts from the beginning, even though early-stage data may not fully utilize such capacity.<n>Motivated by this, we propose EMO, a simple progressive training framework that treats MoE capacity as expandable memory and grows the expert pool over the course of training.
• Score: 21.653288033373745
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sparse Mixture-of-Experts (MoE) models offer a powerful way to scale model size without increasing compute, as per-token FLOPs depend only on k active experts rather than the total pool of E experts. Yet, this asymmetry creates an MoE efficiency paradox in practice: adding more experts balloons memory and communication costs, making actual training inefficient. We argue that this bottleneck arises in part because current MoE training allocates too many experts from the beginning, even though early-stage data may not fully utilize such capacity. Motivated by this, we propose EMO, a simple progressive training framework that treats MoE capacity as expandable memory and grows the expert pool over the course of training. EMO explicitly models sparsity in scaling law to derive stage-wise compute-optimal token budgets for progressive expansion. Empirical results show that EMO matches the performance of a fixed-expert setup in large-scale experiments while improving wall-clock efficiency. It offers a surprisingly simple yet effective path to scalable MoE training, preserving the benefits of large expert pools while reducing both training time and GPU cost.

Related papers

• MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Models [55.732137666068446]
  Universal multimodal embedding (UME) increasingly demands encoder's capacity for handling a broad range of tasks and modalities.<n>MoEMB scales UME along the expert axis through mixture-of-experts (MoE)<n>MoEMB surpasses TTE-based methods with >4x active parameters, using significantly less computes.
  arXiv  Detail & Related papers  (2026-09-08T12:29:50Z)
• Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution [34.06023804017819]
  We explore and leverage intra-expert activation sparsity as a complementary and underexplored dimension of sparsity in MoE models.<n>Surprisingly, substantial intra-expert sparsity is readily available in existing pre-trained MoE models.<n>We extend the MoE execution pipeline of vLLM to leverage intra-expert activation sparsity by skipping the computations of inactive neurons.
  arXiv  Detail & Related papers  (2026-05-09T00:34:55Z)
• LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing [45.54173919316335]
  We propose a novel expert compression paradigm termed expert replacing.<n>It replaces redundant experts with parameter-efficient modules and recovers their capabilities with low training costs.<n> Experimental results show that LightMoE matches the performance of LoRA fine-tuning at a 30% compression ratio.
  arXiv  Detail & Related papers  (2026-03-13T04:33:08Z)
• Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization [60.309915093470416]
  Matryoshka MoE (M-MoE) is a training framework that instills a coarse-to-fine structure directly into the expert ensemble.<n>Our work paves the way for more practical and adaptable deployments of large-scale MoE models.
  arXiv  Detail & Related papers  (2025-09-30T16:56:44Z)
• Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts [43.63398524449102]
  Mixture-of-Experts (MoE) models typically fix the number of activated experts $k$ at both training and inference.<n>We introduce Elastic Mixture-of-Experts (EMoE), a novel training framework that enables MoE models to scale the number of activated experts at inference without incurring additional training overhead.
  arXiv  Detail & Related papers  (2025-09-26T05:29:19Z)
• Faster MoE LLM Inference for Extremely Large Models [75.57674991584608]
  Fine-grained MoE models are gaining popularity, yet research on them remains limited.<n>Reducing the number of activated experts can lead to substantial efficiency improvements in certain scenarios.<n>Our method can increase throughput by at least 10% without any performance degradation.
  arXiv  Detail & Related papers  (2025-05-06T13:41:17Z)
• Accelerating MoE Model Inference with Expert Sharding [1.4733737463429546]
  Mixture of experts (MoE) models achieve state-of-the-art results in language modeling but suffer from inefficient hardware utilization due to imbalanced token routing and communication overhead.<n>We introduce MoEShard, an inference system that achieves perfect load balancing through tensor sharding of MoE experts.
  arXiv  Detail & Related papers  (2025-03-11T14:15:01Z)
• SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts [49.01990048827639]
  We introduce SEER-MoE, a framework for reducing both the memory footprint and compute requirements of pre-trained MoE models. The first stage involves pruning the total number of experts using a heavy-hitters counting guidance, while the second stage employs a regularization-based fine-tuning strategy to recover accuracy loss. Our empirical studies demonstrate the effectiveness of our method, resulting in a sparse MoEs model optimized for inference efficiency with minimal accuracy trade-offs.
  arXiv  Detail & Related papers  (2024-04-07T22:13:43Z)
• Toward Inference-optimal Mixture-of-Expert Large Language Models [55.96674056805708]
  We study the scaling law of MoE-based large language models (LLMs) We find that MoEs with a few (4/8) experts are the most serving efficient solution under the same performance, but costs 2.5-3.5x more in training. We propose to amend the scaling law of MoE by introducing inference efficiency as another metric besides the validation loss.
  arXiv  Detail & Related papers  (2024-04-03T16:33:42Z)
• Task-Specific Expert Pruning for Sparse Mixture-of-Experts [105.20605021416276]
  Mixture-of-Experts (MoE) model is powerful for large-scale pre-training. MoE is hard to be deployed on cloud or mobile environment. We propose a general method to progressively drop the non-professional experts for the target downstream task.
  arXiv  Detail & Related papers  (2022-06-01T07:09:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.