Post-Trained MoE Can Skip Half Experts via Self-Distillation

• URL: http://arxiv.org/abs/2605.18643v1
• Date: Mon, 18 May 2026 16:50:48 GMT
• Title: Post-Trained MoE Can Skip Half Experts via Self-Distillation
• Abstract summary: Mixture-of-Experts (MoE) scales language models efficiently through sparse expert activation.<n>MoE's dynamic variant further reduces computation by adjusting the activated experts in an input-dependent manner.<n>This paper introduces Zero-Expert Self-Distillation Adaptation (ZEDA), a low-cost framework that transforms post-trained static MoE models into efficient dynamic ones.
• Score: 35.17955778301488
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mixture-of-Experts (MoE) scales language models efficiently through sparse expert activation, and its dynamic variant further reduces computation by adjusting the activated experts in an input-dependent manner. Existing dynamic MoE methods usually rely on pre-training from scratch or task-specific adaptation, leaving the practical conversion of fully trained MoE underexplored. Enabling such adaptation would directly alleviate the inference costs by allowing easy tokens to bypass unnecessary expert during serving. This paper introduces Zero-Expert Self-Distillation Adaptation (ZEDA), a low-cost framework that transforms post-trained static MoE models into efficient dynamic ones. To stabilize this architectural conversion, ZEDA injects parameter-free zero-output experts into each MoE layer and adapts the augmented model through two-stage self-distillation, utilizing the original MoE as a frozen teacher and applying a group-level balancing loss. On Qwen3-30B-A3B and GLM-4.7-Flash across 11 benchmarks spanning math, code, and instruction following, ZEDA eliminates over 50% of expert FLOPs at marginal accuracy loss. It outperforms the strongest dynamic MoE baseline by 6.1 and 4.0 points on the two models, and delivers ~1.20$\times$ end-to-end inference speedup.

Related papers

• SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcycling [58.910165627759824]
  Mixture-of-Experts (MoE) models enable efficient scaling, but training them from scratch remains prohibitively expensive.<n>MoE upcycling mitigates this cost by converting pretrained dense models into sparse MoE models.<n>We propose SVD-Partitioned Residual Initialization (SPRI), which distributes SVD-partitioned residuals derived from pretrained feed-forward network weights across routed experts.
  arXiv  Detail & Related papers  (2026-06-15T09:28:24Z)
• BEAM: Binary Expert Activation Masking for Dynamic Routing in MoE [18.898045833395095]
  Mixture-of-Experts (MoE) architectures activate only a subset of experts per token.<n>Existing acceleration methods require costly retraining with architectural changes or suffer from severe performance drop at high sparsity.<n>We propose BEAM, a novel method that learns token-adaptive expert selection via trainable binary masks.
  arXiv  Detail & Related papers  (2026-05-14T06:33:41Z)
• Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution [34.06023804017819]
  We explore and leverage intra-expert activation sparsity as a complementary and underexplored dimension of sparsity in MoE models.<n>Surprisingly, substantial intra-expert sparsity is readily available in existing pre-trained MoE models.<n>We extend the MoE execution pipeline of vLLM to leverage intra-expert activation sparsity by skipping the computations of inactive neurons.
  arXiv  Detail & Related papers  (2026-05-09T00:34:55Z)
• MC#: Mixture Compressor for Mixture-of-Experts Large Models [86.64315380917827]
  Mixture-of-Experts (MoE) effectively scales large language models (LLMs) and vision-language models (VLMs) by increasing capacity through sparse activation.<n>We propose MC# (Mixture-Compressor-sharp), a framework that combines static quantization and dynamic expert pruning.
  arXiv  Detail & Related papers  (2025-10-13T03:12:46Z)
• DualSparse-MoE: Coordinating Tensor/Neuron-Level Sparsity with Expert Partition and Reconstruction [15.261077484922616]
  Mixture of Experts (MoE) has become a mainstream architecture for building Large Language Models (LLMs)<n>We identify dual sparsity at the tensor and neuron levels in pre-trained MoE modules as a key factor for both accuracy and efficiency.<n>We propose DualSparse-MoE, an inference system that integrates dynamic tensor-level dropping with static neuron-level reconstruction.
  arXiv  Detail & Related papers  (2025-08-25T18:08:32Z)
• ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restoration [61.579842548990754]
  Mixture-of-Experts (MoE) Transformer, the backbone of multiple phenomenal language models, leverages sparsity by activating only a fraction of model parameters for each input token.<n>We introduce ResMoE, an innovative MoE approximation framework that utilizes Wasserstein barycenter to extract a common expert (barycenter expert) and approximate the residuals between this barycenter expert and the original ones.
  arXiv  Detail & Related papers  (2025-03-10T03:15:54Z)
• Union of Experts: Adapting Hierarchical Routing to Equivalently Decomposed Transformer [7.230514235208748]
  We propose Union-of-Experts (UoE), which decomposes the transformer model into an equivalent group of experts.<n>In language modeling tasks, UoE achieves an average reduction of 2.38 in perplexity compared to the best-performing MoE method.<n>In image classification, it yields an average accuracy improvement of 1.75% over the best model.
  arXiv  Detail & Related papers  (2025-03-04T11:01:25Z)
• From Dense to Dynamic: Token-Difficulty Driven MoEfication of Pre-Trained LLMs [37.50902921493273]
  Training large language models (LLMs) for different inference constraints is computationally expensive.<n> DynaMoE adapts a pre-trained dense LLM to a token-difficulty-driven Mixture-of-Experts model with minimal fine-tuning cost.<n>Our method achieves similar aggregated accuracy across downstream tasks, despite using only $frac19textth$ of their fine-tuning cost.
  arXiv  Detail & Related papers  (2025-02-17T21:12:57Z)
• MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts [63.67734699877724]
  MoE++ is a general and heterogeneous MoE framework that integrates both Feed-Forward Network(FFN) and zero-computation experts. MoE++ achieves better performance while delivering 1.1-2.1x expert forward throughput compared to a vanilla MoE model of the same size.
  arXiv  Detail & Related papers  (2024-10-09T18:01:27Z)
• Mixture Compressor for Mixture-of-Experts LLMs Gains More [71.0473038084673]
  We propose a training-free Mixture-Compressor for Mixture-of-Experts large language models (MoE-LLMs)<n>Our MC integrates static quantization and dynamic pruning to collaboratively achieve extreme compression for MoE-LLMs with less accuracy loss.<n>For instance, at 2.54 bits, MC compresses 76.6% of the model, with only a 3.8% average accuracy loss.
  arXiv  Detail & Related papers  (2024-10-08T18:09:38Z)
• SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts [49.01990048827639]
  We introduce SEER-MoE, a framework for reducing both the memory footprint and compute requirements of pre-trained MoE models. The first stage involves pruning the total number of experts using a heavy-hitters counting guidance, while the second stage employs a regularization-based fine-tuning strategy to recover accuracy loss. Our empirical studies demonstrate the effectiveness of our method, resulting in a sparse MoEs model optimized for inference efficiency with minimal accuracy trade-offs.
  arXiv  Detail & Related papers  (2024-04-07T22:13:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.