Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models

• URL: http://arxiv.org/abs/2609.04575v1
• Date: Fri, 04 Sep 2026 00:11:28 GMT
• Title: Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models
• Abstract summary: We show that renormalization implicitly calibrates expert output gain to the training top-$k$.<n>We find that removing renormalization entirely is catastrophic, showing that preserving a suitable reference mass is crucial.
• Score: 3.749501286035187
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern fine-grained Mixture-of-Experts (MoE) models route each token to a small number of experts and renormalize their router probabilities. We show that this renormalization implicitly calibrates expert output gain to the training top-$k$: reducing $k$ at inference changes not only which experts are used but also the strength of the expert branch. We separate these effects by activating the top $k_1$ experts while normalizing by the probability mass of the top $k_2$ experts, introducing one integer with no parameters, training, or measurable compute overhead. On Qwen3.6-35B-A3B, reducing from 8 to 4 experts causes a 4.65-point MMLU drop under standard renormalization but only 0.35 points with $k_2=16$, while halving routed-expert compute. The result replicates on the $11\times$ larger Qwen3.5-397B-A17B, where reducing from 10 to 5 experts loses only 0.55 points with an appropriate reference set. Removing renormalization entirely is catastrophic, showing that preserving a suitable reference mass is crucial. We further find that perplexity and downstream accuracy favor different $k_2$, cautioning against selecting MoE compression settings using unlabeled text alone. Analyses also show that expert identity matters substantially more than expert weighting, while balanced and domain-specialized routing leaves limited room for expert pruning.

Related papers

• RAPTOR: Role-Aware Private Training for Mixture-of-Experts [68.5626338219096]
  RAPTOR is a Role-Aware Private Training framework.<n>RAPTOR alternates shared and expert optimization and targets each failure directly.<n> Experiments on Switch Transformer and OLMoE fine-tuning across GLUE tasks, and on DeepSeek-VL2-Tiny, show consistent gains over standard DP baselines.
  arXiv  Detail & Related papers  (2026-09-04T23:16:12Z)
• Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models [16.871402640095745]
  We propose a coverage-aware expert pruning method that uses only generic text corpora for calibration.<n>Our improvements hold with fixed pruning budgets and no downstream calibration data.
  arXiv  Detail & Related papers  (2026-07-02T05:02:18Z)
• Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression [74.00650541246374]
  We propose a structural pruning framework tailored for MoE models.<n>Our approach reduces memory footprint by 5.27$times and consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-06-16T06:53:27Z)
• TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts [57.78282303757474]
  We propose TENP, a structured Trapezoidal ExpertNeuron Pruning framework.<n>We identify and retain important experts, while applying expert neuron pruning (ENP) to less important experts.<n>For ENP, we measure each neuron's projected contribution to the expert output to identify and retain important neurons.
  arXiv  Detail & Related papers  (2026-06-03T07:49:31Z)
• $\infty$-MoE: Generalizing Mixture of Experts to Infinite Experts [43.075289015406355]
  Mixture of Experts (MoE) selects a few feed-forward networks (FFNs) per token, achieving an effective trade-off between computational cost and performance.<n>We propose $infty$-MoE that selects a portion of the parameters of large FFNs based on continuous values sampled for each token.<n>Experiments show that a GPT-2 Small-based $infty$-MoE model, with 129M active and 186M total parameters, achieves comparable performance to a dense GPT-2 Medium with 350M parameters.
  arXiv  Detail & Related papers  (2026-01-25T03:55:51Z)
• Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMs [54.95810313530111]
  DERN is a task-agnostic and retraining-free framework for expert pruning and reconstruction.<n>It improves performance by more than 5% on commonsense reasoning and MMLU benchmarks under 50% expert sparsity.
  arXiv  Detail & Related papers  (2025-09-12T16:09:39Z)
• Unified Sparse Mixture of Experts [14.774596844618396]
  Sparse Mixture of Experts (SMoEs) models scale the capacity of models while maintaining constant computational overhead.<n>This paper proposes a Unified Sparse Mixture of Experts (USMoE) framework that addresses these limitations.
  arXiv  Detail & Related papers  (2025-03-29T07:15:12Z)
• Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization [51.98792406392873]
  Mixture of Experts (MoE) provides a powerful way to decompose dense layers into smaller, modular computations. A major challenge lies in the computational cost of scaling the number of experts high enough to achieve fine-grained specialization. We propose the Multilinear Mixture of Experts ($mu$MoE) layer to address this, focusing on vision models.
  arXiv  Detail & Related papers  (2024-02-19T21:20:22Z)
• Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy [84.11508381847929]
  Sparsely activated Mixture-of-Experts (SMoE) has shown promise to scale up the learning capacity of neural networks. We propose M-SMoE, which leverages routing statistics to guide expert merging. Our MC-SMoE achieves up to 80% memory and a 20% FLOPs reduction, with virtually no loss in performance.
  arXiv  Detail & Related papers  (2023-10-02T16:51:32Z)
• Mixture-of-Experts with Expert Choice Routing [44.777850078713634]
  Prior work allocates a fixed number of experts to each token using a top-k function. We propose a heterogeneous mixture-of-experts employing an expert choice method. Our method improves training convergence time by more than 2x.
  arXiv  Detail & Related papers  (2022-02-18T17:46:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.