Slicing and Dicing: Configuring Optimal Mixtures of Experts

• URL: http://arxiv.org/abs/2605.11689v1
• Date: Tue, 12 May 2026 07:47:22 GMT
• Title: Slicing and Dicing: Configuring Optimal Mixtures of Experts
• Abstract summary: We present the first systematic study of over 2,000 pretraining runs spanning models up to 6.6B total parameters.<n>We exhaustively vary total experts, expert dimension, heterogeneous expert sizing within a single layer, shared expert size and load-balancing mechanisms.<n>Our results suggest a simpler recipe: focus on expert count and granularity, other choices have minimal effect on final quality.
• Score: 49.5613403644084
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mixture-of-Experts (MoE) architectures have become standard in large language models, yet many of their core design choices - expert count, granularity, shared experts, load balancing, token dropping - have only been studied one or two at a time over narrow configuration ranges. It remains an open question whether these choices can be optimized independently, without considering interactions. We present the first systematic study of over 2,000 pretraining runs spanning models up to 6.6B total parameters, in which we exhaustively vary total experts, expert dimension, heterogeneous expert sizing within a single layer, shared expert size and load-balancing mechanisms. We find that at every active-parameter scale that we study, performance consistently improves with total MoE parameters even at extreme active expert parameter ratios like 128.Further, the optimal expert size is nearly invariant to total parameter count and depends only on active parameter count. Third, we see that other choices like shared experts, heterogeneous experts and load-balancing settings have small effects relative to expert count and granularity, although dropless routing yields a consistent gain. Overall, our results suggest a simpler recipe: focus on expert count and granularity, other choices have minimal effect on final quality.

Related papers

• TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts [57.78282303757474]
  We propose TENP, a structured Trapezoidal ExpertNeuron Pruning framework.<n>We identify and retain important experts, while applying expert neuron pruning (ENP) to less important experts.<n>For ENP, we measure each neuron's projected contribution to the expert output to identify and retain important neurons.
  arXiv  Detail & Related papers  (2026-06-03T07:49:31Z)
• MoE Lens -- An Expert Is All You Need [0.09444932939944471]
  Mixture of Experts (MoE) models enable parameter-efficient scaling through sparse expert activations.<n>We present a systematic analysis of expert specialization in MoEs through two complementary approaches.
  arXiv  Detail & Related papers  (2026-03-06T01:35:28Z)
• $\infty$-MoE: Generalizing Mixture of Experts to Infinite Experts [43.075289015406355]
  Mixture of Experts (MoE) selects a few feed-forward networks (FFNs) per token, achieving an effective trade-off between computational cost and performance.<n>We propose $infty$-MoE that selects a portion of the parameters of large FFNs based on continuous values sampled for each token.<n>Experiments show that a GPT-2 Small-based $infty$-MoE model, with 129M active and 186M total parameters, achieves comparable performance to a dense GPT-2 Medium with 350M parameters.
  arXiv  Detail & Related papers  (2026-01-25T03:55:51Z)
• Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations [48.890534958441016]
  This study investigates domain specialization and expert redundancy in large-scale MoE models.<n>We propose a simple yet effective pruning framework, EASY-EP, to identify and retain only the most relevant experts.<n>Experiments on DeepSeek-R1 and DeepSeek-V3-0324 show that our method can achieve comparable performances and $2.99times$ throughput under the same memory budget with full model with only half the experts.
  arXiv  Detail & Related papers  (2025-04-09T11:34:06Z)
• Convergence Rates for Softmax Gating Mixture of Experts [78.3687645289918]
  Mixture of experts (MoE) has emerged as an effective framework to advance the efficiency and scalability of machine learning models.<n>Central to the success of MoE is an adaptive softmax gating mechanism which takes responsibility for determining the relevance of each expert to a given input and then dynamically assigning experts their respective weights.<n>We perform a convergence analysis of parameter estimation and expert estimation under the MoE equipped with the standard softmax gating or its variants, including a dense-to-sparse gating and a hierarchical softmax gating.
  arXiv  Detail & Related papers  (2025-03-05T06:11:24Z)
• Mixture of Efficient Diffusion Experts Through Automatic Interval and Sub-Network Selection [63.96018203905272]
  We propose to reduce the sampling cost by pruning a pretrained diffusion model into a mixture of efficient experts. We demonstrate the effectiveness of our method, DiffPruning, across several datasets.
  arXiv  Detail & Related papers  (2024-09-23T21:27:26Z)
• Mixture of Diverse Size Experts [13.29015039603752]
  The Sparsely-Activated Mixture-of-Experts (MoE) has gained increasing popularity for scaling up large language models (LLMs) without exploding computational costs. We propose the Mixture of Diverse Size Experts (MoDSE), a new MoE architecture with layers designed to have experts of different sizes.
  arXiv  Detail & Related papers  (2024-09-18T08:23:27Z)
• Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study [65.11303133775857]
  Mixture-of-Experts (MoE) computation amalgamates predictions from several specialized sub-models (referred to as experts) Sparse MoE selectively engages only a limited number, or even just one expert, significantly reducing overhead while empirically preserving, and sometimes even enhancing, performance.
  arXiv  Detail & Related papers  (2024-03-26T05:48:02Z)
• Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning [7.094820944028638]
  We propose an extremely parameter-efficient MoE by combining MoE architecture with lightweight experts. Our method generalizes to unseen tasks as it does not depend on any prior task knowledge. Our research underscores the versatility of the mixture of experts architecture, showcasing its ability to deliver robust performance even when subjected to rigorous parameter constraints.
  arXiv  Detail & Related papers  (2023-09-11T13:31:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.