MoE Lens -- An Expert Is All You Need

• URL: http://arxiv.org/abs/2603.05806v1
• Date: Fri, 06 Mar 2026 01:35:28 GMT
• Title: MoE Lens -- An Expert Is All You Need
• Abstract summary: Mixture of Experts (MoE) models enable parameter-efficient scaling through sparse expert activations.<n>We present a systematic analysis of expert specialization in MoEs through two complementary approaches.
• Score: 0.09444932939944471
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mixture of Experts (MoE) models enable parameter-efficient scaling through sparse expert activations, yet optimizing their inference and memory costs remains challenging due to limited understanding of their specialization behavior. We present a systematic analysis of expert specialization in MoEs through two complementary approaches: domain-specific routing patterns and an early decoding framework that tracks expert contributions to output representations. Our analysis of the DeepSeekMoE model reveals that despite having 64 routed experts with 6 active for each layer's computation, the model predominantly relies on a few specialized experts, with the top-weighted expert's output closely approximating the full ensemble prediction. We quantitatively validate these findings through a systematic analysis of the token routing distribution, demonstrating that very few experts handle over 50\% of routing decisions across different specialized domains. Hidden state similarity between single and ensemble experts for every layer is extremely high, with some layers having cosine similarity as high as 0.95 and perplexity increasing by only 5\% when using a single expert across all three domains. Our results indicate that Mixture of Experts models exhibit concentrated expertise highlighting potential opportunities for inference optimization through targeted expert pruning while maintaining model performance and opening avenues towards studying localization of learned knowledge in these models.

Related papers

• DBES: A Systematic Benchmark and Metric Suite for Evaluating Expert Specialization in Large-Scale MoEs [4.089532981955943]
  Expert specialization in Mixture-of-Experts (MoE) models remains poorly understood.<n>We introduce DBES, a comprehensive diagnostic framework combining a multi-domain benchmark with five theoretically grounded metrics.<n>This work provides the first systematic methodology for evaluating expert specialization independently of accuracy metrics.
  arXiv  Detail & Related papers  (2026-05-18T14:50:30Z)
• Slicing and Dicing: Configuring Optimal Mixtures of Experts [49.5613403644084]
  We present the first systematic study of over 2,000 pretraining runs spanning models up to 6.6B total parameters.<n>We exhaustively vary total experts, expert dimension, heterogeneous expert sizing within a single layer, shared expert size and load-balancing mechanisms.<n>Our results suggest a simpler recipe: focus on expert count and granularity, other choices have minimal effect on final quality.
  arXiv  Detail & Related papers  (2026-05-12T07:47:22Z)
• On Bayesian Softmax-Gated Mixture-of-Experts Models [95.06359956519759]
  We study mixture-of-experts models, focusing on the ubiquitous softmax-based gating mechanism.<n>We analyze parameter estimation and derive convergence guarantees based on tailored Voronoi-type losses.<n> Taken together, these results provide one of the first systematic theoretical analyses of mixture-of-experts models with softmax gating.
  arXiv  Detail & Related papers  (2026-04-22T13:37:00Z)
• SD-MoE: Spectral Decomposition for Effective Expert Specialization [29.649486549025138]
  Mixture-of-Experts (MoE) architectures scale Large Language Models via expert specialization induced by conditional computation.<n>Some experts become functionally similar, while others functioning as de facto shared experts, limiting the effective capacity and model performance.<n>We propose Spectral-Decoupled MoE (SD-MoE), which decomposes both parameter and gradient in the spectral space.
  arXiv  Detail & Related papers  (2026-02-13T03:07:26Z)
• The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models [18.428606280260187]
  Mixture of Experts models are widely assumed to achieve domain specialization through sparse routing.<n>We introduce COMMITTEEAUDIT, a framework that analyzes routing behavior at the level of expert groups rather than individual experts.<n>We find that Standing Committees consistently capture the majority of routing mass across domains, layers, and routing budgets.
  arXiv  Detail & Related papers  (2026-01-06T21:29:45Z)
• Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMs [54.95810313530111]
  DERN is a task-agnostic and retraining-free framework for expert pruning and reconstruction.<n>It improves performance by more than 5% on commonsense reasoning and MMLU benchmarks under 50% expert sparsity.
  arXiv  Detail & Related papers  (2025-09-12T16:09:39Z)
• On DeepSeekMoE: Statistical Benefits of Shared Experts and Normalized Sigmoid Gating [75.29576838162714]
  DeepSeekMoE stands out because of two unique features: the deployment of a shared expert strategy and of the normalized sigmoid gating mechanism.<n>We perform a convergence analysis of the expert estimation task to highlight the gains in sample efficiency for both the shared expert strategy and the normalized sigmoid gating.
  arXiv  Detail & Related papers  (2025-05-16T04:58:18Z)
• Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations [48.890534958441016]
  This study investigates domain specialization and expert redundancy in large-scale MoE models.<n>We propose a simple yet effective pruning framework, EASY-EP, to identify and retain only the most relevant experts.<n>Experiments on DeepSeek-R1 and DeepSeek-V3-0324 show that our method can achieve comparable performances and $2.99times$ throughput under the same memory budget with full model with only half the experts.
  arXiv  Detail & Related papers  (2025-04-09T11:34:06Z)
• Convergence Rates for Softmax Gating Mixture of Experts [78.3687645289918]
  Mixture of experts (MoE) has emerged as an effective framework to advance the efficiency and scalability of machine learning models.<n>Central to the success of MoE is an adaptive softmax gating mechanism which takes responsibility for determining the relevance of each expert to a given input and then dynamically assigning experts their respective weights.<n>We perform a convergence analysis of parameter estimation and expert estimation under the MoE equipped with the standard softmax gating or its variants, including a dense-to-sparse gating and a hierarchical softmax gating.
  arXiv  Detail & Related papers  (2025-03-05T06:11:24Z)
• Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study [65.11303133775857]
  Mixture-of-Experts (MoE) computation amalgamates predictions from several specialized sub-models (referred to as experts) Sparse MoE selectively engages only a limited number, or even just one expert, significantly reducing overhead while empirically preserving, and sometimes even enhancing, performance.
  arXiv  Detail & Related papers  (2024-03-26T05:48:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.