TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts

• URL: http://arxiv.org/abs/2606.09885v1
• Date: Wed, 03 Jun 2026 07:49:31 GMT
• Title: TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts
• Abstract summary: We propose TENP, a structured Trapezoidal ExpertNeuron Pruning framework.<n>We identify and retain important experts, while applying expert neuron pruning (ENP) to less important experts.<n>For ENP, we measure each neuron's projected contribution to the expert output to identify and retain important neurons.
• Score: 57.78282303757474
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mixture-of-Experts large language models (LLMs) scale efficiently through sparse activation, yet their deployment is fundamentally constrained by the large static parameter footprint of experts. Existing compression approaches either remove entire experts, disrupting routing topology and harming performance, or rely on unstructured weight pruning with limited practical efficiency. To address the limitations, we propose TENP, a structured Trapezoidal ExpertNeuron Pruning framework. Using a few samples, we identify and retain important experts, while applying expert neuron pruning (ENP) to less important experts, reserving model parameters in a trapezoidal pattern from shallow to deep layers. When evaluating expert importance, we jointly consider both the magnitude of the expert output and its ability to change the direction of the input vector. For ENP, we measure each neuron's projected contribution to the expert output to identify and retain important neurons. We conduct extensive experiments on the Qwen and DeepSeek models. Under a routing expert sparsity of 40% and an average of 63.76% activated expert parameters, the DeepSeek model suffers only a 1-point drop in accuracy compared to the full-parameter model. Moreover, it outperforms the full-parameter model by 10% on code generation tasks.

Related papers

• Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression [74.00650541246374]
  We propose a structural pruning framework tailored for MoE models.<n>Our approach reduces memory footprint by 5.27$times and consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-06-16T06:53:27Z)
• Slicing and Dicing: Configuring Optimal Mixtures of Experts [49.5613403644084]
  We present the first systematic study of over 2,000 pretraining runs spanning models up to 6.6B total parameters.<n>We exhaustively vary total experts, expert dimension, heterogeneous expert sizing within a single layer, shared expert size and load-balancing mechanisms.<n>Our results suggest a simpler recipe: focus on expert count and granularity, other choices have minimal effect on final quality.
  arXiv  Detail & Related papers  (2026-05-12T07:47:22Z)
• Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees [13.67631667766697]
  Sparse Mixture-of-Experts (MoE) allows scaling of language and vision models efficiently by activating only a small subset of experts per input.<n>MoE incurs substantial memory overhead during inference.<n>We propose a theoretically grounded expert-wise mixed precision strategy that assigns bit-width to each expert based on their change in routers l2 norm during training.
  arXiv  Detail & Related papers  (2026-04-07T23:17:23Z)
• REAP the Experts: Why Pruning Prevails for One-Shot MoE compression [5.517309667435783]
  We show that expert pruning is a superior strategy for generative tasks.<n>We propose Router-weighted Expert Activation Pruning (REAP)<n>Our method achieves near-lossless compression on code generation and tool-calling tasks with Qwen3-Coder-480B and Kimi-K2.
  arXiv  Detail & Related papers  (2025-10-15T18:29:28Z)
• Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMs [54.95810313530111]
  DERN is a task-agnostic and retraining-free framework for expert pruning and reconstruction.<n>It improves performance by more than 5% on commonsense reasoning and MMLU benchmarks under 50% expert sparsity.
  arXiv  Detail & Related papers  (2025-09-12T16:09:39Z)
• MEPT: Mixture of Expert Prompt Tuning as a Manifold Mapper [75.6582687942241]
  We propose Mixture of Expert Prompt Tuning (MEPT) as an effective and efficient manifold-mapping framework.<n>MEPT integrates multiple prompt experts to adaptively learn diverse and non-stationary data distributions.<n> Empirical evaluations demonstrate that MEPT outperforms several state-of-the-art parameter efficient baselines on SuperGLUE.
  arXiv  Detail & Related papers  (2025-08-31T21:19:25Z)
• Convergence Rates for Softmax Gating Mixture of Experts [78.3687645289918]
  Mixture of experts (MoE) has emerged as an effective framework to advance the efficiency and scalability of machine learning models.<n>Central to the success of MoE is an adaptive softmax gating mechanism which takes responsibility for determining the relevance of each expert to a given input and then dynamically assigning experts their respective weights.<n>We perform a convergence analysis of parameter estimation and expert estimation under the MoE equipped with the standard softmax gating or its variants, including a dense-to-sparse gating and a hierarchical softmax gating.
  arXiv  Detail & Related papers  (2025-03-05T06:11:24Z)
• Finedeep: Mitigating Sparse Activation in Dense LLMs via Multi-Layer Fine-Grained Experts [82.74439280067492]
  Finedeep is a deep-layered fine-grained expert architecture for dense models.<n>Our framework partitions the feed-forward neural network layers of traditional dense models into small experts.<n>A novel routing mechanism is proposed to determine each expert's contribution.
  arXiv  Detail & Related papers  (2025-02-18T15:09:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.