Does a Global Perspective Help Prune Sparse MoEs Elegantly?

• URL: http://arxiv.org/abs/2604.06542v1
• Date: Wed, 08 Apr 2026 00:41:11 GMT
• Title: Does a Global Perspective Help Prune Sparse MoEs Elegantly?
• Abstract summary: Existing pruning methods typically allocate budgets uniformly across layers, overlooking the heterogeneous redundancy that arises in sparse MoEs.<n>We propose GRAPE (Global Redundancy-Aware Pruning of Experts), a global pruning strategy that dynamically allocates pruning budgets based on cross-layer redundancy.<n>Experiments on Mixtral-8x7B, Mixtral-8x22B, DeepSeek-MoE, Qwen-MoE, and GPT-OSS show that, under the same pruning budget, GRAPE consistently achieves the best average performance.
• Score: 21.594202826999958
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Empirical scaling laws for language models have encouraged the development of ever-larger LLMs, despite their growing computational and memory costs. Sparse Mixture-of-Experts (MoEs) offer a promising alternative by activating only a subset of experts per forward pass, improving efficiency without sacrificing performance. However, the large number of expert parameters still leads to substantial memory consumption. Existing pruning methods typically allocate budgets uniformly across layers, overlooking the heterogeneous redundancy that arises in sparse MoEs. We propose GRAPE (Global Redundancy-Aware Pruning of Experts, a global pruning strategy that dynamically allocates pruning budgets based on cross-layer redundancy. Experiments on Mixtral-8x7B, Mixtral-8x22B, DeepSeek-MoE, Qwen-MoE, and GPT-OSS show that, under the same pruning budget, GRAPE consistently achieves the best average performance. On the three main models reported in the paper, it improves average accuracy over the strongest local baseline by 1.40% on average across pruning settings, with gains of up to 2.45%.

Related papers

• FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models [20.61741593160247]
  Mixture-of-Experts (MoE) language models scale model ability with sparsely activated experts.<n>Existing MoE compression methods are still largely fixed-budget, typically optimizing one compressed endpoint at each chosen target budget.<n>We study a different setting: converting a large pretrained MoE LLM into a nested family of deployable budgetworks across budgets.
  arXiv  Detail & Related papers  (2026-06-26T09:08:03Z)
• Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression [74.00650541246374]
  We propose a structural pruning framework tailored for MoE models.<n>Our approach reduces memory footprint by 5.27$times and consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-06-16T06:53:27Z)
• EvoESAP: Non-Uniform Expert Pruning for Sparse MoE [42.738877185877634]
  We present textbfExpected textbfAcceptance textbfProxy (textbfESAP), a speculative-decoding-inspired, teacher-forced metric that measures how well a pruned model matches the full model.<n>We propose EvoESAP, an evolutionary searching framework that optimize a non-uniform layer-wise sparsity allocation under a fixed global budget.
  arXiv  Detail & Related papers  (2026-03-06T08:02:58Z)
• ODAR: Principled Adaptive Routing for LLM Reasoning via Active Inference [60.958331943869126]
  ODAR-Expert is an adaptive routing framework that optimize the accuracy-efficiency trade-off via principled resource allocation.<n>We show strong and consistent gains, including 98.2% accuracy on MATH and 54.8% on Humanity's Last Exam.
  arXiv  Detail & Related papers  (2026-02-27T05:22:01Z)
• MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping [52.02659589971978]
  We propose MoDES, the first training-free framework that adaptively skips experts to enable efficient and accurate MoE MLLM inference.<n>MoDES significantly enhances inference speed, improving the prefilling time by 2.16$times$ and the decoding time by 1.26$times$.
  arXiv  Detail & Related papers  (2025-11-19T18:48:27Z)
• MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE [12.498106165046233]
  Mixture-of-Experts (MoE) enables efficient scaling of large language models by activating only a subset of experts per input token.<n>MoNE replaces redundant experts with lightweight novices to achieve effective and robust model compression.
  arXiv  Detail & Related papers  (2025-07-01T03:02:59Z)
• Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization [16.627496870394456]
  T'yr-the-Pruner is an efficient end-to-end search-based global structural pruning framework.<n>We introduce an effective local pruning and an expectation error accumulation approach to improve supernet construction.<n>Results show that T'yr-the-Pruner achieves state-of-the-art structural pruning, retaining 97% of the dense model's performance.
  arXiv  Detail & Related papers  (2025-03-12T11:52:49Z)
• MoE-I$^2$: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition [32.97035551579975]
  We introduce a two-stage compression method tailored for MoE to reduce the model size and decrease the computational cost. Experiments on Qwen1.5-MoE-A2.7B, DeepSeek-V2-Lite, and Mixtral-8$times$7B demonstrate that our proposed methods can both reduce the model size and enhance inference efficiency.
  arXiv  Detail & Related papers  (2024-11-01T20:37:58Z)
• Merging Experts into One: Improving Computational Efficiency of Mixture of Experts [71.44422347502409]
  A sparse Mixture of Experts (MoE) can reduce the cost by activating a small subset of parameters. Can we retain the advantages of adding more experts without substantially increasing the computational costs? We propose a computation-efficient approach called textbftexttMerging Experts into One (MEO) which reduces the computation cost to that of a single expert.
  arXiv  Detail & Related papers  (2023-10-15T13:28:42Z)
• ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models [70.45441031021291]
  Large Vision-Language Models (LVLMs) can understand the world comprehensively by integrating rich information from different modalities. LVLMs are often problematic due to their massive computational/energy costs and carbon consumption. We propose Efficient Coarse-to-Fine LayerWise Pruning (ECoFLaP), a two-stage coarse-to-fine weight pruning approach for LVLMs.
  arXiv  Detail & Related papers  (2023-10-04T17:34:00Z)
• Advancing Model Pruning via Bi-level Optimization [89.88761425199598]
  iterative magnitude pruning (IMP) is the predominant pruning method to successfully find 'winning tickets' One-shot pruning methods have been developed, but these schemes are usually unable to find winning tickets as good as IMP. We show that the proposed bi-level optimization-oriented pruning method (termed BiP) is a special class of BLO problems with a bi-linear problem structure.
  arXiv  Detail & Related papers  (2022-10-08T19:19:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.