EvoESAP: Non-Uniform Expert Pruning for Sparse MoE

• URL: http://arxiv.org/abs/2603.06003v1
• Date: Fri, 06 Mar 2026 08:02:58 GMT
• Title: EvoESAP: Non-Uniform Expert Pruning for Sparse MoE
• Abstract summary: We present textbfExpected textbfAcceptance textbfProxy (textbfESAP), a speculative-decoding-inspired, teacher-forced metric that measures how well a pruned model matches the full model.<n>We propose EvoESAP, an evolutionary searching framework that optimize a non-uniform layer-wise sparsity allocation under a fixed global budget.
• Score: 42.738877185877634
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sparse Mixture-of-Experts (SMoE) language models achieve strong capability at low per-token compute, yet deployment remains memory- and throughput-bound because the full expert pool must be stored and served. Post-training expert pruning reduces this cost, but most methods focus on which experts to prune within each layer and default to a uniform layer-wise sparsity allocation, even though the allocation can strongly affect performance. We decouple pruning into within-layer expert ranking and across-layer budget allocation, and introduce \textbf{E}xpected \textbf{S}peculative \textbf{A}cceptance \textbf{P}roxy (\textbf{ESAP}), a speculative-decoding-inspired, teacher-forced metric that measures how well a pruned model matches the full model. ESAP is bounded and stable, enabling cheap comparison of many candidates without costly autoregressive decoding. Building on ESAP, we propose EvoESAP, an evolutionary searching framework that optimizes a non-uniform layer-wise sparsity allocation under a fixed global budget while holding the within-layer pruning order fixed, making it a plug-and-play method with criteria such as Frequency, EAN, SEER, and REAP. Across 7B--30B SMoE LLMs at 25\% and 50\% sparsity, EvoESAP consistently discovers non-uniform allocations that improve open-ended generation (up to \textbf{+19.6\%} on MATH-500 at 50\% sparsity) while preserving competitive multiple-choice accuracy compared with uniform pruning at the same sparsity.

Related papers

• Distribution-Consistent Inference for Dynamic Sparse Mixture-of-Experts [48.294824672909876]
  Inference-time dynamic top-$k$ routing can reduce computation without retraining.<n>We show that reducing the number of activated experts consistently increases the RMS scale and variance of SMoE outputs.<n>We propose Layer-wise Distribution Alignment (LDA), a lightweight inference-time correction that uses layer-wise calibration statistics to align reduced-routing representations with the default configuration.
  arXiv  Detail & Related papers  (2026-09-08T06:23:07Z)
• FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models [20.61741593160247]
  Mixture-of-Experts (MoE) language models scale model ability with sparsely activated experts.<n>Existing MoE compression methods are still largely fixed-budget, typically optimizing one compressed endpoint at each chosen target budget.<n>We study a different setting: converting a large pretrained MoE LLM into a nested family of deployable budgetworks across budgets.
  arXiv  Detail & Related papers  (2026-06-26T09:08:03Z)
• Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression [74.00650541246374]
  We propose a structural pruning framework tailored for MoE models.<n>Our approach reduces memory footprint by 5.27$times and consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-06-16T06:53:27Z)
• SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs [4.035485260002141]
  Sparse Mixture-of-Experts (MoE) models achieve strong quality with low per-token compute, yet their deployment is often limited by the memory wall.<n>We propose textbfSHAPE, a task-driven pruning framework that explicitly models emphintra-layer expert cooperation.
  arXiv  Detail & Related papers  (2026-06-03T08:41:48Z)
• Does a Global Perspective Help Prune Sparse MoEs Elegantly? [21.594202826999958]
  Existing pruning methods typically allocate budgets uniformly across layers, overlooking the heterogeneous redundancy that arises in sparse MoEs.<n>We propose GRAPE (Global Redundancy-Aware Pruning of Experts), a global pruning strategy that dynamically allocates pruning budgets based on cross-layer redundancy.<n>Experiments on Mixtral-8x7B, Mixtral-8x22B, DeepSeek-MoE, Qwen-MoE, and GPT-OSS show that, under the same pruning budget, GRAPE consistently achieves the best average performance.
  arXiv  Detail & Related papers  (2026-04-08T00:41:11Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization [60.87651283510059]
  Group Relative Policy Optimization (GRPO) effectively scales LLM reasoning but incurs prohibitive computational costs.<n>We propose Dynamic Pruning Policy Optimization (DPPO), a framework that enables dynamic pruning while preserving unbiased gradient estimation.<n>To mitigate the data sparsity induced by pruning, we introduce Dense Prompt Packing, a window-based greedy strategy.
  arXiv  Detail & Related papers  (2026-03-04T14:48:53Z)
• UniPruning: Unifying Local Metric and Global Feedback for Scalable Sparse LLMs [46.12497343562301]
  Large Language Models (LLMs) achieve strong performance across diverse tasks but face prohibitive computational and memory costs.<n>We present UniPruning, a unified post-training pruning framework that combines the speed of local saliency metrics with the stability of global coordination.<n>UniPruning consistently delivers competitive or superior perplexity and zero-shot accuracy.
  arXiv  Detail & Related papers  (2025-09-29T13:38:28Z)
• DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning [24.59026258291765]
  Existing MoE pruning methods often lead to suboptimal outcomes and performance degradation due to varying expert redundancy in different MoE layers.<n>We propose a non-uniform pruning strategy, dubbed textbfDiEP, which adaptively adjusts pruning rates at the layer level while jointly learning inter-layer importance.<n>Our method handles exponentially growing non-uniform expert combinations, enabling adaptive gradient-based pruning.
  arXiv  Detail & Related papers  (2025-09-19T15:47:42Z)
• MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on Large Language Models [53.36415620647177]
  Semi-structured sparsity offers a promising solution by strategically retaining $N$ elements out of every $M$ weights.<n>Existing (N:M)-compatible approaches typically fall into two categories: rule-based layerwise greedy search, which suffers from considerable errors, and gradient-driven learning, which incurs prohibitive training costs.<n>We propose a novel linear-space probabilistic framework named MaskPro, which aims to learn a prior categorical distribution for every $M$ consecutive weights and subsequently leverages this distribution to generate the (N:M)-sparsity throughout an $N$-way sampling
  arXiv  Detail & Related papers  (2025-06-15T15:02:59Z)
• Týr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization [16.627496870394456]
  T'yr-the-Pruner is an efficient end-to-end search-based global structural pruning framework.<n>We introduce an effective local pruning and an expectation error accumulation approach to improve supernet construction.<n>Results show that T'yr-the-Pruner achieves state-of-the-art structural pruning, retaining 97% of the dense model's performance.
  arXiv  Detail & Related papers  (2025-03-12T11:52:49Z)
• Towards Efficient Automatic Self-Pruning of Large Language Models [55.90119819642064]
  Post-training structured pruning is a promising solution that prunes Large Language Models without the need for retraining.<n>We argue that the key to mitigating this issue lies in accurately determining the pruning rate for each layer.<n>We introduce $textbfSelf-Pruner$ an end-to-end automatic self-pruning framework for LLMs, which efficiently search layer-wise pruning rates.
  arXiv  Detail & Related papers  (2025-02-20T09:59:50Z)
• Smoothed Normalization for Efficient Distributed Private Optimization [54.197255548244705]
  Federated learning enables machine learning models with privacy of participants.<n>There is no differentially private distributed method for training, non-feedback problems.<n>We introduce a new distributed algorithm $alpha$-$sf NormEC$ with provable convergence guarantees.
  arXiv  Detail & Related papers  (2025-02-19T07:10:32Z)
• [Reproducibility Report] Rigging the Lottery: Making All Tickets Winners [1.6884611234933766]
  $textitRigL$, a sparse training algorithm, claims to directly train sparse networks that match or exceed the performance of existing dense-to-sparse training techniques. We implement $textitRigL$ from scratch in Pytorch and reproduce its performance on CIFAR-10 within 0.1% of the reported value.
  arXiv  Detail & Related papers  (2021-03-29T17:01:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.