Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression

• URL: http://arxiv.org/abs/2606.18304v1
• Date: Tue, 16 Jun 2026 06:53:27 GMT
• Title: Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression
• Abstract summary: We propose a structural pruning framework tailored for MoE models.<n>Our approach reduces memory footprint by 5.27$times and consistently outperforms state-of-the-art baselines.
• Score: 74.00650541246374
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mixture-of-Experts (MoE) models scale compute efficiently, yet remain expensive to deploy due to their substantial memory footprint and inference overhead. Prior compression methods mainly operate at the expert level, either removing entire experts or ranking experts by coarse-grained importance scores. However, such expert-wise decisions are often too coarse to capture fine-grained redundancy, leading to misallocated pruning budgets and limited compression. To address this problem, we observe that information within MoE experts is highly concentrated in a small subset of channels, leaving substantial redundancy even in experts deemed important. Based on this observation, we propose a structural pruning framework tailored for MoE models. Our method reformulates prune-ratio allocation as a channel-score coverage maximization problem and solves it efficiently using an attribution-based approximation. Experiments on DeepSeek and Qwen MoE models show that our method preserves model accuracy under 50% or 25% structured pruning when combined with 4-bit quantization. On Qwen3-30B-A3B, our approach reduces memory footprint by 5.27$\times$ and consistently outperforms state-of-the-art baselines across diverse benchmarks.

Related papers

• Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models [16.871402640095745]
  We propose a coverage-aware expert pruning method that uses only generic text corpora for calibration.<n>Our improvements hold with fixed pruning budgets and no downstream calibration data.
  arXiv  Detail & Related papers  (2026-07-02T05:02:18Z)
• FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models [20.61741593160247]
  Mixture-of-Experts (MoE) language models scale model ability with sparsely activated experts.<n>Existing MoE compression methods are still largely fixed-budget, typically optimizing one compressed endpoint at each chosen target budget.<n>We study a different setting: converting a large pretrained MoE LLM into a nested family of deployable budgetworks across budgets.
  arXiv  Detail & Related papers  (2026-06-26T09:08:03Z)
• TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts [57.78282303757474]
  We propose TENP, a structured Trapezoidal ExpertNeuron Pruning framework.<n>We identify and retain important experts, while applying expert neuron pruning (ENP) to less important experts.<n>For ENP, we measure each neuron's projected contribution to the expert output to identify and retain important neurons.
  arXiv  Detail & Related papers  (2026-06-03T07:49:31Z)
• Does a Global Perspective Help Prune Sparse MoEs Elegantly? [21.594202826999958]
  Existing pruning methods typically allocate budgets uniformly across layers, overlooking the heterogeneous redundancy that arises in sparse MoEs.<n>We propose GRAPE (Global Redundancy-Aware Pruning of Experts), a global pruning strategy that dynamically allocates pruning budgets based on cross-layer redundancy.<n>Experiments on Mixtral-8x7B, Mixtral-8x22B, DeepSeek-MoE, Qwen-MoE, and GPT-OSS show that, under the same pruning budget, GRAPE consistently achieves the best average performance.
  arXiv  Detail & Related papers  (2026-04-08T00:41:11Z)
• FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach [58.345210583013454]
  FineRMoE is an architecture that extends fine-grained expert design to both intermediate and output dimensions.<n>It achieves 6 times higher parameter efficiency, 281 times lower prefill latency, and 136 timese higher decoding throughput during inference.
  arXiv  Detail & Related papers  (2026-03-09T12:00:36Z)
• REAP the Experts: Why Pruning Prevails for One-Shot MoE compression [5.517309667435783]
  We show that expert pruning is a superior strategy for generative tasks.<n>We propose Router-weighted Expert Activation Pruning (REAP)<n>Our method achieves near-lossless compression on code generation and tool-calling tasks with Qwen3-Coder-480B and Kimi-K2.
  arXiv  Detail & Related papers  (2025-10-15T18:29:28Z)
• CAMERA: Multi-Matrix Joint Compression for MoE Models via Micro-Expert Redundancy Analysis [55.099051077903766]
  Large Language Models with Mixture-of-Experts (MoE) suffer from substantial computational and storage overheads.<n>We introduce micro-expert as a finer-grained compression unit that spans across matrices.<n>We propose CAMERA-P, a structured micro-expert pruning framework, and CAMERA-Q, a mixed-precision quantization idea designed for micro-experts.
  arXiv  Detail & Related papers  (2025-08-04T11:42:48Z)
• MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE [12.498106165046233]
  Mixture-of-Experts (MoE) enables efficient scaling of large language models by activating only a subset of experts per input token.<n>MoNE replaces redundant experts with lightweight novices to achieve effective and robust model compression.
  arXiv  Detail & Related papers  (2025-07-01T03:02:59Z)
• HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference [54.40808356999408]
  We present HOBBIT, a mixed precision expert offloading system to enable flexible and efficient MoE inference. Our key insight is that dynamically replacing less critical cache-miss experts with low precision versions can substantially reduce expert-loading latency. HOBBIT achieves up to a 9.93x speedup in decoding compared to state-of-the-art MoE offloading systems.
  arXiv  Detail & Related papers  (2024-11-03T04:25:46Z)
• A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts [49.394145046409044]
  This paper provides the first provably efficient technique for pruning experts in finetuned MoE models. We theoretically prove that prioritizing the pruning of the experts with a smaller change of the routers l2 norm from the pretrained model guarantees the preservation of test accuracy. Although our theoretical analysis is centered on binary classification tasks on simplified MoE architecture, our expert pruning method is verified on large vision MoE models.
  arXiv  Detail & Related papers  (2024-05-26T17:52:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.