FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models

• URL: http://arxiv.org/abs/2606.27866v1
• Date: Fri, 26 Jun 2026 09:08:03 GMT
• Title: FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models
• Abstract summary: Mixture-of-Experts (MoE) language models scale model ability with sparsely activated experts.<n>Existing MoE compression methods are still largely fixed-budget, typically optimizing one compressed endpoint at each chosen target budget.<n>We study a different setting: converting a large pretrained MoE LLM into a nested family of deployable budgetworks across budgets.
• Score: 20.61741593160247
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mixture-of-Experts (MoE) language models scale model ability with sparsely activated experts, making this architecture a standard recipe for modern large models. However, sparse activation does not remove the deployment burden of storing and serving all experts, and the available deployment budget can vary substantially across devices, users, and workloads. Existing MoE compression methods are still largely fixed-budget, typically optimizing one compressed endpoint at each chosen target budget. We study a different setting: converting a large pretrained MoE LLM into a nested family of deployable subnetworks across budgets. Our method first ranks expert FFN channels by their importance, then lets each expert learn a discrete action to prune its channels. By gradually increasing cost pressure, a single action-training run exports a series of action masks from high to low budgets, each of which identifies a reliable smaller subnetwork nested in the ranked base model. Moreover, we use a single recovery fine-tune at a mid pruning budget (40%) to recover degraded model quality and transfer the recovered model to other unseen budgets. Overall, our framework surpasses recent MoE compression baselines. Specifically, on Qwen2-57B-A14B, our method retains ~99.8% of base performance while pruning 50% of routed expert parameters even without fine-tuning. For deployment, our pruned subnetworks deliver real memory reduction and throughput gains, and further support realtime online budget switching with kernel-level co-design.

Related papers

• Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression [74.00650541246374]
  We propose a structural pruning framework tailored for MoE models.<n>Our approach reduces memory footprint by 5.27$times and consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-06-16T06:53:27Z)
• Does a Global Perspective Help Prune Sparse MoEs Elegantly? [21.594202826999958]
  Existing pruning methods typically allocate budgets uniformly across layers, overlooking the heterogeneous redundancy that arises in sparse MoEs.<n>We propose GRAPE (Global Redundancy-Aware Pruning of Experts), a global pruning strategy that dynamically allocates pruning budgets based on cross-layer redundancy.<n>Experiments on Mixtral-8x7B, Mixtral-8x22B, DeepSeek-MoE, Qwen-MoE, and GPT-OSS show that, under the same pruning budget, GRAPE consistently achieves the best average performance.
  arXiv  Detail & Related papers  (2026-04-08T00:41:11Z)
• EvoESAP: Non-Uniform Expert Pruning for Sparse MoE [42.738877185877634]
  We present textbfExpected textbfAcceptance textbfProxy (textbfESAP), a speculative-decoding-inspired, teacher-forced metric that measures how well a pruned model matches the full model.<n>We propose EvoESAP, an evolutionary searching framework that optimize a non-uniform layer-wise sparsity allocation under a fixed global budget.
  arXiv  Detail & Related papers  (2026-03-06T08:02:58Z)
• FlexRank: Nested Low-Rank Knowledge Decomposition for Adaptive Model Deployment [20.331469310989956]
  We argue that importance-ordered nested components can be extracted from pretrained models, and selectively activated on the available computational budget.<n>Our approach enables a "train-once, deploy-everywhere" paradigm that offers a graceful trade-off between cost and performance without training from scratch for each budget.
  arXiv  Detail & Related papers  (2026-02-02T19:01:40Z)
• Don't Start Over: A Cost-Effective Framework for Migrating Personalized Prompts Between LLMs [51.79252689855809]
  Personalization in Large Language Models (LLMs) often relies on user-specific soft prompts.<n>We propose the Prompt-level User Migration Adapter (PUMA), a framework to efficiently migrate personalized prompts across incompatible models.<n>Experiments on three large-scale datasets show our method matches or even surpasses the performance of retraining from scratch, reducing computational cost by up to 98%.
  arXiv  Detail & Related papers  (2026-01-17T12:30:31Z)
• Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs [80.72350166388601]
  Nemotron Elastic is a framework for building reasoning-oriented LLMs.<n>It embeds nested submodels within a single parent model.<n>Each of these submodels shares weights with the parent model and can be extracted zero-shot during deployment.
  arXiv  Detail & Related papers  (2025-11-20T18:59:21Z)
• Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization [60.309915093470416]
  Matryoshka MoE (M-MoE) is a training framework that instills a coarse-to-fine structure directly into the expert ensemble.<n>Our work paves the way for more practical and adaptable deployments of large-scale MoE models.
  arXiv  Detail & Related papers  (2025-09-30T16:56:44Z)
• Pruning General Large Language Models into Customized Expert Models [78.96891010334852]
  Large language models (LLMs) have revolutionized natural language processing, yet their substantial model sizes often require substantial computational resources.<n>We propose a method to prune a large general model into a smaller lightweight expert model.<n>By identifying and pruning irrelevant neurons of each dimension, $textttCus-Prun$ creates expert models without any post-training.
  arXiv  Detail & Related papers  (2025-06-03T07:47:30Z)
• D$^{2}$MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving [14.607254882119507]
  Combination of experts (MoE) model is a sparse variant of large language models (LLMs)<n>Despite its benefits, MoE is still too expensive to deploy on resource-constrained edge devices.<n>We propose D$2$MoE, an algorithm-system co-design framework that matches diverse task requirements by dynamically allocating the most proper bit-width to each expert.
  arXiv  Detail & Related papers  (2025-04-17T05:37:35Z)
• Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design [59.00758127310582]
  We propose a novel framework Read-ME that transforms pre-trained dense LLMs into smaller MoE models. Our approach employs activation sparsity to extract experts. Read-ME outperforms other popular open-source dense models of similar scales.
  arXiv  Detail & Related papers  (2024-10-24T19:48:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.