ConMoE: Expert-Pool Consolidation via Prototype Reassignment for MoE Compression

• URL: http://arxiv.org/abs/2605.29350v1
• Date: Thu, 28 May 2026 04:44:22 GMT
• Title: ConMoE: Expert-Pool Consolidation via Prototype Reassignment for MoE Compression
• Abstract summary: Mixture-of-Experts (MoE) language models reduce per-token computation but still require storing and serving all experts.<n>We formulate post-training MoE compression as expert-pool consolidation.<n>We propose ConMoE, a train-free prototype remapping framework that selects retained experts.
• Score: 4.740789393042061
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mixture-of-Experts (MoE) language models reduce per-token computation but still require storing and serving all experts, making deployment memory-intensive. Existing post-training compression methods mainly shrink this cost by pruning experts or merging their weights. We formulate post-training MoE compression as expert-pool consolidation: retaining a smaller set of pretrained experts as reusable prototypes and deterministically remapping each original expert reference to one selected prototype. This view separates the reduced expert pool from the reuse structure that represents the original expert slots, and allows prototype sharing within local layer scopes while preserving the original router interface. We propose ConMoE, a train-free prototype remapping framework that selects retained experts using calibration-based contribution and replaceability signals, then redirects original expert calls to the selected prototypes without weight updates or post-compression fine-tuning. Experiments on three pretrained MoE language models show that ConMoE matches or outperforms strong pruning and merging baselines in several settings, achieving the best average score on deepseek-moe-16b-base at both 25% and 50% routed-expert reduction, while remaining competitive on Qwen3-30B-A3B and OLMoE-1B-7B-0125. Ablations indicate that deterministic reassignment is the most stable component, whereas broader cross-layer sharing and post-hoc weight fusion are model-dependent.

Related papers

• Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs [50.677556944160216]
  MoEXBench is a systematic benchmark for evaluating composable MoE compression as an end-to-end deployment workflow.<n>It measures composable-compression quality, workload and architecture, pruning/quantization/KV cache sensitivity, and deployment efficiency on commodity hardware.
  arXiv  Detail & Related papers  (2026-08-22T00:01:18Z)
• Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation [49.8459545291965]
  Mixture-of-experts (MoE) models have recently moved beyond routing a fixed number of complete experts.<n>We study this dependency by decomposing sparsely upcycled feed-forward experts into key-value channels.<n>We instantiate it in UniF-MoE, a unified framework for token-adaptive MoE computation.
  arXiv  Detail & Related papers  (2026-08-11T02:40:58Z)
• FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models [20.61741593160247]
  Mixture-of-Experts (MoE) language models scale model ability with sparsely activated experts.<n>Existing MoE compression methods are still largely fixed-budget, typically optimizing one compressed endpoint at each chosen target budget.<n>We study a different setting: converting a large pretrained MoE LLM into a nested family of deployable budgetworks across budgets.
  arXiv  Detail & Related papers  (2026-06-26T09:08:03Z)
• Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression [74.00650541246374]
  We propose a structural pruning framework tailored for MoE models.<n>Our approach reduces memory footprint by 5.27$times and consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-06-16T06:53:27Z)
• SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs [4.035485260002141]
  Sparse Mixture-of-Experts (MoE) models achieve strong quality with low per-token compute, yet their deployment is often limited by the memory wall.<n>We propose textbfSHAPE, a task-driven pruning framework that explicitly models emphintra-layer expert cooperation.
  arXiv  Detail & Related papers  (2026-06-03T08:41:48Z)
• DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts [56.175408382091796]
  We show that replacing the standard weighted-summation aggregation with structural aggregation expands the expert-combination space without altering the experts or router.<n>We propose DAG-MoE, a sparse MoE framework that employs a lightweight module to automatically learn the optimal aggregation structure among the selected experts.
  arXiv  Detail & Related papers  (2026-05-31T07:08:16Z)
• Pruning and Distilling Mixture-of-Experts into Dense Language Models [13.03679207895992]
  Mixture-of-Experts (MoE) is now the dominant architecture for frontier language models.<n>We present the first systematic framework for converting a trained MoE into a standard fully dense architecture.<n>We evaluate 7 scoring, 5 grouping, and 2 magnitude scaling methods across a range of selected expert counts on Qwen3-30B-A3B, yielding 350 configurations.
  arXiv  Detail & Related papers  (2026-05-27T09:27:36Z)
• Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts [27.40662720929157]
  Mixture-of-Experts (MoE) has become the dominant architecture for scaling large language models.<n>We propose expert upcycling, a method for progressively expanding MoE capacity by increasing the number of experts.<n>In our 7B-13B total parameter experiments, the upcycled model matches the fixed-size baseline on validation loss while saving 32% of GPU hours.
  arXiv  Detail & Related papers  (2026-04-21T05:53:33Z)
• Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcycling [64.1662577506775]
  Cluster-aware Upcycling breaks expert symmetry and encourages early specialization aligned with the data distribution.<n>When evaluated on CLIP ViT-B/32 and ViT-B/16, Cluster-aware Upcycling consistently outperforms existing methods.
  arXiv  Detail & Related papers  (2026-04-15T05:52:25Z)
• REAM: Merging Improves Pruning of Experts in LLMs [6.52958560748865]
  Traditional approaches to reduce memory requirements include weight pruning and quantization.<n>We propose a novel method, Router-weighted Expert Activation Merging (REAM)<n>Instead of removing experts, REAM groups them and merges their weights, better preserving original performance.
  arXiv  Detail & Related papers  (2026-04-06T02:08:09Z)
• PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference [17.441141633991197]
  We introduce PuzzleMoE, a training-free MoE compression method that achieves high accuracy and efficient inference through two key innovations.<n>First, PuzzleMoE performs sparse expert merging by identifying element-wise weight redundancy and specialization.<n>Second, to avoid the overhead of storing binary masks and signs, PuzzleMoE introduces a bit-packed encoding scheme that reuses underutilized exponent bits.
  arXiv  Detail & Related papers  (2025-11-06T20:53:02Z)
• MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE [12.498106165046233]
  Mixture-of-Experts (MoE) enables efficient scaling of large language models by activating only a subset of experts per input token.<n>MoNE replaces redundant experts with lightweight novices to achieve effective and robust model compression.
  arXiv  Detail & Related papers  (2025-07-01T03:02:59Z)
• ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restoration [61.579842548990754]
  Mixture-of-Experts (MoE) Transformer, the backbone of multiple phenomenal language models, leverages sparsity by activating only a fraction of model parameters for each input token.<n>We introduce ResMoE, an innovative MoE approximation framework that utilizes Wasserstein barycenter to extract a common expert (barycenter expert) and approximate the residuals between this barycenter expert and the original ones.
  arXiv  Detail & Related papers  (2025-03-10T03:15:54Z)
• A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts [49.394145046409044]
  This paper provides the first provably efficient technique for pruning experts in finetuned MoE models. We theoretically prove that prioritizing the pruning of the experts with a smaller change of the routers l2 norm from the pretrained model guarantees the preservation of test accuracy. Although our theoretical analysis is centered on binary classification tasks on simplified MoE architecture, our expert pruning method is verified on large vision MoE models.
  arXiv  Detail & Related papers  (2024-05-26T17:52:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.