AIMER: Calibration-Free Task-Agnostic MoE Pruning

• URL: http://arxiv.org/abs/2603.18492v1
• Date: Thu, 19 Mar 2026 04:54:37 GMT
• Title: AIMER: Calibration-Free Task-Agnostic MoE Pruning
• Abstract summary: We introduce AIMER, a simple calibration-free criterion that yields clear within-layer score separation and distinct expert stratification.<n>Across 7B to 30B MoE language models at 25% and 50% pruning ratios over 16 benchmarks, AIMER consistently delivers competitive or stronger overall performance.
• Score: 12.732331870876182
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mixture-of-Experts (MoE) language models increase parameter capacity without proportional per-token compute, but the deployment still requires storing all experts, making expert pruning important for reducing memory and serving overhead. Existing task-agnostic expert pruning methods are typically calibration-dependent: they estimate expert importance from routing or activation statistics on a calibration set, which makes pruning outcomes sensitive to the choice of calibration set and adds substantial preprocessing cost. We introduce AIMER (\textbf{A}bsolute mean over root mean square \textbf{IM}portance for \textbf{E}xpert \textbf{R}anking), a simple calibration-free criterion that yields clear within-layer score separation and distinct expert stratification. Across 7B to 30B MoE language models at 25\% and 50\% pruning ratios over 16 benchmarks, AIMER consistently delivers competitive or stronger overall performance against state-of-the-art calibration-based expert pruning baselines with only 0.22--1.27 seconds for scoring the experts.

Related papers

• When Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts Models [14.959514582161956]
  Expert pruning reduces the memory and serving cost of Mixture-of-Experts (MoE) models.<n>We observe that this assumption breaks down under over-dispersed routing, in which tokens are distributed nearly uniformly across experts.<n>We propose Minimax Expert Score Allocation (MESA), a domain-aware method that iteratively boosts importance scores for experts serving whichever domain is currently worst-affected.
  arXiv  Detail & Related papers  (2026-09-03T20:21:20Z)
• Target-Aware Calibration Data Selection for Preserving Uncertainty in Quantized Language Models [21.86864148430797]
  Quantization is widely used to deploy large language models, but its effect on uncertainty behavior is rarely treated as a primary objective.<n>We frame calibration-data selection for quantization as a target-dependent uncertainty-preservation problem.<n>We introduce Doubt-Preserving Quantization (DPQ), a lightweight pre-quantization recipe family that uses full-precision predictions.
  arXiv  Detail & Related papers  (2026-08-21T12:07:49Z)
• Rethinking Post-Hoc Calibration in Semantic Segmentation [2.8002121919083685]
  Post-hoc calibration offers a practical way to correct confidence estimates without retraining the segmentation model.<n>We study two such issues: first, adding a constant to all logits leaves the softmax probabilities unchanged, but several standard calibrators can still depend on this arbitrary offset.<n>Second, post-hoc calibration is typically fitted by minimizing a likelihood-based objective, whereas segmentation models are trained with task-specific metrics such as Dice.
  arXiv  Detail & Related papers  (2026-07-02T09:01:21Z)
• Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models [16.871402640095745]
  We propose a coverage-aware expert pruning method that uses only generic text corpora for calibration.<n>Our improvements hold with fixed pruning budgets and no downstream calibration data.
  arXiv  Detail & Related papers  (2026-07-02T05:02:18Z)
• Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression [74.00650541246374]
  We propose a structural pruning framework tailored for MoE models.<n>Our approach reduces memory footprint by 5.27$times and consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-06-16T06:53:27Z)
• AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization [104.10572215799239]
  Mixed-precision quantization can substantially reduce the footprint by assigning different bit-widths to different experts.<n>Existing approaches rely on calibration data to estimate expert importance and determine bit allocation.<n>We propose AlphaQ, a calibration-free bit-allocation method for MoE quantization.
  arXiv  Detail & Related papers  (2026-06-03T15:03:18Z)
• Slicing and Dicing: Configuring Optimal Mixtures of Experts [49.5613403644084]
  We present the first systematic study of over 2,000 pretraining runs spanning models up to 6.6B total parameters.<n>We exhaustively vary total experts, expert dimension, heterogeneous expert sizing within a single layer, shared expert size and load-balancing mechanisms.<n>Our results suggest a simpler recipe: focus on expert count and granularity, other choices have minimal effect on final quality.
  arXiv  Detail & Related papers  (2026-05-12T07:47:22Z)
• A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs [64.8510381475827]
  Sparse Mixture-of-Experts (SMoE) architectures are increasingly used to scale large language models efficiently.<n>SMoE models often suffer from severe load imbalance across experts, where a small subset of experts receives most tokens while others are underutilized.<n>We present a systematic analysis of expert routing during inference and identify three findings: (i) load imbalance persists and worsens with larger batch sizes, (ii) selection frequency does not reliably reflect expert importance, and (iii) overall expert workload and importance can be estimated using a small calibration set.
  arXiv  Detail & Related papers  (2026-02-23T15:11:16Z)
• ECO: Quantized Training without Full-Precision Master Weights [58.97082407934466]
  Error-Compensating (ECO) eliminates master weights by applying updates directly to quantized parameters.<n>We show that ECO converges to a constant-radius neighborhood of the optimum, while naive master-weight removal can incur an error that is inversely proportional to the learning rate.
  arXiv  Detail & Related papers  (2026-01-29T18:35:01Z)
• Scalable Utility-Aware Multiclass Calibration [53.28176049547449]
  Utility calibration is a general framework that measures the calibration error relative to a specific utility function.<n>We demonstrate how this framework can unify and re-interpret several existing calibration metrics.
  arXiv  Detail & Related papers  (2025-10-29T12:32:14Z)
• REAP the Experts: Why Pruning Prevails for One-Shot MoE compression [5.517309667435783]
  We show that expert pruning is a superior strategy for generative tasks.<n>We propose Router-weighted Expert Activation Pruning (REAP)<n>Our method achieves near-lossless compression on code generation and tool-calling tasks with Qwen3-Coder-480B and Kimi-K2.
  arXiv  Detail & Related papers  (2025-10-15T18:29:28Z)
• MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE [12.498106165046233]
  Mixture-of-Experts (MoE) enables efficient scaling of large language models by activating only a subset of experts per input token.<n>MoNE replaces redundant experts with lightweight novices to achieve effective and robust model compression.
  arXiv  Detail & Related papers  (2025-07-01T03:02:59Z)
• Adaptive Set-Mass Calibration with Conformal Prediction [60.47079469141295]
  We develop a new calibration procedure that starts with conformal prediction to obtain a set of labels that gives the desired coverage.<n>We then instantiate two simple post-hoc calibrators: a mass normalization and a temperature scaling-based rule, tuned to the conformal constraint.
  arXiv  Detail & Related papers  (2025-05-21T12:18:15Z)
• Unified Sparse Mixture of Experts [14.774596844618396]
  Sparse Mixture of Experts (SMoEs) models scale the capacity of models while maintaining constant computational overhead.<n>This paper proposes a Unified Sparse Mixture of Experts (USMoE) framework that addresses these limitations.
  arXiv  Detail & Related papers  (2025-03-29T07:15:12Z)
• How to Prune Your Language Model: Recovering Accuracy on the "Sparsity May Cry'' Benchmark [60.72725673114168]
  We revisit the question of accurate BERT-pruning during fine-tuning on downstream datasets. We propose a set of general guidelines for successful pruning, even on the challenging SMC benchmark.
  arXiv  Detail & Related papers  (2023-12-21T03:11:30Z)
• Calibration by Distribution Matching: Trainable Kernel Calibration Metrics [56.629245030893685]
  We introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression. These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization. We provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions.
  arXiv  Detail & Related papers  (2023-10-31T06:19:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.