Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees

• URL: http://arxiv.org/abs/2604.06515v1
• Date: Tue, 07 Apr 2026 23:17:23 GMT
• Title: Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees
• Abstract summary: Sparse Mixture-of-Experts (MoE) allows scaling of language and vision models efficiently by activating only a small subset of experts per input.<n>MoE incurs substantial memory overhead during inference.<n>We propose a theoretically grounded expert-wise mixed precision strategy that assigns bit-width to each expert based on their change in routers l2 norm during training.
• Score: 13.67631667766697
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sparse Mixture-of-Experts (MoE) allows scaling of language and vision models efficiently by activating only a small subset of experts per input. While this reduces computation, the large number of parameters still incurs substantial memory overhead during inference. Post-training quantization has been explored to address this issue. Because uniform quantization suffers from significant accuracy loss at low bit-widths, mixed-precision methods have been recently explored; however, they often require substantial computation for bit-width allocation and overlook the varying sensitivity of model performance to the quantization of different experts. We propose a theoretically grounded expert-wise mixed precision strategy that assigns bit-width to each expert primarily based on their change in routers l2 norm during training. Experts with smaller changes are shown to capture less frequent but critical features, and model performance is more sensitive to the quantization of these experts, thus requiring higher precision. Furthermore, to avoid allocating experts to lower precision that inject high quantization noise, experts with large maximum intra-neuron variance are also allocated higher precision. Experiments on large-scale MoE models, including Switch Transformer and Mixtral, show that our method achieves higher accuracy than existing approaches, while also reducing inference cost and incurring only negligible overhead for bit-width assignment.

Related papers

• AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization [104.10572215799239]
  Mixed-precision quantization can substantially reduce the footprint by assigning different bit-widths to different experts.<n>Existing approaches rely on calibration data to estimate expert importance and determine bit allocation.<n>We propose AlphaQ, a calibration-free bit-allocation method for MoE quantization.
  arXiv  Detail & Related papers  (2026-06-03T15:03:18Z)
• TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts [57.78282303757474]
  We propose TENP, a structured Trapezoidal ExpertNeuron Pruning framework.<n>We identify and retain important experts, while applying expert neuron pruning (ENP) to less important experts.<n>For ENP, we measure each neuron's projected contribution to the expert output to identify and retain important neurons.
  arXiv  Detail & Related papers  (2026-06-03T07:49:31Z)
• GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs [40.50894372538921]
  Mixed-precision quantizations mitigate memory overhead by allocating expert-wise bit-widths based on their importance.<n>Existing methods rely on layer-wise importance estimation and overlook router shifts induced by quantization.<n>We propose Global Expert-level Mixed-precision Quantization (GEMQ) to overcome these limitations.
  arXiv  Detail & Related papers  (2026-05-21T22:23:38Z)
• TileQ: Efficient Low-Rank Quantization of Mixture-of-Experts with 2D Tiling [3.4199566861263135]
  Mixture-of-Experts (MoE) models achieve remarkable performance by sparsely activating specialized experts.<n>Low-rank quantization offers a promising route to compress MoE models, but existing methods still incur nonnegligible memory overhead and inference latency.<n>We propose textscTileQ, a fine-tuning-free post-training quantization to share low-rank factors across both input and output dimensions of MoE experts.
  arXiv  Detail & Related papers  (2026-05-10T03:10:20Z)
• Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization [3.6899131505284455]
  Post-Training Quantization (PTQ) has emerged as an effective technique for alleviating the substantial computational and memory overheads of Vision-Language Models (VLMs)<n>We propose textbfQuant Experts (QE), a token-aware adaptive error compensation with mixture-of-experts for VLMs quantization.
  arXiv  Detail & Related papers  (2026-02-27T14:47:48Z)
• A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs [64.8510381475827]
  Sparse Mixture-of-Experts (SMoE) architectures are increasingly used to scale large language models efficiently.<n>SMoE models often suffer from severe load imbalance across experts, where a small subset of experts receives most tokens while others are underutilized.<n>We present a systematic analysis of expert routing during inference and identify three findings: (i) load imbalance persists and worsens with larger batch sizes, (ii) selection frequency does not reliably reflect expert importance, and (iii) overall expert workload and importance can be estimated using a small calibration set.
  arXiv  Detail & Related papers  (2026-02-23T15:11:16Z)
• MoPEQ: Mixture of Mixed Precision Quantized Experts [0.1262792599323502]
  Mixed Precision Quantization assigns different precisions to different layers of an LLM/VLM based on layer sensitivity and importance within the model.<n>We propose a Post Training Quantization algorithm, MoPEQ, that assigns optimal bit width to each expert.<n>Our method balances accuracy and model size by analyzing each expert's sensitivity using Hessian trace approximation.
  arXiv  Detail & Related papers  (2025-09-02T17:04:59Z)
• Progressive Element-wise Gradient Estimation for Neural Network Quantization [2.1413624861650358]
  Quantization-Aware Training (QAT) methods rely on the Straight-Through Estimator (STE) to address the non-differentiability of discretization functions.<n>We propose Progressive Element-wise Gradient Estimation (PEGE) to address discretization errors between continuous and quantized values.<n>PEGE consistently outperforms existing backpropagation methods and enables low-precision models to match or even outperform the accuracy of their full-precision counterparts.
  arXiv  Detail & Related papers  (2025-08-27T15:59:36Z)
• MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance [10.817003682434425]
  Mixture-of-Experts (MoE) large language models (LLMs) leverage dynamic routing and sparse activation to enhance efficiency and scalability.<n>Post-training quantization (PTQ) encounters severe accuracy degradation and diminished performance when applied to MoE models.<n>This paper investigates the impact of MoE's sparse and dynamic characteristics on quantization.
  arXiv  Detail & Related papers  (2025-05-02T08:51:55Z)
• Quantize What Counts: More for Keys, Less for Values [63.51476878610841]
  Large Language Models (LLMs) suffer inference-time memory bottlenecks dominated by the attention Key-Value ( KV) cache.<n>This paper proposes two theorems that anchor mixed-precision KV quantization in the intrinsic geometry of Transformer models.
  arXiv  Detail & Related papers  (2025-02-20T22:24:27Z)
• Compensate Quantization Errors+: Quantized Models Are Inquisitive Learners [51.32182730502002]
  We introduce Singular-value Diagonal Expansion to refine weight distributions to achieve better quantization alignment.<n>Our plug-and-play weight-quantization methods demonstrate substantial performance improvements over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-07-22T09:45:16Z)
• On-Chip Hardware-Aware Quantization for Mixed Precision Neural Networks [52.97107229149988]
  We propose an On-Chip Hardware-Aware Quantization framework, performing hardware-aware mixed-precision quantization on deployed edge devices. For efficiency metrics, we built an On-Chip Quantization Aware pipeline, which allows the quantization process to perceive the actual hardware efficiency of the quantization operator. For accuracy metrics, we propose Mask-Guided Quantization Estimation technology to effectively estimate the accuracy impact of operators in the on-chip scenario.
  arXiv  Detail & Related papers  (2023-09-05T04:39:34Z)
• AMED: Automatic Mixed-Precision Quantization for Edge Devices [3.5223695602582614]
  Quantized neural networks are well known for reducing the latency, power consumption, and model size without significant harm to the performance. Mixed-precision quantization offers better utilization of customized hardware that supports arithmetic operations at different bitwidths.
  arXiv  Detail & Related papers  (2022-05-30T21:23:22Z)
• Mixed Precision of Quantization of Transformer Language Models for Speech Recognition [67.95996816744251]
  State-of-the-art neural language models represented by Transformers are becoming increasingly complex and expensive for practical applications. Current low-bit quantization methods are based on uniform precision and fail to account for the varying performance sensitivity at different parts of the system to quantization errors. The optimal local precision settings are automatically learned using two techniques. Experiments conducted on Penn Treebank (PTB) and a Switchboard corpus trained LF-MMI TDNN system.
  arXiv  Detail & Related papers  (2021-11-29T09:57:00Z)
• DAQ: Distribution-Aware Quantization for Deep Image Super-Resolution Networks [49.191062785007006]
  Quantizing deep convolutional neural networks for image super-resolution substantially reduces their computational costs. Existing works either suffer from a severe performance drop in ultra-low precision of 4 or lower bit-widths, or require a heavy fine-tuning process to recover the performance. We propose a novel distribution-aware quantization scheme (DAQ) which facilitates accurate training-free quantization in ultra-low precision.
  arXiv  Detail & Related papers  (2020-12-21T10:19:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.