GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs

• URL: http://arxiv.org/abs/2605.23078v1
• Date: Thu, 21 May 2026 22:23:38 GMT
• Title: GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
• Abstract summary: Mixed-precision quantizations mitigate memory overhead by allocating expert-wise bit-widths based on their importance.<n>Existing methods rely on layer-wise importance estimation and overlook router shifts induced by quantization.<n>We propose Global Expert-level Mixed-precision Quantization (GEMQ) to overcome these limitations.
• Score: 40.50894372538921
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mixture-of-Experts Large Language Models (MoE-LLMs) achieve strong performance but incur substantial memory overhead due to massive expert parameters. Mixed-precision quantization mitigates this cost by allocating expert-wise bit-widths based on their importance, approaching the accuracy-memory Pareto frontier and enabling extreme low-bit quantization. However, existing methods rely on layer-wise importance estimation and overlook router shifts induced by quantization, resulting in suboptimal allocation and routing. In this work, we propose Global Expert-level Mixed-precision Quantization (GEMQ) to overcome these limitations via (1) a global linear-programming formulation that captures model-wide expert importance based on quantization error analysis, and (2) efficient router fine-tuning to adapt routing to quantized experts. These components are integrated into a progressive quantization framework that iteratively refines importance estimation and allocation. Experiments demonstrate that GEMQ significantly reduces memory and accelerates inference with minimal accuracy degradation. Source code is available at https://github.com/jndeng/GEMQ .

Related papers

• AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization [104.10572215799239]
  Mixed-precision quantization can substantially reduce the footprint by assigning different bit-widths to different experts.<n>Existing approaches rely on calibration data to estimate expert importance and determine bit allocation.<n>We propose AlphaQ, a calibration-free bit-allocation method for MoE quantization.
  arXiv  Detail & Related papers  (2026-06-03T15:03:18Z)
• WINDQuant: Weight-Informed Neural Decision-Making for Global Mixed-Precision LLM Quantization [40.655670203062805]
  WINDQuant is a reinforcement-learning-based allocation controller for ultra-low-bit LLM quantization.<n>It learns how to assign bit-widths and quantization treatments to fine-grained column chunks under a global storage budget.<n>Experiments on LLaMA models demonstrate that WINDQuant achieves competitive performance in ultra-low-bit settings.
  arXiv  Detail & Related papers  (2026-05-26T07:46:13Z)
• Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees [13.67631667766697]
  Sparse Mixture-of-Experts (MoE) allows scaling of language and vision models efficiently by activating only a small subset of experts per input.<n>MoE incurs substantial memory overhead during inference.<n>We propose a theoretically grounded expert-wise mixed precision strategy that assigns bit-width to each expert based on their change in routers l2 norm during training.
  arXiv  Detail & Related papers  (2026-04-07T23:17:23Z)
• ECO: Quantized Training without Full-Precision Master Weights [58.97082407934466]
  Error-Compensating (ECO) eliminates master weights by applying updates directly to quantized parameters.<n>We show that ECO converges to a constant-radius neighborhood of the optimum, while naive master-weight removal can incur an error that is inversely proportional to the learning rate.
  arXiv  Detail & Related papers  (2026-01-29T18:35:01Z)
• Mixed-Precision Quantization for Language Models: Techniques and Prospects [10.345914140081925]
  Quantization has emerged as an essential compression technique to reduce model size, alleviate memory bottlenecks, and accelerate inference.<n>Mixed-precision quantization offers a promising alternative by selectively allocating precision across layers or within tensors to balance efficiency and accuracy.
  arXiv  Detail & Related papers  (2025-10-19T12:16:40Z)
• End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost [53.25965863436039]
  Quantization-aware training (QAT) provides a more principled solution, but its reliance on backpropagation incurs prohibitive memory costs.<n>We propose ZeroQAT, a zeroth-order optimization-based QAT framework that supports both weight and activation quantization.<n>Experiments show that ZeroQAT consistently outperforms representative PTQ and QAT baselines while requiring significantly less memory.
  arXiv  Detail & Related papers  (2025-08-21T01:18:27Z)
• MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models with Temporal Distillation [74.34220141721231]
  We present MPQ-DMv2, an improved textbfMixed textbfPrecision textbfQuantization framework for extremely low-bit textbfDiffusion textbfModels.
  arXiv  Detail & Related papers  (2025-07-06T08:16:50Z)
• FineQ: Software-Hardware Co-Design for Low-Bit Fine-Grained Mixed-Precision Quantization of LLMs [13.951330786310262]
  FineQ is a software- hardware co-design for low-bit fine-grained mixed-precision quantization of large language models.<n>It partitions the weights into finer-grained clusters and considers the distribution of outliers within these clusters.<n>It achieves higher model accuracy compared to the SOTA mixed-precision quantization algorithm at a close average bit-width.
  arXiv  Detail & Related papers  (2025-04-28T12:47:23Z)
• Channel-Wise Mixed-Precision Quantization for Large Language Models [47.00361921910259]
  Large Language Models (LLMs) have demonstrated remarkable success across a wide range of language tasks.<n>Weight-only quantization presents a promising solution to reduce the memory footprint of LLMs.<n>We introduce Channel-Wise Mixed-Precision Quantization (CMPQ), a novel mixed-precision quantization method.
  arXiv  Detail & Related papers  (2024-10-16T21:34:41Z)
• Compensate Quantization Errors+: Quantized Models Are Inquisitive Learners [51.32182730502002]
  We introduce Singular-value Diagonal Expansion to refine weight distributions to achieve better quantization alignment.<n>Our plug-and-play weight-quantization methods demonstrate substantial performance improvements over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-07-22T09:45:16Z)
• Fully Quantized Image Super-Resolution Networks [81.75002888152159]
  We propose a Fully Quantized image Super-Resolution framework (FQSR) to jointly optimize efficiency and accuracy. We apply our quantization scheme on multiple mainstream super-resolution architectures, including SRResNet, SRGAN and EDSR. Our FQSR using low bits quantization can achieve on par performance compared with the full-precision counterparts on five benchmark datasets.
  arXiv  Detail & Related papers  (2020-11-29T03:53:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.