MoBiE: Efficient Inference of Mixture of Binary Experts under Post-Training Quantization

• URL: http://arxiv.org/abs/2604.06798v3
• Date: Tue, 14 Apr 2026 02:11:06 GMT
• Title: MoBiE: Efficient Inference of Mixture of Binary Experts under Post-Training Quantization
• Abstract summary: MoBiE is a binarization framework tailored for Mixture-of-Experts (MoE) based large language models (LLMs)<n>MoBiE consistently outperforms state-of-the-art binary methods across multiple MoE-based LLMs and benchmarks.
• Score: 11.19613037505662
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mixture-of-Experts (MoE) based large language models (LLMs) offer strong performance but suffer from high memory and computation costs. Weight binarization provides extreme efficiency, yet existing binary methods designed for dense LLMs struggle with MoE-specific issues, including cross-expert redundancy, task-agnostic importance estimation, and quantization-induced routing shifts. To this end, we propose MoBiE, the first binarization framework tailored for MoE-based LLMs. MoBiE is built on three core innovations: 1. using joint SVD decomposition to reduce cross-expert redundancy; 2. integrating global loss gradients into local Hessian metrics to enhance weight importance estimation; 3. introducing an error constraint guided by the input null space to mitigate routing distortion. Notably, MoBiE achieves these optimizations while incurring no additional storage overhead, striking a balance between efficiency and model performance. Extensive experiments demonstrate that MoBiE consistently outperforms state-of-the-art binary methods across multiple MoE-based LLMs and benchmarks. For example, on Qwen3-30B-A3B, MoBiE reduces perplexity by 52.2$\%$, improves average zero-shot performance by 43.4$\%$, achieves over 2 $\times$ inference speedup, and further shortens quantization time. The code is available at https://github.com/Kishon-zzx/MoBiE.

Related papers

• LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models [129.64558694024126]
  Diffusion language models (dLLMs) offer an alternative to autoregressive (AR) language modeling.<n>We characterize how optimization hyper parameters, compute allocation, and architecture scale for MoE dLLMs.<n>We train LLaDA MoE v2, a 30B-A3B dLLM, from scratch on 23.5T tokens.
  arXiv  Detail & Related papers  (2026-08-04T10:53:02Z)
• BitsMoE: Efficient Spectral Energy-Guided Bit Allocation for MoE LLM Quantization [5.878850231726241]
  Mixture-of-Experts (MoE) large language models reduce per-token computation through sparse expert activation.<n>Existing MoE compression methods struggle in the ultra-low-bit regime.<n>We propose BitsMoE, a spectral-energy-guided bit-allocation framework for MoE LLM quantization.
  arXiv  Detail & Related papers  (2026-05-22T13:05:53Z)
• BEAM: Binary Expert Activation Masking for Dynamic Routing in MoE [18.898045833395095]
  Mixture-of-Experts (MoE) architectures activate only a subset of experts per token.<n>Existing acceleration methods require costly retraining with architectural changes or suffer from severe performance drop at high sparsity.<n>We propose BEAM, a novel method that learns token-adaptive expert selection via trainable binary masks.
  arXiv  Detail & Related papers  (2026-05-14T06:33:41Z)
• ELMoE-3D: Leveraging Intrinsic Elasticity of MoE for Hybrid-Bonding-Enabled Self-Speculative Decoding in On-Premises Serving [3.8457393423256363]
  We propose ELMoE-3D, a hybrid-bonding framework that unifies cache-based acceleration and speculative decoding.<n>On our 3D-stacked hardware, ELMoE-3D achieves an average $6.6times$ speedup and $4.4times$ energy efficiency gain over naive MoE serving on xPU 1-16.
  arXiv  Detail & Related papers  (2026-04-16T05:12:51Z)
• MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping [52.02659589971978]
  We propose MoDES, the first training-free framework that adaptively skips experts to enable efficient and accurate MoE MLLM inference.<n>MoDES significantly enhances inference speed, improving the prefilling time by 2.16$times$ and the decoding time by 1.26$times$.
  arXiv  Detail & Related papers  (2025-11-19T18:48:27Z)
• PT$^2$-LLM: Post-Training Ternarization for Large Language Models [52.4629647715623]
  Large Language Models (LLMs) have shown impressive capabilities across diverse tasks, but their large memory and compute demands hinder deployment.<n>We propose PT$2$-LLM, a post-training ternarization framework tailored for LLMs.<n>At its core is an Asymmetric Ternary Quantizer equipped with a two-stage refinement pipeline.
  arXiv  Detail & Related papers  (2025-09-27T03:01:48Z)
• MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE [12.498106165046233]
  Mixture-of-Experts (MoE) enables efficient scaling of large language models by activating only a subset of experts per input token.<n>MoNE replaces redundant experts with lightweight novices to achieve effective and robust model compression.
  arXiv  Detail & Related papers  (2025-07-01T03:02:59Z)
• HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference [54.40808356999408]
  We present HOBBIT, a mixed precision expert offloading system to enable flexible and efficient MoE inference. Our key insight is that dynamically replacing less critical cache-miss experts with low precision versions can substantially reduce expert-loading latency. HOBBIT achieves up to a 9.93x speedup in decoding compared to state-of-the-art MoE offloading systems.
  arXiv  Detail & Related papers  (2024-11-03T04:25:46Z)
• DB-LLM: Accurate Dual-Binarization for Efficient LLMs [83.70686728471547]
  Large language models (LLMs) have significantly advanced the field of natural language processing. Existing ultra-low-bit quantization always causes severe accuracy drops. We propose a novel Dual-Binarization method for LLMs, namely DB-LLM.
  arXiv  Detail & Related papers  (2024-02-19T09:04:30Z)
• SqueezeLLM: Dense-and-Sparse Quantization [80.32162537942138]
  Main bottleneck for generative inference with LLMs is memory bandwidth, rather than compute, for single batch inference. We introduce SqueezeLLM, a post-training quantization framework that enables lossless compression to ultra-low precisions of up to 3-bit. Our framework incorporates two novel ideas: (i) sensitivity-based non-uniform quantization, which searches for the optimal bit precision assignment based on second-order information; and (ii) the Dense-and-Sparse decomposition that stores outliers and sensitive weight values in an efficient sparse format.
  arXiv  Detail & Related papers  (2023-06-13T08:57:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.