Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs
Abstract Overview
This paper introduces MoE-XBench, a benchmark for evaluating compression in mixture-of-experts large language models as an end-to-end deployment workflow rather than as isolated steps. The benchmark studies 10 MoE models spanning 30B to 235B parameters across standard-attention, hybrid linear-attention, and sliding-window attention families, and evaluates seven workloads. It covers expert pruning, weight quantization, and KV-cache compression both individually and in combination, with an eight-module evaluation suite that measures quality retention, robustness, sensitivity, and deployment efficiency. The study compares accuracy, memory, latency, and hardware efficiency on commodity and accelerator hardware under single-batch deployment settings.
Novelty
The primary contribution is framing MoE model compression as a composable deployment pipeline that benchmarks the joint interactions among expert pruning, weight quantization, and KV-cache compression. The paper also introduces an eight-module normalized scoring framework that systematically separates model quality and sensitivity from real hardware deployment efficiency across multiple MoE architecture families.
Results
The experiments reveal that nominal compression rate does not reliably predict quality loss: expert pruning is the dominant source of degradation, whereas moderate weight quantization preserves average quality more effectively. Furthermore, memory footprint reductions do not guarantee throughput improvements, as long-context KV-cache compression and low-bit dequantization can add substantial decode latency overheads. Average accuracy was also shown to mask notable task-specific regressions in coding and knowledge workloads as well as architecture-dependent sensitivity.
Key Points
- MoE-XBench evaluates pruning, quantization, and KV-cache compression both separately and in combination across 10 MoE models, seven workloads, and multiple hardware platforms.
- The empirical results indicate that expert pruning dominates quality loss, while quantization and KV-cache compression have smaller or more context-dependent effects on model quality.
- Efficiency gains are non-monotonic: memory reduction is portable across hardware, but throughput gains depend heavily on context length, bit-width format, and backend dequantization overheads.