Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models

• URL: http://arxiv.org/abs/2502.12947v1
• Date: Tue, 18 Feb 2025 15:30:34 GMT
• Title: Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models
• Abstract summary: We show that non-activated experts in MoE models possess valuable knowledge that benefits student models.<n>We propose two intuitive MoE-specific KD methods for the first time: Knowledge Augmentation (KA) and Student-Aware Router (SAR)
• Score: 30.70974942397732
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With the emergence of Mixture-of-Experts (MoE), the efficient scaling of model size has accelerated the development of large language models in recent years. However, their high memory requirements prevent their use in resource-constrained environments. While knowledge distillation (KD) has been a proven method for model compression, its application to MoE teacher models remains underexplored. Through our investigation, we discover that non-activated experts in MoE models possess valuable knowledge that benefits student models. We further demonstrate that existing KD methods are not optimal for compressing MoE models, as they fail to leverage this knowledge effectively. To address this, we propose two intuitive MoE-specific KD methods for the first time: Knowledge Augmentation (KA) and Student-Aware Router (SAR), both designed to effectively extract knowledge from all experts. Specifically, KA augments knowledge by sampling experts multiple times, while SAR uses all experts and adjusts the expert weights through router training to provide optimal knowledge. Extensive experiments show that our methods outperform conventional KD methods, demonstrating their effectiveness for MoE teacher models.

Related papers

• MoEKD: Mixture-of-Experts Knowledge Distillation for Robust and High-Performing Compressed Code Models [6.25009626782699]
  Large language models for code have achieved strong performance across diverse software analytics tasks.<n> Knowledge distillation (KD) offers a practical solution by transferring knowledge from a large model to a smaller and more efficient model.<n>MoEKD decomposes the distillation process into expert and router training, aggregation of expert knowledge through a learned routing mechanism, and distillation from the aggregated knowledge.
  arXiv  Detail & Related papers  (2026-03-13T17:49:01Z)
• Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization [60.309915093470416]
  Matryoshka MoE (M-MoE) is a training framework that instills a coarse-to-fine structure directly into the expert ensemble.<n>Our work paves the way for more practical and adaptable deployments of large-scale MoE models.
  arXiv  Detail & Related papers  (2025-09-30T16:56:44Z)
• An Empirical Study of Knowledge Distillation for Code Understanding Tasks [19.64130505527951]
  Knowledge distillation (KD) addresses limitations by transferring knowledge from large teacher models to compact student models.<n>This paper systematically investigates the effectiveness and usage of KD in code understanding tasks.
  arXiv  Detail & Related papers  (2025-08-21T10:24:48Z)
• Personalized Exercise Recommendation with Semantically-Grounded Knowledge Tracing [54.44838681588145]
  ExRec is a framework for personalized exercise recommendation with semantically-grounded knowledge tracing.<n>We show that ExRec generalizes robustly to new, unseen questions and that it produces interpretable student learning trajectories.
  arXiv  Detail & Related papers  (2025-07-15T07:54:04Z)
• Does Knowledge Distillation Matter for Large Language Model based Bundle Generation? [13.491190612749534]
  Knowledge distillation offers a promising solution, transferring expertise from large teacher models to compact student models. This study systematically investigates knowledge distillation approaches for bundle generation, aiming to minimize computational demands while preserving performance. We propose a comprehensive KD framework that (i) progressively extracts knowledge (patterns, rules, deep thoughts); (ii) captures varying quantities of distilled knowledge through different strategies; and (iii) exploits complementary LLM adaptation techniques for domain-specific adaptation and enhanced efficiency.
  arXiv  Detail & Related papers  (2025-04-24T03:18:16Z)
• CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Experts [36.385301311200905]
  Mixture-of-Experts (MoE) models address that by allowing the model size to grow without substantially raising training or inference costs.<n>MoE models face challenges regarding knowledge sharing among experts, making their performance somehow sensitive to routing accuracy.<n>In this paper, we propose CartesianMoE, which implements more effective knowledge sharing among experts in more like a multiplication'' manner.
  arXiv  Detail & Related papers  (2024-10-21T14:55:59Z)
• Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts [75.85448576746373]
  We propose a method of grouping and pruning similar experts to improve the model's parameter efficiency. We validate the effectiveness of our method by pruning three state-of-the-art MoE architectures. The evaluation shows that our method outperforms other model pruning methods on a range of natural language tasks.
  arXiv  Detail & Related papers  (2024-07-12T17:25:02Z)
• Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models [57.582219834039506]
  We introduce the training methodologies implemented in the development of Skywork-MoE, a high-performance mixture-of-experts (MoE) large language model (LLM) with 146 billion parameters and 16 experts. It is based on the pre-existing dense checkpoints of our Skywork-13B model.
  arXiv  Detail & Related papers  (2024-06-03T03:58:41Z)
• A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts [49.394145046409044]
  This paper provides the first provably efficient technique for pruning experts in finetuned MoE models. We theoretically prove that prioritizing the pruning of the experts with a smaller change of the routers l2 norm from the pretrained model guarantees the preservation of test accuracy. Although our theoretical analysis is centered on binary classification tasks on simplified MoE architecture, our expert pruning method is verified on large vision MoE models.
  arXiv  Detail & Related papers  (2024-05-26T17:52:58Z)
• SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts [49.01990048827639]
  We introduce SEER-MoE, a framework for reducing both the memory footprint and compute requirements of pre-trained MoE models. The first stage involves pruning the total number of experts using a heavy-hitters counting guidance, while the second stage employs a regularization-based fine-tuning strategy to recover accuracy loss. Our empirical studies demonstrate the effectiveness of our method, resulting in a sparse MoEs model optimized for inference efficiency with minimal accuracy trade-offs.
  arXiv  Detail & Related papers  (2024-04-07T22:13:43Z)
• HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts [25.504602853436047]
  Mixture of Experts (MoE) for language models has been proven effective in augmenting the capacity of models by dynamically routing each input token to a specific subset of experts for processing. We propose HyperMoE, a novel MoE framework built upon Hypernetworks. This framework integrates the computational processes of MoE with the concept of knowledge transferring in multi-task learning.
  arXiv  Detail & Related papers  (2024-02-20T02:09:55Z)
• Comparative Knowledge Distillation [102.35425896967791]
  Traditional Knowledge Distillation (KD) assumes readily available access to teacher models for frequent inference. We propose Comparative Knowledge Distillation (CKD), which encourages student models to understand the nuanced differences in a teacher model's interpretations of samples. CKD consistently outperforms state of the art data augmentation and KD techniques.
  arXiv  Detail & Related papers  (2023-11-03T21:55:33Z)
• Knowledge Distillation for Adaptive MRI Prostate Segmentation Based on Limit-Trained Multi-Teacher Models [4.711401719735324]
  Knowledge Distillation (KD) has been proposed as a compression method and an acceleration technology. KD is an efficient learning strategy that can transfer knowledge from a burdensome model to a lightweight model. We develop a KD-based deep model for prostate MRI segmentation in this work by combining features-based distillation with Kullback-Leibler divergence, Lovasz, and Dice losses.
  arXiv  Detail & Related papers  (2023-03-16T17:15:08Z)
• Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classification [106.08067870620218]
  We propose a self-paced knowledge distillation framework, termed Learning From Multiple Experts (LFME) We refer to these models as 'Experts', and the proposed LFME framework aggregates the knowledge from multiple 'Experts' to learn a unified student model. We conduct extensive experiments and demonstrate that our method is able to achieve superior performances compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-01-06T12:57:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.