MoEKD: Mixture-of-Experts Knowledge Distillation for Robust and High-Performing Compressed Code Models

• URL: http://arxiv.org/abs/2603.13213v1
• Date: Fri, 13 Mar 2026 17:49:01 GMT
• Title: MoEKD: Mixture-of-Experts Knowledge Distillation for Robust and High-Performing Compressed Code Models
• Abstract summary: Large language models for code have achieved strong performance across diverse software analytics tasks.<n> Knowledge distillation (KD) offers a practical solution by transferring knowledge from a large model to a smaller and more efficient model.<n>MoEKD decomposes the distillation process into expert and router training, aggregation of expert knowledge through a learned routing mechanism, and distillation from the aggregated knowledge.
• Score: 6.25009626782699
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large language models for code have achieved strong performance across diverse software analytics tasks, yet their real-world adoption remains limited by high computational demands, slow inference speeds, significant energy consumption, and environmental impact. Knowledge distillation (KD) offers a practical solution by transferring knowledge from a large model to a smaller and more efficient model. Despite its effectiveness, recent studies show that models distilled from a single source often exhibit degraded adversarial robustness, even when robustness-aware distillation techniques are employed. These observations suggest a fundamental limitation of single-source distillation in simultaneously transferring high-quality and robust knowledge. To overcome this limitation, we propose Mixture of Experts Knowledge Distillation (MoEKD), a KD framework that leverages a Mixture of Experts (MoE) architecture to enable more effective and robust knowledge transfer from multiple specialized experts into a compact model. MoEKD decomposes the distillation process into expert and router training, aggregation of expert knowledge through a learned routing mechanism, and distillation from the aggregated knowledge. We evaluate MoEKD on the vulnerability detection task using CodeBERT and GraphCodeBERT models. Experimental results show that MoEKD not only improves adversarial robustness by up to 35.8%, but also enhances predictive performance by up to 13%, compared to state-of-the-art KD baselines, including Compressor and AVATAR. Furthermore, an ablation study demonstrates that aggregating expert knowledge enables ultra-compact models to maintain competitive performance even when their size is reduced by approximately half. Overall, these results highlight the effectiveness of multi-expert knowledge aggregation in addressing key limitations of existing single-source KD approaches.

Related papers

• UniComp: A Unified Evaluation of Large Language Model Compression via Pruning, Quantization and Distillation [23.560232846931456]
  We introduce UniComp, a unified evaluation framework for comparing pruning, quantization, and knowledge distillation.<n>UniComp evaluates compressed models along three dimensions: performance, reliability, and efficiency.
  arXiv  Detail & Related papers  (2026-02-09T19:20:56Z)
• Towards Distillation-Resistant Large Language Models: An Information-Theoretic Perspective [52.25797439810419]
  Existing defenses focus exclusively on text-based distillation, leaving the important logit-based distillation largely unexplored.<n>We characterize distillation-relevant information in teacher outputs using the conditional mutual information (CMI) between teacher logits and input queries conditioned on ground-truth labels.<n>We derive a CMI-inspired anti-distillation objective to optimize this transformation, which effectively removes distillation-relevant information while preserving output utility.
  arXiv  Detail & Related papers  (2026-02-03T11:16:59Z)
• Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMs [30.543860587982895]
  We introduce the concept of textbfKnowledge Purification, which consolidates the rationales from multiple teacher LLMs into a single rationale.<n>Our experiments demonstrate that these methods not only improve the performance of the distilled model but also effectively alleviate knowledge conflicts.
  arXiv  Detail & Related papers  (2026-02-01T07:19:57Z)
• Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models [30.70974942397732]
  We show that non-activated experts in MoE models possess valuable knowledge that benefits student models.<n>We propose two intuitive MoE-specific KD methods for the first time: Knowledge Augmentation (KA) and Student-Aware Router (SAR)
  arXiv  Detail & Related papers  (2025-02-18T15:30:34Z)
• Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts [75.85448576746373]
  We propose a method of grouping and pruning similar experts to improve the model's parameter efficiency.<n>We validate the effectiveness of our method by pruning three state-of-the-art MoE architectures.<n>The evaluation shows that our method outperforms other model pruning methods on a range of natural language tasks.
  arXiv  Detail & Related papers  (2024-07-12T17:25:02Z)
• Teaching with Uncertainty: Unleashing the Potential of Knowledge Distillation in Object Detection [47.0507287491627]
  We propose a novel feature-based distillation paradigm with knowledge uncertainty for object detection. By leveraging the Monte Carlo dropout technique, we introduce knowledge uncertainty into the training process of the student model. Our method performs effectively during the KD process without requiring intricate structures or extensive computational resources.
  arXiv  Detail & Related papers  (2024-06-11T06:51:02Z)
• Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning [3.1423836318272773]
  Knowledge distillation (KD) improves the performance of efficient and lightweight models.<n>Most existing KD techniques rely on Kullback-Leibler (KL) divergence.<n>We propose a Robustness-Reinforced Knowledge Distillation (R2KD) that leverages correlation distance and network pruning.
  arXiv  Detail & Related papers  (2023-11-23T11:34:48Z)
• On the benefits of knowledge distillation for adversarial robustness [53.41196727255314]
  We show that knowledge distillation can be used directly to boost the performance of state-of-the-art models in adversarial robustness. We present Adversarial Knowledge Distillation (AKD), a new framework to improve a model's robust performance.
  arXiv  Detail & Related papers  (2022-03-14T15:02:13Z)
• MixKD: Towards Efficient Distillation of Large-scale Language Models [129.73786264834894]
  We propose MixKD, a data-agnostic distillation framework, to endow the resulting model with stronger generalization ability. We prove from a theoretical perspective that under reasonable conditions MixKD gives rise to a smaller gap between the error and the empirical error. Experiments under a limited-data setting and ablation studies further demonstrate the advantages of the proposed approach.
  arXiv  Detail & Related papers  (2020-11-01T18:47:51Z)
• Residual Knowledge Distillation [96.18815134719975]
  This work proposes Residual Knowledge Distillation (RKD), which further distills the knowledge by introducing an assistant (A) In this way, S is trained to mimic the feature maps of T, and A aids this process by learning the residual error between them. Experiments show that our approach achieves appealing results on popular classification datasets, CIFAR-100 and ImageNet.
  arXiv  Detail & Related papers  (2020-02-21T07:49:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.