DKD-KAN: A Lightweight knowledge-distilled KAN intrusion detection framework, based on MLP and KAN

• URL: http://arxiv.org/abs/2603.03486v1
• Date: Tue, 03 Mar 2026 20:02:11 GMT
• Title: DKD-KAN: A Lightweight knowledge-distilled KAN intrusion detection framework, based on MLP and KAN
• Abstract summary: A light-weight intrusion detection framework is proposed that utilizes the Kolmogorov-Arnold Network (KAN) to capture complex features in the data.<n>A high-capacity KAN network is first trained to detect attacks performed on the test bed.<n>This model then serves as a teacher to guide a much smaller multilayer perceptron (MLP) student model via DKD.<n>The resulting DKD-MLP model contains only 2,522 and 1,622 parameters for WADI and SWaT datasets, which are significantly smaller than the number of parameters of the KAN teacher model.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cyber-security systems often operate in resource-constrained environments, such as edge environments and real-time monitoring systems, where model size and inference time are crucial. A light-weight intrusion detection framework is proposed that utilizes the Kolmogorov-Arnold Network (KAN) to capture complex features in the data, with the efficiency of decoupled knowledge distillation (DKD) training approach. A high-capacity KAN network is first trained to detect attacks performed on the test bed. This model then serves as a teacher to guide a much smaller multilayer perceptron (MLP) student model via DKD. The resulting DKD-MLP model contains only 2,522 and 1,622 parameters for WADI and SWaT datasets, which are significantly smaller than the number of parameters of the KAN teacher model. This is highly appropriate for deployment in resource-constrained devices with limited computational resources. Despite its low size, the student model maintains a high performance. Our approach demonstrate the practicality of using KAN as a knowledge-rich teacher to train much smaller student models, without considerable drop in accuracy in intrusion detection frameworks. We have validated our approach on two publicly available datasets. We report F1-score improvements of 4.18% on WADI and 3.07% on SWaT when using the DKD-MLP model, compared to the bare student model. The implementation of this paper is available on our GitHub repository.

Related papers

• Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails [16.73080036450313]
  Large language models (LLMs) achieve strong performance across many tasks, but their high computational cost limits deployment in resource-constrained environments.<n> Knowledge Distillation (KD) offers a practical solution by transferring knowledge from a teacher model of a larger size to a smaller student model.
  arXiv  Detail & Related papers  (2026-06-22T07:19:43Z)
• Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models [62.5501109475725]
  Knowledge distillation (KD) is a technique that compresses large teacher models by training smaller student models to mimic them. This paper introduces Online Knowledge Distillation (OKD), where the teacher network integrates small online modules to concurrently train with the student model. OKD achieves or exceeds the performance of leading methods in various model architectures and sizes, reducing training time by up to fourfold.
  arXiv  Detail & Related papers  (2024-09-19T07:05:26Z)
• DistiLLM: Towards Streamlined Distillation for Large Language Models [53.46759297929675]
  DistiLLM is a more effective and efficient KD framework for auto-regressive language models. DisiLLM comprises two components: (1) a novel skew Kullback-Leibler divergence loss, where we unveil and leverage its theoretical properties, and (2) an adaptive off-policy approach designed to enhance the efficiency in utilizing student-generated outputs.
  arXiv  Detail & Related papers  (2024-02-06T11:10:35Z)
• Knowledge Distillation of LLM for Automatic Scoring of Science Education Assessments [4.541309099803903]
  This study proposes a method for knowledge distillation (KD) of fine-tuned Large Language Models (LLMs) We specifically target the challenge of deploying these models on resource-constrained devices.
  arXiv  Detail & Related papers  (2023-12-26T01:24:25Z)
• Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning [3.1423836318272773]
  Knowledge distillation (KD) improves the performance of efficient and lightweight models.<n>Most existing KD techniques rely on Kullback-Leibler (KL) divergence.<n>We propose a Robustness-Reinforced Knowledge Distillation (R2KD) that leverages correlation distance and network pruning.
  arXiv  Detail & Related papers  (2023-11-23T11:34:48Z)
• Towards a robust and reliable deep learning approach for detection of compact binary mergers in gravitational wave data [0.0]
  We develop a deep learning model stage-wise and work towards improving its robustness and reliability. We retrain the model in a novel framework involving a generative adversarial network (GAN) Although absolute robustness is practically impossible to achieve, we demonstrate some fundamental improvements earned through such training.
  arXiv  Detail & Related papers  (2023-06-20T18:00:05Z)
• MiniLLM: Knowledge Distillation of Large Language Models [112.93051247165089]
  Knowledge Distillation (KD) is a promising technique for reducing the high computational demand of large language models (LLMs) We propose a KD approach that distills LLMs into smaller language models. Our method is scalable for different model families with 120M to 13B parameters.
  arXiv  Detail & Related papers  (2023-06-14T14:44:03Z)
• Knowledge Distillation with Representative Teacher Keys Based on Attention Mechanism for Image Classification Model Compression [1.503974529275767]
  knowledge distillation (KD) has been recognized as one of the effective method of model compression to decrease the model parameters. Inspired by attention mechanism, we propose a novel KD method called representative teacher key (RTK) Our proposed RTK can effectively improve the classification accuracy of the state-of-the-art attention-based KD method.
  arXiv  Detail & Related papers  (2022-06-26T05:08:50Z)
• Training strategy for a lightweight countermeasure model for automatic speaker verification [6.174721516017139]
  This work proposes training strategies for a lightweight CM model for ASV. In the evalua- tion phase of the ASVspoof 2021 Logical Access task, the lightweight ResNetSE model reaches min t-DCF 0.2695 and EER 3.54%.
  arXiv  Detail & Related papers  (2022-03-31T13:52:43Z)
• How and When Adversarial Robustness Transfers in Knowledge Distillation? [137.11016173468457]
  This paper studies how and when the adversarial robustness can be transferred from a teacher model to a student model in Knowledge distillation (KD) We show that standard KD training fails to preserve adversarial robustness, and we propose KD with input gradient alignment (KDIGA) for remedy. Under certain assumptions, we prove that the student model using our proposed KDIGA can achieve at least the same certified robustness as the teacher model.
  arXiv  Detail & Related papers  (2021-10-22T21:30:53Z)
• Efficient training of lightweight neural networks using Online Self-Acquired Knowledge Distillation [51.66271681532262]
  Online Self-Acquired Knowledge Distillation (OSAKD) is proposed, aiming to improve the performance of any deep neural model in an online manner. We utilize k-nn non-parametric density estimation technique for estimating the unknown probability distributions of the data samples in the output feature space.
  arXiv  Detail & Related papers  (2021-08-26T14:01:04Z)
• MixKD: Towards Efficient Distillation of Large-scale Language Models [129.73786264834894]
  We propose MixKD, a data-agnostic distillation framework, to endow the resulting model with stronger generalization ability. We prove from a theoretical perspective that under reasonable conditions MixKD gives rise to a smaller gap between the error and the empirical error. Experiments under a limited-data setting and ablation studies further demonstrate the advantages of the proposed approach.
  arXiv  Detail & Related papers  (2020-11-01T18:47:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.