Enhancing deep learning models for time series classification via knowledge distillation

• URL: http://arxiv.org/abs/2607.06796v1
• Date: Tue, 07 Jul 2026 20:51:22 GMT
• Title: Enhancing deep learning models for time series classification via knowledge distillation
• Abstract summary: Knowledge Distillation (KD) addresses this by transferring knowledge from a large teacher model to a smaller, more efficient student model.<n>We evaluate our approach on UCR Archive, the largest benchmark repository of time series datasets.
• Score: 0.49387059996366106
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep learning has achieved remarkable success in various domains including time series analysis, computer vision and natural language processing. However, high computational and memory demands of state-of-the-art architectures pose challenges for deployment in resource-limited environments. Knowledge Distillation (KD) addresses this by transferring knowledge from a large teacher model to a smaller, more efficient student model while maintaining competitive performance. In this work, we investigate the effectiveness of KD for Time Series Classification (TSC) across three architectures: the classical Fully Convolutional Network (FCN), the convolutional Inception model and the transformer-based ConvTran model. We evaluate our approach on UCR Archive, the largest benchmark repository of time series datasets, by modifying architectural components such as convolutional filters, Inception modules and attention heads across the three architectures. Our results consistently show that KD most effectively benefits student models of intermediate complexity across all three architectures, with the distilled FCN student reducing parameters by a factor of 38, the distilled Inception student achieving nearly the same performance as the teacher with 42% fewer parameters and the distilled ConvTran student with 2 attention heads showing the most significant improvement through distillation. To encourage further research and reproducibility, we provide our implementation at https://github.com/MSD-IRIMAS/KD-4-TSC.

Related papers

• Empirical Evaluation of Knowledge Distillation from Transformers to Subquadratic Language Models [3.287942619833188]
  We systematically evaluate the transferability of knowledge distillation from a Transformer teacher model to eight subquadratic student architectures.<n>Our study investigates which subquadratic model can most effectively approximate the teacher model's learned representations through knowledge distillation.
  arXiv  Detail & Related papers  (2025-04-19T17:49:52Z)
• Perspective-Aware Teaching: Adapting Knowledge for Heterogeneous Distillation [36.18463357719423]
  We introduce a perspective-aware teaching (PAT) KD framework to enable feature distillation across diverse architectures.<n>Our framework comprises two key components. First, we design prompt tuning blocks that incorporate student feedback, allowing teacher features to adapt to the student model's learning process.
  arXiv  Detail & Related papers  (2025-01-15T15:56:06Z)
• Exploring Kolmogorov-Arnold Networks for Interpretable Time Series Classification [4.489167091323233]
  Kolmogorov-Arnold Networks (KANs) have been proposed as a more interpretable alternative to deep learning.<n>This paper explores the KAN architecture for time series classification utilising 117 datasets from UCR benchmark archive.
  arXiv  Detail & Related papers  (2024-11-22T13:01:36Z)
• Look Into the LITE in Deep Learning for Time Series Classification [6.708543240320757]
  Deep learning models have been shown to be a powerful solution for Time Series Classification.<n>We present a new architecture for TSC, the Light Inception with boosTing tEchnique (LITE) with only 2.34% of the number of parameters of the state-of-the-art InceptionTime model.<n>The LITE architecture, trained on the UCR, is 2.78 times faster than InceptionTime and consumes 2.79 times less CO2 and power.
  arXiv  Detail & Related papers  (2024-09-04T16:53:46Z)
• One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation [69.65734716679925]
  Knowledge distillation has proven to be a highly effective approach for enhancing model performance through a teacher-student training scheme. Most existing distillation methods are designed under the assumption that the teacher and student models belong to the same model family. We propose a simple yet effective one-for-all KD framework called OFA-KD, which significantly improves the distillation performance between heterogeneous architectures.
  arXiv  Detail & Related papers  (2023-10-30T11:13:02Z)
• Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation [56.053397775016755]
  We propose a sequential approach to knowledge distillation that progressively transfers the knowledge of a set of teacher detectors to a given lightweight student. To the best of our knowledge, we are the first to successfully distill knowledge from Transformer-based teacher detectors to convolution-based students.
  arXiv  Detail & Related papers  (2023-08-17T17:17:08Z)
• EmbedDistill: A Geometric Knowledge Distillation for Information Retrieval [83.79667141681418]
  Large neural models (such as Transformers) achieve state-of-the-art performance for information retrieval (IR) We propose a novel distillation approach that leverages the relative geometry among queries and documents learned by the large teacher model. We show that our approach successfully distills from both dual-encoder (DE) and cross-encoder (CE) teacher models to 1/10th size asymmetric students that can retain 95-97% of the teacher performance.
  arXiv  Detail & Related papers  (2023-01-27T22:04:37Z)
• Directed Acyclic Graph Factorization Machines for CTR Prediction via Knowledge Distillation [65.62538699160085]
  We propose a Directed Acyclic Graph Factorization Machine (KD-DAGFM) to learn the high-order feature interactions from existing complex interaction models for CTR prediction via Knowledge Distillation. KD-DAGFM achieves the best performance with less than 21.5% FLOPs of the state-of-the-art method on both online and offline experiments.
  arXiv  Detail & Related papers  (2022-11-21T03:09:42Z)
• Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models [68.9288651177564]
  We present a novel MoE architecture based on matrix product operators (MPO) from quantum many-body physics. With the decomposed MPO structure, we can reduce the parameters of the original MoE architecture. Experiments on the three well-known downstream natural language datasets based on GPT2 show improved performance and efficiency in increasing model capacity.
  arXiv  Detail & Related papers  (2022-03-02T13:44:49Z)
• AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family [10.51711053229702]
  State-of-the-art results in deep learning have been improving steadily, in good part due to the use of larger models. While Knowledge Distillation (KD) theoretically enables small student models to emulate larger teacher models, in practice selecting a good student architecture requires considerable human expertise. In this paper, we propose to instead search for a family of student architectures sharing the property of being good at learning from a given teacher.
  arXiv  Detail & Related papers  (2021-11-05T15:20:37Z)
• Efficient Crowd Counting via Structured Knowledge Transfer [122.30417437707759]
  Crowd counting is an application-oriented task and its inference efficiency is crucial for real-world applications. We propose a novel Structured Knowledge Transfer framework to generate a lightweight but still highly effective student network. Our models obtain at least 6.5$times$ speed-up on an Nvidia 1080 GPU and even achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-03-23T08:05:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.