Label-Guided Knowledge Distillation for 3D-CNNs in Action Recognition

• URL: http://arxiv.org/abs/2609.13024v1
• Date: Fri, 11 Sep 2026 16:21:21 GMT
• Title: Label-Guided Knowledge Distillation for 3D-CNNs in Action Recognition
• Abstract summary: We introduce Label-Guided Knowledge Distillation (LGKD) to guide the distillation of student model features using ground truth labels.<n>Our method entails two components: sample-wise distillation and class-wise distillation, enabling the student model to learn feature representation of the teacher model at two levels.
• Score: 10.351322944851095
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As a key model compression technique, knowledge distillation aims to transfer knowledge from a high-capacity teacher model to a lightweight student model for enhancing the latter's performance. In this work, we reviewed the feature knowledge distillation for 3D-CNNs and observed that most feature distillation methods in video analysis are simple adaptations of those used in image analysis, often neglecting the differences of video features in the temporal dimension. To address this issue, we proposed Label-Guided Knowledge Distillation (LGKD) to guide the distillation of student model features using ground truth labels. Our method entails two components: sample-wise distillation and class-wise distillation, enabling the student model to learn feature representation of the teacher model at two levels. Sample-wise distillation utilizes label information and the teacher's probability distribution to guide the learning of features that significantly impact temporal accuracy while mitigating noise. Meanwhile, class-wise feature distillation employs a prototype network to further capture the relational knowledge among samples within the same category, enhancing the student's ability to learn higher-dimensional semantic information and improving model generalization. To demonstrate the effectiveness and superiority of our method, we conducted comprehensive experiments on two benchmark action recognition datasets, UCF101 and HMDB51, achieving competitive results.

Related papers

• Pedagogically-Inspired Data Synthesis for Language Model Knowledge Distillation [63.302074484672424]
  We propose a pedagogically-inspired framework for knowledge distillation.<n>Our approach identifies knowledge deficiencies in student models, organizes knowledge delivery through progressive curricula, and adapts representations to match cognitive capacity of student models.<n>Our framework particularly excels in complex reasoning tasks, showing 19.2% improvement on MATH and 22.3% on HumanEval compared with state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-02-12T17:00:36Z)
• Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation [64.15918654558816]
  Self-distillation (SSD) training strategy is introduced for filtering and weighting teacher representation to distill from task-relevant representations only.<n> Experimental results on real-world affective computing, wearable/biosignal datasets from the UCR Archive, the HAR dataset, and image classification datasets show that the proposed SSD method can outperform state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-04-19T14:08:56Z)
• Dual Knowledge Distillation for Efficient Sound Event Detection [20.236008919003083]
  Sound event detection (SED) is essential for recognizing specific sounds and their temporal locations within acoustic signals. We introduce a novel framework referred to as dual knowledge distillation for developing efficient SED systems.
  arXiv  Detail & Related papers  (2024-02-05T07:30:32Z)
• Knowledge Diffusion for Distillation [53.908314960324915]
  The representation gap between teacher and student is an emerging topic in knowledge distillation (KD) We state that the essence of these methods is to discard the noisy information and distill the valuable information in the feature. We propose a novel KD method dubbed DiffKD, to explicitly denoise and match features using diffusion models.
  arXiv  Detail & Related papers  (2023-05-25T04:49:34Z)
• AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression [26.474962405945316]
  We present a novel attribution-driven knowledge distillation approach to compress pre-trained language models. To enhance the knowledge transfer of model reasoning and generalization, we explore multi-view attribution distillation on all potential decisions of the teacher.
  arXiv  Detail & Related papers  (2023-05-17T07:40:12Z)
• Exploring Inconsistent Knowledge Distillation for Object Detection with Data Augmentation [66.25738680429463]
  Knowledge Distillation (KD) for object detection aims to train a compact detector by transferring knowledge from a teacher model. We propose inconsistent knowledge distillation (IKD) which aims to distill knowledge inherent in the teacher model's counter-intuitive perceptions. Our method outperforms state-of-the-art KD baselines on one-stage, two-stage and anchor-free object detectors.
  arXiv  Detail & Related papers  (2022-09-20T16:36:28Z)
• Normalized Feature Distillation for Semantic Segmentation [6.882655287146012]
  We propose a simple yet effective feature distillation method called normalized feature distillation (NFD) Our method achieves state-of-the-art distillation results for semantic segmentation on Cityscapes, VOC 2012, and ADE20K datasets.
  arXiv  Detail & Related papers  (2022-07-12T01:54:25Z)
• On the benefits of knowledge distillation for adversarial robustness [53.41196727255314]
  We show that knowledge distillation can be used directly to boost the performance of state-of-the-art models in adversarial robustness. We present Adversarial Knowledge Distillation (AKD), a new framework to improve a model's robust performance.
  arXiv  Detail & Related papers  (2022-03-14T15:02:13Z)
• Self Regulated Learning Mechanism for Data Efficient Knowledge Distillation [8.09591217280048]
  A novel data-efficient approach to transfer the knowledge from a teacher model to a student model is presented. The teacher model uses self-regulation to select appropriate samples for training and identifies their significance in the process. During distillation, the significance information can be used along with the soft-targets to supervise the students.
  arXiv  Detail & Related papers  (2021-02-14T10:43:13Z)
• Knowledge Distillation Meets Self-Supervision [109.6400639148393]
  Knowledge distillation involves extracting "dark knowledge" from a teacher network to guide the learning of a student network. We show that the seemingly different self-supervision task can serve as a simple yet powerful solution. By exploiting the similarity between those self-supervision signals as an auxiliary task, one can effectively transfer the hidden information from the teacher to the student.
  arXiv  Detail & Related papers  (2020-06-12T12:18:52Z)
• Understanding and Improving Knowledge Distillation [13.872105118381938]
  Knowledge Distillation (KD) is a model-agnostic technique to improve model quality while having a fixed capacity budget. This paper categorizes teacher's knowledge into three hierarchical levels and study its effects on knowledge distillation.
  arXiv  Detail & Related papers  (2020-02-10T04:21:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.