Self-Feature Regularization: Self-Feature Distillation Without Teacher Models

• URL: http://arxiv.org/abs/2103.07350v2
• Date: Tue, 16 Mar 2021 17:10:13 GMT
• Title: Self-Feature Regularization: Self-Feature Distillation Without Teacher Models
• Authors: Wenxuan Fan, Zhenyan Hou
• Abstract summary: Self-Feature Regularization(SFR) is proposed, which uses features in the deep layers to supervise feature learning in the shallow layers. We firstly use generalization-l2 loss to match local features and a many-to-one approach to distill more intensively in the channel dimension.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Knowledge distillation is the process of transferring the knowledge from a large model to a small model. In this process, the small model learns the generalization ability of the large model and retains the performance close to that of the large model. Knowledge distillation provides a training means to migrate the knowledge of models, facilitating model deployment and speeding up inference. However, previous distillation methods require pre-trained teacher models, which still bring computational and storage overheads. In this paper, a novel general training framework called Self-Feature Regularization~(SFR) is proposed, which uses features in the deep layers to supervise feature learning in the shallow layers, retains more semantic information. Specifically, we firstly use EMD-l2 loss to match local features and a many-to-one approach to distill features more intensively in the channel dimension. Then dynamic label smoothing is used in the output layer to achieve better performance. Experiments further show the effectiveness of our proposed framework.

Related papers

• Machine Unlearning on Pre-trained Models by Residual Feature Alignment Using LoRA [15.542668474378633]
  We propose a novel and efficient machine unlearning method on pre-trained models. We leverage LoRA to decompose the model's intermediate features into pre-trained features and residual features. The method aims to learn the zero residuals on the retained set and shifted residuals on the unlearning set.
  arXiv  Detail & Related papers  (2024-11-13T08:56:35Z)
• Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models [62.5501109475725]
  Knowledge distillation (KD) is a technique that compresses large teacher models by training smaller student models to mimic them. This paper introduces Online Knowledge Distillation (OKD), where the teacher network integrates small online modules to concurrently train with the student model. OKD achieves or exceeds the performance of leading methods in various model architectures and sizes, reducing training time by up to fourfold.
  arXiv  Detail & Related papers  (2024-09-19T07:05:26Z)
• Efficient Knowledge Deletion from Trained Models through Layer-wise Partial Machine Unlearning [2.3496568239538083]
  This paper introduces a novel class of machine unlearning algorithms. First method is partial amnesiac unlearning, integration of layer-wise pruning with amnesiac unlearning. Second method assimilates layer-wise partial-updates into label-flipping and optimization-based unlearning.
  arXiv  Detail & Related papers  (2024-03-12T12:49:47Z)
• Knowledge Distillation for Road Detection based on cross-model Semi-Supervised Learning [17.690698736544626]
  We propose an integrated approach that combines knowledge distillation and semi-supervised learning methods. This hybrid approach leverages the robust capabilities of large models to effectively utilise large unlabelled data. The proposed semi-supervised learning-based knowledge distillation (SSLKD) approach demonstrates a notable improvement in the performance of the student model.
  arXiv  Detail & Related papers  (2024-02-07T22:50:47Z)
• Generative Model-based Feature Knowledge Distillation for Action Recognition [11.31068233536815]
  Our paper introduces an innovative knowledge distillation framework, with the generative model for training a lightweight student model. The efficacy of our approach is demonstrated through comprehensive experiments on diverse popular datasets.
  arXiv  Detail & Related papers  (2023-12-14T03:55:29Z)
• FD-Align: Feature Discrimination Alignment for Fine-tuning Pre-Trained Models in Few-Shot Learning [21.693779973263172]
  In this paper, we introduce a fine-tuning approach termed Feature Discrimination Alignment (FD-Align) Our method aims to bolster the model's generalizability by preserving the consistency of spurious features. Once fine-tuned, the model can seamlessly integrate with existing methods, leading to performance improvements.
  arXiv  Detail & Related papers  (2023-10-23T17:12:01Z)
• Towards Efficient Task-Driven Model Reprogramming with Foundation Models [52.411508216448716]
  Vision foundation models exhibit impressive power, benefiting from the extremely large model capacity and broad training data. However, in practice, downstream scenarios may only support a small model due to the limited computational resources or efficiency considerations. This brings a critical challenge for the real-world application of foundation models: one has to transfer the knowledge of a foundation model to the downstream task.
  arXiv  Detail & Related papers  (2023-04-05T07:28:33Z)
• Prototype-guided Cross-task Knowledge Distillation for Large-scale Models [103.04711721343278]
  Cross-task knowledge distillation helps to train a small student model to obtain a competitive performance. We propose a Prototype-guided Cross-task Knowledge Distillation (ProC-KD) approach to transfer the intrinsic local-level object knowledge of a large-scale teacher network to various task scenarios.
  arXiv  Detail & Related papers  (2022-12-26T15:00:42Z)
• MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation [68.30497162547768]
  We propose MoEBERT, which uses a Mixture-of-Experts structure to increase model capacity and inference speed. We validate the efficiency and effectiveness of MoEBERT on natural language understanding and question answering tasks.
  arXiv  Detail & Related papers  (2022-04-15T23:19:37Z)
• Powerpropagation: A sparsity inducing weight reparameterisation [65.85142037667065]
  We introduce Powerpropagation, a new weight- parameterisation for neural networks that leads to inherently sparse models. Models trained in this manner exhibit similar performance, but have a distribution with markedly higher density at zero, allowing more parameters to be pruned safely. Here, we combine Powerpropagation with a traditional weight-pruning technique as well as recent state-of-the-art sparse-to-sparse algorithms, showing superior performance on the ImageNet benchmark.
  arXiv  Detail & Related papers  (2021-10-01T10:03:57Z)
• Reinforced Multi-Teacher Selection for Knowledge Distillation [54.72886763796232]
  knowledge distillation is a popular method for model compression. Current methods assign a fixed weight to a teacher model in the whole distillation. Most of the existing methods allocate an equal weight to every teacher model. In this paper, we observe that, due to the complexity of training examples and the differences in student model capability, learning differentially from teacher models can lead to better performance of student models distilled.
  arXiv  Detail & Related papers  (2020-12-11T08:56:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.