Less is More: Task-aware Layer-wise Distillation for Language Model Compression

• URL: http://arxiv.org/abs/2210.01351v3
• Date: Mon, 5 Jun 2023 22:40:20 GMT
• Title: Less is More: Task-aware Layer-wise Distillation for Language Model Compression
• Abstract summary: Layer-wise distillation is a powerful tool to compress large models (i.e. teacher models) into small ones. We propose a novel Task-aware layEr-wise Distillation (TED) TED designs task-aware filters to align the hidden representations of the student and the teacher at each layer.
• Score: 68.30497162547766
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Layer-wise distillation is a powerful tool to compress large models (i.e. teacher models) into small ones (i.e., student models). The student distills knowledge from the teacher by mimicking the hidden representations of the teacher at every intermediate layer. However, layer-wise distillation is difficult. Since the student has a smaller model capacity than the teacher, it is often under-fitted. Furthermore, the hidden representations of the teacher contain redundant information that the student does not necessarily need for the target task's learning. To address these challenges, we propose a novel Task-aware layEr-wise Distillation (TED). TED designs task-aware filters to align the hidden representations of the student and the teacher at each layer. The filters select the knowledge that is useful for the target task from the hidden representations. As such, TED reduces the knowledge gap between the two models and helps the student to fit better on the target task. We evaluate TED in two scenarios: continual pre-training and fine-tuning. TED demonstrates significant and consistent improvements over existing distillation methods in both scenarios. Code is available at https://github.com/cliang1453/task-aware-distillation.

Related papers

• Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation [64.15918654558816]
  Self-distillation (SSD) training strategy is introduced for filtering and weighting teacher representation to distill from task-relevant representations only. Experimental results on real-world affective computing, wearable/biosignal datasets from the UCR Archive, the HAR dataset, and image classification datasets show that the proposed SSD method can outperform state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-04-19T14:08:56Z)
• Triplet Knowledge Distillation [73.39109022280878]
  In Knowledge Distillation, the teacher is generally much larger than the student, making the solution of the teacher likely to be difficult for the student to learn. To ease the mimicking difficulty, we introduce a triplet knowledge distillation mechanism named TriKD.
  arXiv  Detail & Related papers  (2023-05-25T12:12:31Z)
• ERNIE 3.0 Tiny: Frustratingly Simple Method to Improve Task-Agnostic Distillation Generalization [36.338614215561805]
  Task-agnostic knowledge distillation attempts to address the problem of deploying large pretrained language model in resource-constrained scenarios. We show that we can leverage multi-task learning in task-agnostic distillation to advance the generalization of the resulted student.
  arXiv  Detail & Related papers  (2023-01-09T15:12:50Z)
• Representation Consolidation for Training Expert Students [54.90754502493968]
  We show that a multi-head, multi-task distillation method is sufficient to consolidate representations from task-specific teacher(s) and improve downstream performance. Our method can also combine the representational knowledge of multiple teachers trained on one or multiple domains into a single model.
  arXiv  Detail & Related papers  (2021-07-16T17:58:18Z)
• Fixing the Teacher-Student Knowledge Discrepancy in Distillation [72.4354883997316]
  We propose a novel student-dependent distillation method, knowledge consistent distillation, which makes teacher's knowledge more consistent with the student. Our method is very flexible that can be easily combined with other state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-03-31T06:52:20Z)
• ALP-KD: Attention-Based Layer Projection for Knowledge Distillation [30.896957367331137]
  Two neural networks, namely a teacher and a student, are coupled together during training. The teacher network is supposed to be a trustworthy predictor and the student tries to mimic its predictions. In such a setting, distillation only happens for final predictions whereas the student could also benefit from teacher's supervision for internal components.
  arXiv  Detail & Related papers  (2020-12-27T22:30:13Z)
• Progressive Network Grafting for Few-Shot Knowledge Distillation [60.38608462158474]
  We introduce a principled dual-stage distillation scheme tailored for few-shot data. In the first step, we graft the student blocks one by one onto the teacher, and learn the parameters of the grafted block intertwined with those of the other teacher blocks. Experiments demonstrate that our approach, with only a few unlabeled samples, achieves gratifying results on CIFAR10, CIFAR100, and ILSVRC-2012.
  arXiv  Detail & Related papers  (2020-12-09T08:34:36Z)
• Reducing the Teacher-Student Gap via Spherical Knowledge Disitllation [67.75526580926149]
  Knowledge distillation aims at obtaining a compact and effective model by learning the mapping function from a much larger one. We investigate the capacity gap problem by study the gap of confidence between teacher and student. We find that the magnitude of confidence is not necessary for knowledge distillation and could harm the student performance if the student are forced to learn confidence.
  arXiv  Detail & Related papers  (2020-10-15T03:03:36Z)
• Channel Distillation: Channel-Wise Attention for Knowledge Distillation [3.6269274596116476]
  We propose a new distillation method, which contains two transfer distillation strategies and a loss decay strategy. First, Channel Distillation (CD) transfers the channel information from the teacher to the student. Second, Guided Knowledge Distillation (GKD) only enables the student to mimic the correct output of the teacher.
  arXiv  Detail & Related papers  (2020-06-02T14:59:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.