UNIC: Universal Classification Models via Multi-teacher Distillation

• URL: http://arxiv.org/abs/2408.05088v1
• Date: Fri, 9 Aug 2024 14:18:57 GMT
• Title: UNIC: Universal Classification Models via Multi-teacher Distillation
• Authors: Mert Bulent Sariyildiz, Philippe Weinzaepfel, Thomas Lucas, Diane Larlus, Yannis Kalantidis,
• Abstract summary: We seek to learn a unique encoder able to take from several complementary pretrained models. We propose to learn such an encoder via multi-teacher distillation.
• Score: 29.299698704883813
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pretrained models have become a commodity and offer strong results on a broad range of tasks. In this work, we focus on classification and seek to learn a unique encoder able to take from several complementary pretrained models. We aim at even stronger generalization across a variety of classification tasks. We propose to learn such an encoder via multi-teacher distillation. We first thoroughly analyse standard distillation when driven by multiple strong teachers with complementary strengths. Guided by this analysis, we gradually propose improvements to the basic distillation setup. Among those, we enrich the architecture of the encoder with a ladder of expendable projectors, which increases the impact of intermediate features during distillation, and we introduce teacher dropping, a regularization mechanism that better balances the teachers' influence. Our final distillation strategy leads to student models of the same capacity as any of the teachers, while retaining or improving upon the performance of the best teacher for each task. Project page and code: https://europe.naverlabs.com/unic

Related papers

• HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers [49.79405257763856]
  This paper focuses on task-agnostic distillation. It produces a compact pre-trained model that can be easily fine-tuned on various tasks with small computational costs and memory footprints. We propose Homotopic Distillation (HomoDistil), a novel task-agnostic distillation approach equipped with iterative pruning.
  arXiv  Detail & Related papers  (2023-02-19T17:37:24Z)
• Supervision Complexity and its Role in Knowledge Distillation [65.07910515406209]
  We study the generalization behavior of a distilled student. The framework highlights a delicate interplay among the teacher's accuracy, the student's margin with respect to the teacher predictions, and the complexity of the teacher predictions. We demonstrate efficacy of online distillation and validate the theoretical findings on a range of image classification benchmarks and model architectures.
  arXiv  Detail & Related papers  (2023-01-28T16:34:47Z)
• Efficient Knowledge Distillation from Model Checkpoints [36.329429655242535]
  We show that a weak snapshot ensemble of several intermediate models from a same training trajectory can outperform a strong ensemble of independently trained and fully converged models. We propose an optimal intermediate teacher selection algorithm based on maximizing the total task-related mutual information.
  arXiv  Detail & Related papers  (2022-10-12T17:55:30Z)
• Knowledge Distillation with the Reused Teacher Classifier [31.22117343316628]
  We show that a simple knowledge distillation technique is enough to significantly narrow down the teacher-student performance gap. Our technique achieves state-of-the-art results at the modest cost of compression ratio due to the added projector.
  arXiv  Detail & Related papers  (2022-03-26T06:28:46Z)
• Delta Distillation for Efficient Video Processing [68.81730245303591]
  We propose a novel knowledge distillation schema coined as Delta Distillation. We demonstrate that these temporal variations can be effectively distilled due to the temporal redundancies within video frames. As a by-product, delta distillation improves the temporal consistency of the teacher model.
  arXiv  Detail & Related papers  (2022-03-17T20:13:30Z)
• Representation Consolidation for Training Expert Students [54.90754502493968]
  We show that a multi-head, multi-task distillation method is sufficient to consolidate representations from task-specific teacher(s) and improve downstream performance. Our method can also combine the representational knowledge of multiple teachers trained on one or multiple domains into a single model.
  arXiv  Detail & Related papers  (2021-07-16T17:58:18Z)
• Teacher's pet: understanding and mitigating biases in distillation [61.44867470297283]
  Several works have shown that distillation significantly boosts the student's overall performance. However, are these gains uniform across all data subgroups? We show that distillation can harm performance on certain subgroups. We present techniques which soften the teacher influence for subgroups where it is less reliable.
  arXiv  Detail & Related papers  (2021-06-19T13:06:25Z)
• Distilling Knowledge via Intermediate Classifier Heads [0.5584060970507505]
  Knowledge distillation is a transfer-learning approach to train a resource-limited student model with the guide of a pre-trained larger teacher model. We introduce knowledge distillation via intermediate heads to mitigate the impact of the capacity gap. Our experiments on various teacher-student pairs and datasets have demonstrated that the proposed approach outperforms the canonical knowledge distillation approach.
  arXiv  Detail & Related papers  (2021-02-28T12:52:52Z)
• Reinforced Multi-Teacher Selection for Knowledge Distillation [54.72886763796232]
  knowledge distillation is a popular method for model compression. Current methods assign a fixed weight to a teacher model in the whole distillation. Most of the existing methods allocate an equal weight to every teacher model. In this paper, we observe that, due to the complexity of training examples and the differences in student model capability, learning differentially from teacher models can lead to better performance of student models distilled.
  arXiv  Detail & Related papers  (2020-12-11T08:56:39Z)
• Cascaded channel pruning using hierarchical self-distillation [26.498907514590165]
  We propose an approach for filter-level pruning with hierarchical knowledge distillation based on the teacher, teaching-assistant, and student framework. Our method makes use of teaching assistants at intermediate pruning levels that share the same architecture and weights as the target student.
  arXiv  Detail & Related papers  (2020-08-16T00:19:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.