A Comprehensive Review of Knowledge Distillation in Computer Vision

• URL: http://arxiv.org/abs/2404.00936v4
• Date: Tue, 23 Jul 2024 17:30:56 GMT
• Title: A Comprehensive Review of Knowledge Distillation in Computer Vision
• Authors: Gousia Habib, Tausifa jan Saleem, Sheikh Musa Kaleem, Tufail Rouf, Brejesh Lall,
• Abstract summary: This review paper examines the current state of research on knowledge distillation, a technique for compressing complex models into smaller and simpler ones. The paper provides an overview of the major principles and techniques associated with knowledge distillation and reviews the applications of knowledge distillation in the domain of computer vision.
• Score: 4.9407806800208816
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep learning techniques have been demonstrated to surpass preceding cutting-edge machine learning techniques in recent years, with computer vision being one of the most prominent examples. However, deep learning models suffer from significant drawbacks when deployed in resource-constrained environments due to their large model size and high complexity. Knowledge Distillation is one of the prominent solutions to overcome this challenge. This review paper examines the current state of research on knowledge distillation, a technique for compressing complex models into smaller and simpler ones. The paper provides an overview of the major principles and techniques associated with knowledge distillation and reviews the applications of knowledge distillation in the domain of computer vision. The review focuses on the benefits of knowledge distillation, as well as the problems that must be overcome to improve its effectiveness.

Related papers

• Can a student Large Language Model perform as well as it's teacher? [0.0]
  Knowledge distillation aims to transfer knowledge from a high-capacity "teacher" model to a streamlined "student" model. This paper provides a comprehensive overview of the knowledge distillation paradigm.
  arXiv  Detail & Related papers  (2023-10-03T20:34:59Z)
• Knowledge Distillation via Token-level Relationship Graph [12.356770685214498]
  We propose a novel method called Knowledge Distillation with Token-level Relationship Graph (TRG) By employing TRG, the student model can effectively emulate higher-level semantic information from the teacher model. We conduct experiments to evaluate the effectiveness of the proposed method against several state-of-the-art approaches.
  arXiv  Detail & Related papers  (2023-06-20T08:16:37Z)
• Machine Unlearning: A Survey [56.79152190680552]
  A special need has arisen where, due to privacy, usability, and/or the right to be forgotten, information about some specific samples needs to be removed from a model, called machine unlearning. This emerging technology has drawn significant interest from both academics and industry due to its innovation and practicality. No study has analyzed this complex topic or compared the feasibility of existing unlearning solutions in different kinds of scenarios. The survey concludes by highlighting some of the outstanding issues with unlearning techniques, along with some feasible directions for new research opportunities.
  arXiv  Detail & Related papers  (2023-06-06T10:18:36Z)
• Knowledge-enhanced Neural Machine Reasoning: A Review [67.51157900655207]
  We introduce a novel taxonomy that categorizes existing knowledge-enhanced methods into two primary categories and four subcategories. We elucidate the current application domains and provide insight into promising prospects for future research.
  arXiv  Detail & Related papers  (2023-02-04T04:54:30Z)
• Deep Learning to See: Towards New Foundations of Computer Vision [88.69805848302266]
  This book criticizes the supposed scientific progress in the field of computer vision. It proposes the investigation of vision within the framework of information-based laws of nature.
  arXiv  Detail & Related papers  (2022-06-30T15:20:36Z)
• A Closer Look at Knowledge Distillation with Features, Logits, and Gradients [81.39206923719455]
  Knowledge distillation (KD) is a substantial strategy for transferring learned knowledge from one neural network model to another. This work provides a new perspective to motivate a set of knowledge distillation strategies by approximating the classical KL-divergence criteria with different knowledge sources. Our analysis indicates that logits are generally a more efficient knowledge source and suggests that having sufficient feature dimensions is crucial for the model design.
  arXiv  Detail & Related papers  (2022-03-18T21:26:55Z)
• Knowledge as Invariance -- History and Perspectives of Knowledge-augmented Machine Learning [69.99522650448213]
  Research in machine learning is at a turning point. Research interests are shifting away from increasing the performance of highly parameterized models to exceedingly specific tasks. This white paper provides an introduction and discussion of this emerging field in machine learning research.
  arXiv  Detail & Related papers  (2020-12-21T15:07:19Z)
• A Selective Survey on Versatile Knowledge Distillation Paradigm for Neural Network Models [3.770437296936382]
  We review the characteristics of knowledge distillation from the hypothesis that the three important ingredients of knowledge distillation are distilled knowledge and loss, teacher-student paradigm, and the distillation process. We present some future works in knowledge distillation including explainable knowledge distillation where the analytical analysis of the performance gain is studied and the self-supervised learning which is a hot research topic in deep learning community.
  arXiv  Detail & Related papers  (2020-11-30T05:22:02Z)
• Knowledge Distillation in Deep Learning and its Applications [0.6875312133832078]
  Deep learning models are relatively large, and it is hard to deploy such models on resource-limited devices. One possible solution is knowledge distillation whereby a smaller model (student model) is trained by utilizing the information from a larger model (teacher model)
  arXiv  Detail & Related papers  (2020-07-17T14:43:52Z)
• Knowledge Distillation: A Survey [87.51063304509067]
  Deep neural networks have been successful in both industry and academia, especially for computer vision tasks. It is a challenge to deploy these cumbersome deep models on devices with limited resources. Knowledge distillation effectively learns a small student model from a large teacher model.
  arXiv  Detail & Related papers  (2020-06-09T21:47:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.