Leveraging Topological Guidance for Improved Knowledge Distillation

• URL: http://arxiv.org/abs/2407.05316v1
• Date: Sun, 7 Jul 2024 10:09:18 GMT
• Title: Leveraging Topological Guidance for Improved Knowledge Distillation
• Authors: Eun Som Jeon, Rahul Khurana, Aishani Pathak, Pavan Turaga,
• Abstract summary: We propose a framework called Topological Guidance-based Knowledge Distillation (TGD) for image classification tasks. We utilize KD to train a superior lightweight model and provide topological features with multiple teachers simultaneously. We introduce a mechanism for integrating features from different teachers and reducing the knowledge gap between teachers and the student.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep learning has shown its efficacy in extracting useful features to solve various computer vision tasks. However, when the structure of the data is complex and noisy, capturing effective information to improve performance is very difficult. To this end, topological data analysis (TDA) has been utilized to derive useful representations that can contribute to improving performance and robustness against perturbations. Despite its effectiveness, the requirements for large computational resources and significant time consumption in extracting topological features through TDA are critical problems when implementing it on small devices. To address this issue, we propose a framework called Topological Guidance-based Knowledge Distillation (TGD), which uses topological features in knowledge distillation (KD) for image classification tasks. We utilize KD to train a superior lightweight model and provide topological features with multiple teachers simultaneously. We introduce a mechanism for integrating features from different teachers and reducing the knowledge gap between teachers and the student, which aids in improving performance. We demonstrate the effectiveness of our approach through diverse empirical evaluations.

Related papers

• KBAlign: Efficient Self Adaptation on Specific Knowledge Bases [75.78948575957081]
  Large language models (LLMs) usually rely on retrieval-augmented generation to exploit knowledge materials in an instant manner. We propose KBAlign, an approach designed for efficient adaptation to downstream tasks involving knowledge bases. Our method utilizes iterative training with self-annotated data such as Q&A pairs and revision suggestions, enabling the model to grasp the knowledge content efficiently.
  arXiv  Detail & Related papers  (2024-11-22T08:21:03Z)
• Topological Persistence Guided Knowledge Distillation for Wearable Sensor Data [15.326571438985466]
  topological features obtained by topological data analysis (TDA) have been suggested as a potential solution. There are two significant obstacles to using topological features in deep learning. We propose to use two teacher networks, one trained on the raw time-series data, and another trained on persistence images generated by TDA methods. A robust student model is distilled, which uses only the time-series data as an input, while implicitly preserving topological features.
  arXiv  Detail & Related papers  (2024-07-07T10:08:34Z)
• Relative Difficulty Distillation for Semantic Segmentation [54.76143187709987]
  We propose a pixel-level KD paradigm for semantic segmentation named Relative Difficulty Distillation (RDD) RDD allows the teacher network to provide effective guidance on learning focus without additional optimization goals. Our research showcases that RDD can integrate with existing KD methods to improve their upper performance bound.
  arXiv  Detail & Related papers  (2024-07-04T08:08:25Z)
• Exploring Graph-based Knowledge: Multi-Level Feature Distillation via Channels Relational Graph [8.646512035461994]
  In visual tasks, large teacher models capture essential features and deep information, enhancing performance. We propose a distillation framework based on graph knowledge, including a multi-level feature alignment strategy. We emphasize spectral embedding (SE) as a key technique in our distillation process, which merges the student's feature space with the relational knowledge and structural complexities similar to the teacher network.
  arXiv  Detail & Related papers  (2024-05-14T12:37:05Z)
• Explaining the Power of Topological Data Analysis in Graph Machine Learning [6.2340401953289275]
  Topological Data Analysis (TDA) has been praised by researchers for its ability to capture intricate shapes and structures within data. We meticulously test claims on TDA through a comprehensive set of experiments and validate their merits. We find that TDA does not significantly enhance the predictive power of existing methods in our specific experiments, while incurring significant computational costs.
  arXiv  Detail & Related papers  (2024-01-08T21:47:35Z)
• Knowledge Diffusion for Distillation [53.908314960324915]
  The representation gap between teacher and student is an emerging topic in knowledge distillation (KD) We state that the essence of these methods is to discard the noisy information and distill the valuable information in the feature. We propose a novel KD method dubbed DiffKD, to explicitly denoise and match features using diffusion models.
  arXiv  Detail & Related papers  (2023-05-25T04:49:34Z)
• GIF: A General Graph Unlearning Strategy via Influence Function [63.52038638220563]
  Graph Influence Function (GIF) is a model-agnostic unlearning method that can efficiently and accurately estimate parameter changes in response to a $epsilon$-mass perturbation in deleted data. We conduct extensive experiments on four representative GNN models and three benchmark datasets to justify GIF's superiority in terms of unlearning efficacy, model utility, and unlearning efficiency.
  arXiv  Detail & Related papers  (2023-04-06T03:02:54Z)
• Directed Acyclic Graph Factorization Machines for CTR Prediction via Knowledge Distillation [65.62538699160085]
  We propose a Directed Acyclic Graph Factorization Machine (KD-DAGFM) to learn the high-order feature interactions from existing complex interaction models for CTR prediction via Knowledge Distillation. KD-DAGFM achieves the best performance with less than 21.5% FLOPs of the state-of-the-art method on both online and offline experiments.
  arXiv  Detail & Related papers  (2022-11-21T03:09:42Z)
• On effects of Knowledge Distillation on Transfer Learning [0.0]
  We propose a machine learning architecture we call TL+KD that combines knowledge distillation with transfer learning. We show that using guidance and knowledge from a larger teacher network during fine-tuning, we can improve the student network to achieve better validation performances like accuracy.
  arXiv  Detail & Related papers  (2022-10-18T08:11:52Z)
• Learning Knowledge Representation with Meta Knowledge Distillation for Single Image Super-Resolution [82.89021683451432]
  We propose a model-agnostic meta knowledge distillation method under the teacher-student architecture for the single image super-resolution task. Experiments conducted on various single image super-resolution datasets demonstrate that our proposed method outperforms existing defined knowledge representation related distillation methods.
  arXiv  Detail & Related papers  (2022-07-18T02:41:04Z)
• An Empirical Comparison of Deep Learning Models for Knowledge Tracing on Large-Scale Dataset [10.329254031835953]
  Knowledge tracing is a problem of modeling each student's mastery of knowledge concepts. Recent release of large-scale student performance dataset citechoi 2019ednet motivates the analysis of performance of deep learning approaches.
  arXiv  Detail & Related papers  (2021-01-16T04:58:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.