Collaborative Knowledge Distillation via a Learning-by-Education Node Community

• URL: http://arxiv.org/abs/2410.00074v1
• Date: Mon, 30 Sep 2024 14:22:28 GMT
• Title: Collaborative Knowledge Distillation via a Learning-by-Education Node Community
• Authors: Anestis Kaimakamidis, Ioannis Mademlis, Ioannis Pitas,
• Abstract summary: Learning-by-Education Node Community framework (LENC) for Collaborative Knowledge Distillation (CKD) is presented. LENC addresses the challenges of handling diverse training data distributions and the limitations of individual Deep Neural Network (DNN) node learning abilities. It achieves state-of-the-art performance in on-line unlabelled CKD.
• Score: 19.54023115706067
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: A novel Learning-by-Education Node Community framework (LENC) for Collaborative Knowledge Distillation (CKD) is presented, which facilitates continual collective learning through effective knowledge exchanges among diverse deployed Deep Neural Network (DNN) peer nodes. These DNNs dynamically and autonomously adopt either the role of a student, seeking knowledge, or that of a teacher, imparting knowledge, fostering a collaborative learning environment. The proposed framework enables efficient knowledge transfer among participating DNN nodes as needed, while enhancing their learning capabilities and promoting their collaboration. LENC addresses the challenges of handling diverse training data distributions and the limitations of individual DNN node learning abilities. It ensures the exploitation of the best available teacher knowledge upon learning a new task and protects the DNN nodes from catastrophic forgetting. Additionally, it innovates by enabling collaborative multitask knowledge distillation, while addressing the problem of task-agnostic continual learning, as DNN nodes have no information on task boundaries. Experimental evaluation on a proof-of-concept implementation demonstrates the LENC framework's functionalities and benefits across multiple DNN learning and inference scenarios. The conducted experiments showcase its ability to gradually maximize the average test accuracy of the community of interacting DNN nodes in image classification problems, by appropriately leveraging the collective knowledge of all node peers. The LENC framework achieves state-of-the-art performance in on-line unlabelled CKD.

Related papers

• DeepKnowledge: Generalisation-Driven Deep Learning Testing [2.526146573337397]
  DeepKnowledge is a systematic testing methodology for DNN-based systems. It aims to enhance robustness and reduce the residual risk of 'black box' models. We report improvements of up to 10 percentage points over state-of-the-art coverage criteria for detecting adversarial attacks.
  arXiv  Detail & Related papers  (2024-03-25T13:46:09Z)
• Enhancing Efficient Continual Learning with Dynamic Structure Development of Spiking Neural Networks [6.407825206595442]
  Children possess the ability to learn multiple cognitive tasks sequentially. Existing continual learning frameworks are usually applicable to Deep Neural Networks (DNNs) We propose Dynamic Structure Development of Spiking Neural Networks (DSD-SNN) for efficient and adaptive continual learning.
  arXiv  Detail & Related papers  (2023-08-09T07:36:40Z)
• Shared Growth of Graph Neural Networks via Prompted Free-direction Knowledge Distillation [39.35619721100205]
  We propose the first Free-direction Knowledge Distillation framework via reinforcement learning for graph neural networks (GNNs) Our core idea is to collaboratively learn two shallower GNNs to exchange knowledge between them. Experiments on five benchmark datasets demonstrate our approaches outperform the base GNNs in a large margin.
  arXiv  Detail & Related papers  (2023-07-02T10:03:01Z)
• Boosting Graph Neural Networks via Adaptive Knowledge Distillation [18.651451228086643]
  Graph neural networks (GNNs) have shown remarkable performance on diverse graph mining tasks. Knowledge distillation (KD) is developed to combine the diverse knowledge from multiple models. We propose a novel adaptive KD framework, called BGNN, which sequentially transfers knowledge from multiple GNNs into a student GNN.
  arXiv  Detail & Related papers  (2022-10-12T04:48:50Z)
• Continual Prune-and-Select: Class-incremental learning with specialized subnetworks [66.4795381419701]
  Continual-Prune-and-Select (CP&S) is capable of sequentially learning 10 tasks from ImageNet-1000 keeping an accuracy around 94% with negligible forgetting. This is a first-of-its-kind result in class-incremental learning.
  arXiv  Detail & Related papers  (2022-08-09T10:49:40Z)
• Robust Knowledge Adaptation for Dynamic Graph Neural Networks [61.8505228728726]
  We propose Ada-DyGNN: a robust knowledge Adaptation framework via reinforcement learning for Dynamic Graph Neural Networks. Our approach constitutes the first attempt to explore robust knowledge adaptation via reinforcement learning. Experiments on three benchmark datasets demonstrate that Ada-DyGNN achieves the state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-07-22T02:06:53Z)
• FreeKD: Free-direction Knowledge Distillation for Graph Neural Networks [31.980564414833175]
  It is difficult to train a satisfactory teacher GNN due to the well-known over-parametrized and over-smoothing issues. We propose the first Free-direction Knowledge Distillation framework via Reinforcement learning for GNNs, called FreeKD. Our FreeKD is a general and principled framework which can be naturally compatible with GNNs of different architectures.
  arXiv  Detail & Related papers  (2022-06-14T02:24:38Z)
• Knowledge Enhanced Neural Networks for relational domains [83.9217787335878]
  We focus on a specific method, KENN, a Neural-Symbolic architecture that injects prior logical knowledge into a neural network. In this paper, we propose an extension of KENN for relational data.
  arXiv  Detail & Related papers  (2022-05-31T13:00:34Z)
• Node2Seq: Towards Trainable Convolutions in Graph Neural Networks [59.378148590027735]
  We propose a graph network layer, known as Node2Seq, to learn node embeddings with explicitly trainable weights for different neighboring nodes. For a target node, our method sorts its neighboring nodes via attention mechanism and then employs 1D convolutional neural networks (CNNs) to enable explicit weights for information aggregation. In addition, we propose to incorporate non-local information for feature learning in an adaptive manner based on the attention scores.
  arXiv  Detail & Related papers  (2021-01-06T03:05:37Z)
• Neural Networks Enhancement with Logical Knowledge [83.9217787335878]
  We propose an extension of KENN for relational data. The results show that KENN is capable of increasing the performances of the underlying neural network even in the presence relational data.
  arXiv  Detail & Related papers  (2020-09-13T21:12:20Z)
• DEPARA: Deep Attribution Graph for Deep Knowledge Transferability [91.06106524522237]
  We propose the DEeP Attribution gRAph (DEPARA) to investigate the transferability of knowledge learned from PR-DNNs. In DEPARA, nodes correspond to the inputs and are represented by their vectorized attribution maps with regards to the outputs of the PR-DNN. The knowledge transferability of two PR-DNNs is measured by the similarity of their corresponding DEPARAs.
  arXiv  Detail & Related papers  (2020-03-17T02:07:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.