Distilling a Deep Neural Network into a Takagi-Sugeno-Kang Fuzzy Inference System

• URL: http://arxiv.org/abs/2010.04974v1
• Date: Sat, 10 Oct 2020 10:58:05 GMT
• Title: Distilling a Deep Neural Network into a Takagi-Sugeno-Kang Fuzzy Inference System
• Authors: Xiangming Gu and Xiang Cheng
• Abstract summary: Deep neural networks (DNNs) demonstrate great success in classification tasks. However, they act as black boxes and we don't know how they make decisions in a particular classification task. We propose to distill the knowledge from a DNN into a fuzzy inference system (FIS), which is Takagi-Sugeno-Kang (TSK)-type in this paper.
• Score: 9.82399898215447
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural networks (DNNs) demonstrate great success in classification tasks. However, they act as black boxes and we don't know how they make decisions in a particular classification task. To this end, we propose to distill the knowledge from a DNN into a fuzzy inference system (FIS), which is Takagi-Sugeno-Kang (TSK)-type in this paper. The model has the capability to express the knowledge acquired by a DNN based on fuzzy rules, thus explaining a particular decision much easier. Knowledge distillation (KD) is applied to create a TSK-type FIS that generalizes better than one directly from the training data, which is guaranteed through experiments in this paper. To further improve the performances, we modify the baseline method of KD and obtain good results.

Related papers

• BKDSNN: Enhancing the Performance of Learning-based Spiking Neural Networks Training with Blurred Knowledge Distillation [20.34272550256856]
  Spiking neural networks (SNNs) mimic biological neural system to convey information via discrete spikes. Our work achieves state-of-the-art performance for training SNNs on both static and neuromorphic datasets.
  arXiv  Detail & Related papers  (2024-07-12T08:17:24Z)
• DeepKnowledge: Generalisation-Driven Deep Learning Testing [2.526146573337397]
  DeepKnowledge is a systematic testing methodology for DNN-based systems. It aims to enhance robustness and reduce the residual risk of 'black box' models. We report improvements of up to 10 percentage points over state-of-the-art coverage criteria for detecting adversarial attacks.
  arXiv  Detail & Related papers  (2024-03-25T13:46:09Z)
• Bayesian Neural Networks with Domain Knowledge Priors [52.80929437592308]
  We propose a framework for integrating general forms of domain knowledge into a BNN prior. We show that BNNs using our proposed domain knowledge priors outperform those with standard priors.
  arXiv  Detail & Related papers  (2024-02-20T22:34:53Z)
• Shared Growth of Graph Neural Networks via Prompted Free-direction Knowledge Distillation [39.35619721100205]
  We propose the first Free-direction Knowledge Distillation framework via reinforcement learning for graph neural networks (GNNs) Our core idea is to collaboratively learn two shallower GNNs to exchange knowledge between them. Experiments on five benchmark datasets demonstrate our approaches outperform the base GNNs in a large margin.
  arXiv  Detail & Related papers  (2023-07-02T10:03:01Z)
• FreeKD: Free-direction Knowledge Distillation for Graph Neural Networks [31.980564414833175]
  It is difficult to train a satisfactory teacher GNN due to the well-known over-parametrized and over-smoothing issues. We propose the first Free-direction Knowledge Distillation framework via Reinforcement learning for GNNs, called FreeKD. Our FreeKD is a general and principled framework which can be naturally compatible with GNNs of different architectures.
  arXiv  Detail & Related papers  (2022-06-14T02:24:38Z)
• Knowledge Enhanced Neural Networks for relational domains [83.9217787335878]
  We focus on a specific method, KENN, a Neural-Symbolic architecture that injects prior logical knowledge into a neural network. In this paper, we propose an extension of KENN for relational data.
  arXiv  Detail & Related papers  (2022-05-31T13:00:34Z)
• KDExplainer: A Task-oriented Attention Model for Explaining Knowledge Distillation [59.061835562314066]
  We introduce a novel task-oriented attention model, termed as KDExplainer, to shed light on the working mechanism underlying the vanilla KD. We also introduce a portable tool, dubbed as virtual attention module (VAM), that can be seamlessly integrated with various deep neural networks (DNNs) to enhance their performance under KD.
  arXiv  Detail & Related papers  (2021-05-10T08:15:26Z)
• Interpretable Embedding Procedure Knowledge Transfer via Stacked Principal Component Analysis and Graph Neural Network [26.55774782646948]
  This paper proposes a method of generating interpretable embedding procedure (IEP) knowledge based on principal component analysis. Experimental results show that the student network trained by the proposed KD method improves 2.28% in the CIFAR100 dataset. We also demonstrate that the embedding procedure knowledge is interpretable via visualization of the proposed KD process.
  arXiv  Detail & Related papers  (2021-04-28T03:40:37Z)
• On Self-Distilling Graph Neural Network [64.00508355508106]
  We propose the first teacher-free knowledge distillation method for GNNs, termed GNN Self-Distillation (GNN-SD) The method is built upon the proposed neighborhood discrepancy rate (NDR), which quantifies the non-smoothness of the embedded graph in an efficient way. We also summarize a generic GNN-SD framework that could be exploited to induce other distillation strategies.
  arXiv  Detail & Related papers  (2020-11-04T12:29:33Z)
• Boosting Deep Neural Networks with Geometrical Prior Knowledge: A Survey [77.99182201815763]
  Deep Neural Networks (DNNs) achieve state-of-the-art results in many different problem settings. DNNs are often treated as black box systems, which complicates their evaluation and validation. One promising field, inspired by the success of convolutional neural networks (CNNs) in computer vision tasks, is to incorporate knowledge about symmetric geometrical transformations.
  arXiv  Detail & Related papers  (2020-06-30T14:56:05Z)
• COLAM: Co-Learning of Deep Neural Networks and Soft Labels via Alternating Minimization [60.07531696857743]
  Co-Learns DNNs and soft labels through Alternating Minimization of two objectives. We propose COLAM framework that Co-Learns DNNs and soft labels through Alternating Minimization of two objectives.
  arXiv  Detail & Related papers  (2020-04-26T17:50:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.