Utilizing Explainable AI for Quantization and Pruning of Deep Neural Networks

• URL: http://arxiv.org/abs/2008.09072v1
• Date: Thu, 20 Aug 2020 16:52:58 GMT
• Title: Utilizing Explainable AI for Quantization and Pruning of Deep Neural Networks
• Authors: Muhammad Sabih, Frank Hannig and Juergen Teich
• Abstract summary: Recent efforts to understand and explain AI (Artificial Intelligence) methods have led to a new research area, termed as explainable AI. Recent efforts to understand and explain AI (Artificial Intelligence) methods have led to a new research area, termed as explainable AI. In this paper, we utilize explainable AI methods: mainly DeepLIFT method.
• Score: 0.495186171543858
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: For many applications, utilizing DNNs (Deep Neural Networks) requires their implementation on a target architecture in an optimized manner concerning energy consumption, memory requirement, throughput, etc. DNN compression is used to reduce the memory footprint and complexity of a DNN before its deployment on hardware. Recent efforts to understand and explain AI (Artificial Intelligence) methods have led to a new research area, termed as explainable AI. Explainable AI methods allow us to understand better the inner working of DNNs, such as the importance of different neurons and features. The concepts from explainable AI provide an opportunity to improve DNN compression methods such as quantization and pruning in several ways that have not been sufficiently explored so far. In this paper, we utilize explainable AI methods: mainly DeepLIFT method. We use these methods for (1) pruning of DNNs; this includes structured and unstructured pruning of \ac{CNN} filters pruning as well as pruning weights of fully connected layers, (2) non-uniform quantization of DNN weights using clustering algorithm; this is also referred to as Weight Sharing, and (3) integer-based mixed-precision quantization; this is where each layer of a DNN may use a different number of integer bits. We use typical image classification datasets with common deep learning image classification models for evaluation. In all these three cases, we demonstrate significant improvements as well as new insights and opportunities from the use of explainable AI in DNN compression.

Related papers

• SimpleMind adds thinking to deep neural networks [3.888848425698769]
  Deep neural networks (DNNs) detect patterns in data and have shown versatility and strong performance in many computer vision applications. DNNs alone are susceptible to obvious mistakes that violate simple, common sense concepts and are limited in their ability to use explicit knowledge to guide their search and decision making. This paper introduces SimpleMind, an open-source software framework for Cognitive AI focused on medical image understanding.
  arXiv  Detail & Related papers  (2022-12-02T03:38:20Z)
• Towards Better Out-of-Distribution Generalization of Neural Algorithmic Reasoning Tasks [51.8723187709964]
  We study the OOD generalization of neural algorithmic reasoning tasks. The goal is to learn an algorithm from input-output pairs using deep neural networks.
  arXiv  Detail & Related papers  (2022-11-01T18:33:20Z)
• Anisotropic, Sparse and Interpretable Physics-Informed Neural Networks for PDEs [0.0]
  We present ASPINN, an anisotropic extension of our earlier work called SPINN--Sparse, Physics-informed, and Interpretable Neural Networks--to solve PDEs. ASPINNs generalize radial basis function networks. We also streamline the training of ASPINNs into a form that is closer to that of supervised learning algorithms.
  arXiv  Detail & Related papers  (2022-07-01T12:24:43Z)
• Sub-bit Neural Networks: Learning to Compress and Accelerate Binary Neural Networks [72.81092567651395]
  Sub-bit Neural Networks (SNNs) are a new type of binary quantization design tailored to compress and accelerate BNNs. SNNs are trained with a kernel-aware optimization framework, which exploits binary quantization in the fine-grained convolutional kernel space. Experiments on visual recognition benchmarks and the hardware deployment on FPGA validate the great potentials of SNNs.
  arXiv  Detail & Related papers  (2021-10-18T11:30:29Z)
• Mining the Weights Knowledge for Optimizing Neural Network Structures [1.995792341399967]
  We introduce a switcher neural network (SNN) that uses as inputs the weights of a task-specific neural network (called TNN for short) By mining the knowledge contained in the weights, the SNN outputs scaling factors for turning off neurons in the TNN. In terms of accuracy, we outperform baseline networks and other structure learning methods stably and significantly.
  arXiv  Detail & Related papers  (2021-10-11T05:20:56Z)
• Fully Spiking Variational Autoencoder [66.58310094608002]
  Spiking neural networks (SNNs) can be run on neuromorphic devices with ultra-high speed and ultra-low energy consumption. In this study, we build a variational autoencoder (VAE) with SNN to enable image generation.
  arXiv  Detail & Related papers  (2021-09-26T06:10:14Z)
• Quantized Neural Networks via {-1, +1} Encoding Decomposition and Acceleration [83.84684675841167]
  We propose a novel encoding scheme using -1, +1 to decompose quantized neural networks (QNNs) into multi-branch binary networks. We validate the effectiveness of our method on large-scale image classification, object detection, and semantic segmentation tasks.
  arXiv  Detail & Related papers  (2021-06-18T03:11:15Z)
• Pruning and Slicing Neural Networks using Formal Verification [0.2538209532048866]
  Deep neural networks (DNNs) play an increasingly important role in various computer systems. In order to create these networks, engineers typically specify a desired topology, and then use an automated training algorithm to select the network's weights. Here, we propose to address this challenge by harnessing recent advances in DNN verification.
  arXiv  Detail & Related papers  (2021-05-28T07:53:50Z)
• A Practical Tutorial on Graph Neural Networks [49.919443059032226]
  Graph neural networks (GNNs) have recently grown in popularity in the field of artificial intelligence (AI) This tutorial exposes the power and novelty of GNNs to AI practitioners.
  arXiv  Detail & Related papers  (2020-10-11T12:36:17Z)
• Boosting Deep Neural Networks with Geometrical Prior Knowledge: A Survey [77.99182201815763]
  Deep Neural Networks (DNNs) achieve state-of-the-art results in many different problem settings. DNNs are often treated as black box systems, which complicates their evaluation and validation. One promising field, inspired by the success of convolutional neural networks (CNNs) in computer vision tasks, is to incorporate knowledge about symmetric geometrical transformations.
  arXiv  Detail & Related papers  (2020-06-30T14:56:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.