Lossless Compression of Deep Neural Networks

• URL: http://arxiv.org/abs/2001.00218v3
• Date: Sat, 22 Feb 2020 16:09:43 GMT
• Title: Lossless Compression of Deep Neural Networks
• Authors: Thiago Serra, Abhinav Kumar, Srikumar Ramalingam
• Abstract summary: Deep neural networks have been successful in many predictive modeling tasks, such as image and language recognition. It is challenging to deploy these networks under limited computational resources, such as in mobile devices. We introduce an algorithm that removes units and layers of a neural network while not changing the output that is produced.
• Score: 17.753357839478575
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural networks have been successful in many predictive modeling tasks, such as image and language recognition, where large neural networks are often used to obtain good accuracy. Consequently, it is challenging to deploy these networks under limited computational resources, such as in mobile devices. In this work, we introduce an algorithm that removes units and layers of a neural network while not changing the output that is produced, which thus implies a lossless compression. This algorithm, which we denote as LEO (Lossless Expressiveness Optimization), relies on Mixed-Integer Linear Programming (MILP) to identify Rectified Linear Units (ReLUs) with linear behavior over the input domain. By using L1 regularization to induce such behavior, we can benefit from training over a larger architecture than we would later use in the environment where the trained neural network is deployed.

Related papers

• Erasure Coded Neural Network Inference via Fisher Averaging [28.243239815823205]
  Erasure-coded computing has been successfully used in cloud systems to reduce tail latency caused by factors such as straggling servers and heterogeneous traffic variations. We design a method to code over neural networks, that is, given two or more neural network models, how to construct a coded model whose output is a linear combination of the outputs of the given neural networks. We conduct experiments to perform erasure coding over neural networks trained on real-world vision datasets and show that the accuracy of the decoded outputs using COIN is significantly higher than other baselines.
  arXiv  Detail & Related papers  (2024-09-02T18:46:26Z)
• Graph Neural Networks for Learning Equivariant Representations of Neural Networks [55.04145324152541]
  We propose to represent neural networks as computational graphs of parameters. Our approach enables a single model to encode neural computational graphs with diverse architectures. We showcase the effectiveness of our method on a wide range of tasks, including classification and editing of implicit neural representations.
  arXiv  Detail & Related papers  (2024-03-18T18:01:01Z)
• When Deep Learning Meets Polyhedral Theory: A Survey [6.899761345257773]
  In the past decade, deep became the prevalent methodology for predictive modeling thanks to the remarkable accuracy of deep neural learning. Meanwhile, the structure of neural networks converged back to simplerwise and linear functions.
  arXiv  Detail & Related papers  (2023-04-29T11:46:53Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• Variable Bitrate Neural Fields [75.24672452527795]
  We present a dictionary method for compressing feature grids, reducing their memory consumption by up to 100x. We formulate the dictionary optimization as a vector-quantized auto-decoder problem which lets us learn end-to-end discrete neural representations in a space where no direct supervision is available.
  arXiv  Detail & Related papers  (2022-06-15T17:58:34Z)
• Predictive Coding: Towards a Future of Deep Learning beyond Backpropagation? [41.58529335439799]
  The backpropagation of error algorithm used to train deep neural networks has been fundamental to the successes of deep learning. Recent work has developed the idea into a general-purpose algorithm able to train neural networks using only local computations. We show the substantially greater flexibility of predictive coding networks against equivalent deep neural networks.
  arXiv  Detail & Related papers  (2022-02-18T22:57:03Z)
• Quantized Neural Networks via {-1, +1} Encoding Decomposition and Acceleration [83.84684675841167]
  We propose a novel encoding scheme using -1, +1 to decompose quantized neural networks (QNNs) into multi-branch binary networks. We validate the effectiveness of our method on large-scale image classification, object detection, and semantic segmentation tasks.
  arXiv  Detail & Related papers  (2021-06-18T03:11:15Z)
• Reduced-Order Neural Network Synthesis with Robustness Guarantees [0.0]
  Machine learning algorithms are being adapted to run locally on board, potentially hardware limited, devices to improve user privacy, reduce latency and be more energy efficient. To address this issue, a method to automatically synthesize reduced-order neural networks (having fewer neurons) approxing the input/output mapping of a larger one is introduced. Worst-case bounds for this approximation error are obtained and the approach can be applied to a wide variety of neural networks architectures.
  arXiv  Detail & Related papers  (2021-02-18T12:03:57Z)
• Robust error bounds for quantised and pruned neural networks [1.8083503268672914]
  Machine learning algorithms are moving towards decentralisation with the data and algorithms stored, and even trained, locally on devices. The device hardware becomes the main bottleneck for model capability in this set-up, creating a need for slimmed down, more efficient neural networks. A semi-definite program is introduced to bound the worst-case error caused by pruning or quantising a neural network. It is hoped that the computed bounds will provide certainty to the performance of these algorithms when deployed on safety-critical systems.
  arXiv  Detail & Related papers  (2020-11-30T22:19:44Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)
• Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment [84.57874289554839]
  Training deep neural networks on large-scale datasets requires significant hardware resources. Backpropagation, the workhorse for training these networks, is an inherently sequential process that is difficult to parallelize. We propose a neuro-biologically-plausible alternative to backprop that can be used to train deep networks.
  arXiv  Detail & Related papers  (2020-02-10T16:20:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.