Growing Artificial Neural Networks

• URL: http://arxiv.org/abs/2006.06629v1
• Date: Thu, 11 Jun 2020 17:25:51 GMT
• Title: Growing Artificial Neural Networks
• Authors: John Mixter and Ali Akoglu
• Abstract summary: Pruning is a legitimate method for reducing the size of a neural network to fit in low SWaP hardware. We propose an algorithm, Artificial Neurogenesis (ANG), that grows rather than prunes the network. ANG accomplishes this by using the training data to determine critical connections between layers before the actual training takes place.
• Score: 0.9475982252982436
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pruning is a legitimate method for reducing the size of a neural network to fit in low SWaP hardware, but the networks must be trained and pruned offline. We propose an algorithm, Artificial Neurogenesis (ANG), that grows rather than prunes the network and enables neural networks to be trained and executed in low SWaP embedded hardware. ANG accomplishes this by using the training data to determine critical connections between layers before the actual training takes place. Our experiments use a modified LeNet-5 as a baseline neural network that achieves a test accuracy of 98.74% using a total of 61,160 weights. An ANG grown network achieves a test accuracy of 98.80% with only 21,211 weights.

Related papers

• Verified Neural Compressed Sensing [58.98637799432153]
  We develop the first (to the best of our knowledge) provably correct neural networks for a precise computational task. We show that for modest problem dimensions (up to 50), we can train neural networks that provably recover a sparse vector from linear and binarized linear measurements. We show that the complexity of the network can be adapted to the problem difficulty and solve problems where traditional compressed sensing methods are not known to provably work.
  arXiv  Detail & Related papers  (2024-05-07T12:20:12Z)
• Set-Based Training for Neural Network Verification [8.97708612393722]
  Small input perturbations can significantly affect the outputs of a neural network. In safety-critical environments, the inputs often contain noisy sensor data. We employ an end-to-end set-based training procedure that trains robust neural networks for formal verification.
  arXiv  Detail & Related papers  (2024-01-26T15:52:41Z)
• Adaptive Neural Networks Using Residual Fitting [2.546014024559691]
  We present a network-growth method that searches for explainable error in the network's residuals and grows the network if sufficient error is detected. Within these tasks, the growing network can often achieve better performance than small networks that do not grow.
  arXiv  Detail & Related papers  (2023-01-13T19:52:30Z)
• Training Spiking Neural Networks with Local Tandem Learning [96.32026780517097]
  Spiking neural networks (SNNs) are shown to be more biologically plausible and energy efficient than their predecessors. In this paper, we put forward a generalized learning rule, termed Local Tandem Learning (LTL) We demonstrate rapid network convergence within five training epochs on the CIFAR-10 dataset while having low computational complexity.
  arXiv  Detail & Related papers  (2022-10-10T10:05:00Z)
• Neural Capacitance: A New Perspective of Neural Network Selection via Edge Dynamics [85.31710759801705]
  Current practice requires expensive computational costs in model training for performance prediction. We propose a novel framework for neural network selection by analyzing the governing dynamics over synaptic connections (edges) during training. Our framework is built on the fact that back-propagation during neural network training is equivalent to the dynamical evolution of synaptic connections.
  arXiv  Detail & Related papers  (2022-01-11T20:53:15Z)
• Adder Neural Networks [75.54239599016535]
  We present adder networks (AdderNets) to trade massive multiplications in deep neural networks. In AdderNets, we take the $ell_p$-norm distance between filters and input feature as the output response. We show that the proposed AdderNets can achieve 75.7% Top-1 accuracy 92.3% Top-5 accuracy using ResNet-50 on the ImageNet dataset.
  arXiv  Detail & Related papers  (2021-05-29T04:02:51Z)
• Post-training deep neural network pruning via layer-wise calibration [70.65691136625514]
  We propose a data-free extension of the approach for computer vision models based on automatically-generated synthetic fractal images. When using real data, we are able to get a ResNet50 model on ImageNet with 65% sparsity rate in 8-bit precision in a post-training setting.
  arXiv  Detail & Related papers  (2021-04-30T14:20:51Z)
• Bayesian Neural Networks at Scale: A Performance Analysis and Pruning Study [2.3605348648054463]
  This work explores the use of high performance computing with distributed training to address the challenges of training BNNs at scale. We present a performance and scalability comparison of training the VGG-16 and Resnet-18 models on a Cray-XC40 cluster.
  arXiv  Detail & Related papers  (2020-05-23T23:15:34Z)
• A Hybrid Method for Training Convolutional Neural Networks [3.172761915061083]
  We propose a hybrid method that uses both backpropagation and evolutionary strategies to train Convolutional Neural Networks. We show that the proposed hybrid method is capable of improving upon regular training in the task of image classification.
  arXiv  Detail & Related papers  (2020-04-15T17:52:48Z)
• Lossless Compression of Deep Neural Networks [17.753357839478575]
  Deep neural networks have been successful in many predictive modeling tasks, such as image and language recognition. It is challenging to deploy these networks under limited computational resources, such as in mobile devices. We introduce an algorithm that removes units and layers of a neural network while not changing the output that is produced.
  arXiv  Detail & Related papers  (2020-01-01T15:04:43Z)
• AdderNet: Do We Really Need Multiplications in Deep Learning? [159.174891462064]
  We present adder networks (AdderNets) to trade massive multiplications in deep neural networks for much cheaper additions to reduce computation costs. We develop a special back-propagation approach for AdderNets by investigating the full-precision gradient. As a result, the proposed AdderNets can achieve 74.9% Top-1 accuracy 91.7% Top-5 accuracy using ResNet-50 on the ImageNet dataset.
  arXiv  Detail & Related papers  (2019-12-31T06:56:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.