Hidden-Fold Networks: Random Recurrent Residuals Using Sparse Supermasks

• URL: http://arxiv.org/abs/2111.12330v1
• Date: Wed, 24 Nov 2021 08:24:31 GMT
• Title: Hidden-Fold Networks: Random Recurrent Residuals Using Sparse Supermasks
• Authors: \'Angel L\'opez Garc\'ia-Arias, Masanori Hashimoto, Masato Motomura, Jaehoon Yu
• Abstract summary: Deep neural networks (DNNs) are so over-parametrized that recent research has found them to contain a subnetwork with high accuracy. This paper proposes blending these lines of research into a highly compressed yet accurate model: Hidden-Fold Networks (HFNs) It achieves equivalent performance to ResNet50 on CIFAR100 while occupying 38.5x less memory, and similar performance to ResNet34 on ImageNet with a memory size 26.8x smaller.
• Score: 1.0814638303152528
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural networks (DNNs) are so over-parametrized that recent research has found them to already contain a subnetwork with high accuracy at their randomly initialized state. Finding these subnetworks is a viable alternative training method to weight learning. In parallel, another line of work has hypothesized that deep residual networks (ResNets) are trying to approximate the behaviour of shallow recurrent neural networks (RNNs) and has proposed a way for compressing them into recurrent models. This paper proposes blending these lines of research into a highly compressed yet accurate model: Hidden-Fold Networks (HFNs). By first folding ResNet into a recurrent structure and then searching for an accurate subnetwork hidden within the randomly initialized model, a high-performing yet tiny HFN is obtained without ever updating the weights. As a result, HFN achieves equivalent performance to ResNet50 on CIFAR100 while occupying 38.5x less memory, and similar performance to ResNet34 on ImageNet with a memory size 26.8x smaller. The HFN will become even more attractive by minimizing data transfers while staying accurate when it runs on highly-quantized and randomly-weighted DNN inference accelerators. Code available at https://github.com/Lopez-Angel/hidden-fold-networks

Related papers

• Bayesian Inference Accelerator for Spiking Neural Networks [3.145754107337963]
  spiking neural networks (SNNs) have the potential to reduce computational area and power. In this work, we demonstrate an optimization framework for developing and implementing efficient Bayesian SNNs in hardware. We demonstrate accuracies comparable to Bayesian binary networks with full-precision Bernoulli parameters, while requiring up to $25times$ less spikes.
  arXiv  Detail & Related papers  (2024-01-27T16:27:19Z)
• Spikingformer: Spike-driven Residual Learning for Transformer-based Spiking Neural Network [19.932683405796126]
  Spiking neural networks (SNNs) offer a promising energy-efficient alternative to artificial neural networks. SNNs suffer from non-spike computations caused by the structure of their residual connection. We develop Spikingformer, a pure transformer-based spiking neural network.
  arXiv  Detail & Related papers  (2023-04-24T09:44:24Z)
• An Exact Mapping From ReLU Networks to Spiking Neural Networks [3.1701886344065255]
  We propose an exact mapping from a network with Rectified Linear Units (ReLUs) to an SNN that fires exactly one spike per neuron. More generally our work shows that an arbitrary deep ReLU network can be replaced by an energy-efficient single-spike neural network without any loss of performance.
  arXiv  Detail & Related papers  (2022-12-23T18:31:09Z)
• Network Augmentation for Tiny Deep Learning [73.57192520534585]
  We introduce Network Augmentation (NetAug), a new training method for improving the performance of tiny neural networks. We demonstrate the effectiveness of NetAug on image classification and object detection.
  arXiv  Detail & Related papers  (2021-10-17T18:48:41Z)
• Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNets [65.28292822614418]
  Giant formula for simultaneously enlarging the resolution, depth and width provides us a Rubik's cube for neural networks. This paper aims to explore the twisting rules for obtaining deep neural networks with minimum model sizes and computational costs.
  arXiv  Detail & Related papers  (2020-10-28T08:49:45Z)
• ESPN: Extremely Sparse Pruned Networks [50.436905934791035]
  We show that a simple iterative mask discovery method can achieve state-of-the-art compression of very deep networks. Our algorithm represents a hybrid approach between single shot network pruning methods and Lottery-Ticket type approaches.
  arXiv  Detail & Related papers  (2020-06-28T23:09:27Z)
• Principal Component Networks: Parameter Reduction Early in Training [10.14522349959932]
  We show how to find small networks that exhibit the same performance as their over parameterized counterparts. We use PCA to find a basis of high variance for layer inputs and represent layer weights using these directions. We also show that ResNet-20 PCNs outperform deep ResNet-110 networks while training faster.
  arXiv  Detail & Related papers  (2020-06-23T21:40:24Z)
• Efficient Integer-Arithmetic-Only Convolutional Neural Networks [87.01739569518513]
  We replace conventional ReLU with Bounded ReLU and find that the decline is due to activation quantization. Our integer networks achieve equivalent performance as the corresponding FPN networks, but have only 1/4 memory cost and run 2x faster on modern GPU.
  arXiv  Detail & Related papers  (2020-06-21T08:23:03Z)
• DRU-net: An Efficient Deep Convolutional Neural Network for Medical Image Segmentation [2.3574651879602215]
  Residual network (ResNet) and densely connected network (DenseNet) have significantly improved the training efficiency and performance of deep convolutional neural networks (DCNNs) We propose an efficient network architecture by considering advantages of both networks.
  arXiv  Detail & Related papers  (2020-04-28T12:16:24Z)
• Improved Residual Networks for Image and Video Recognition [98.10703825716142]
  Residual networks (ResNets) represent a powerful type of convolutional neural network (CNN) architecture. We show consistent improvements in accuracy and learning convergence over the baseline. Our proposed approach allows us to train extremely deep networks, while the baseline shows severe optimization issues.
  arXiv  Detail & Related papers  (2020-04-10T11:09:50Z)
• Widening and Squeezing: Towards Accurate and Efficient QNNs [125.172220129257]
  Quantization neural networks (QNNs) are very attractive to the industry because their extremely cheap calculation and storage overhead, but their performance is still worse than that of networks with full-precision parameters. Most of existing methods aim to enhance performance of QNNs especially binary neural networks by exploiting more effective training techniques. We address this problem by projecting features in original full-precision networks to high-dimensional quantization features.
  arXiv  Detail & Related papers  (2020-02-03T04:11:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.