Depthwise Multiception Convolution for Reducing Network Parameters without Sacrificing Accuracy

• URL: http://arxiv.org/abs/2011.03701v1
• Date: Sat, 7 Nov 2020 05:33:54 GMT
• Title: Depthwise Multiception Convolution for Reducing Network Parameters without Sacrificing Accuracy
• Authors: Guoqing Bao, Manuel B. Graeber and Xiuying Wang
• Abstract summary: Multiception convolution introduces layer-wise multiscale kernels to learn representations of all individual input channels simultaneously. It significantly reduces the number of parameters of standard convolution-based models by 32.48% on average while still preserving accuracy.
• Score: 2.0088802641040604
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep convolutional neural networks have been proven successful in multiple benchmark challenges in recent years. However, the performance improvements are heavily reliant on increasingly complex network architecture and a high number of parameters, which require ever increasing amounts of storage and memory capacity. Depthwise separable convolution (DSConv) can effectively reduce the number of required parameters through decoupling standard convolution into spatial and cross-channel convolution steps. However, the method causes a degradation of accuracy. To address this problem, we present depthwise multiception convolution, termed Multiception, which introduces layer-wise multiscale kernels to learn multiscale representations of all individual input channels simultaneously. We have carried out the experiment on four benchmark datasets, i.e. Cifar-10, Cifar-100, STL-10 and ImageNet32x32, using five popular CNN models, Multiception achieved accuracy promotion in all models and demonstrated higher accuracy performance compared to related works. Meanwhile, Multiception significantly reduces the number of parameters of standard convolution-based models by 32.48% on average while still preserving accuracy.

Related papers

• Neural Networks at a Fraction with Pruned Quaternions [0.0]
  Pruning is one technique to remove unnecessary weights and reduce resource requirements for training and inference. For ML tasks where the input data is multi-dimensional, using higher-dimensional data embeddings such as complex numbers or quaternions has been shown to reduce the parameter count while maintaining accuracy. We find that for some architectures, at very high sparsity levels, quaternion models provide higher accuracies than their real counterparts.
  arXiv  Detail & Related papers  (2023-08-13T14:25:54Z)
• Sequence Modeling with Multiresolution Convolutional Memory [27.218134279968062]
  We build a new building block for sequence modeling called a MultiresLayer. The key component of our model is the multiresolution convolution, capturing multiscale trends in the input sequence. Our model yields state-of-the-art performance on a number of sequence classification and autoregressive density estimation tasks.
  arXiv  Detail & Related papers  (2023-05-02T17:50:54Z)
• Edge Inference with Fully Differentiable Quantized Mixed Precision Neural Networks [1.131071436917293]
  Quantizing parameters and operations to lower bit-precision offers substantial memory and energy savings for neural network inference. This paper proposes a new quantization approach for mixed precision convolutional neural networks (CNNs) targeting edge-computing.
  arXiv  Detail & Related papers  (2022-06-15T18:11:37Z)
• Residual-Concatenate Neural Network with Deep Regularization Layers for Binary Classification [3.1871776847712523]
  We train a deep neural network that uses many regularization layers with residual and concatenation process for best fit with Polycystic Ovary Syndrome Diagnosis prognostication. The network was built with improvements from every step of failure to meet the needs of the data and achieves an accuracy of 99.3% seamlessly.
  arXiv  Detail & Related papers  (2022-05-25T13:42:35Z)
• DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers [105.74546828182834]
  We show a hardware-efficient dynamic inference regime, named dynamic weight slicing, which adaptively slice a part of network parameters for inputs with diverse difficulty levels. We present dynamic slimmable network (DS-Net) and dynamic slice-able network (DS-Net++) by input-dependently adjusting filter numbers of CNNs and multiple dimensions in both CNNs and transformers.
  arXiv  Detail & Related papers  (2021-09-21T09:57:21Z)
• Manifold Regularized Dynamic Network Pruning [102.24146031250034]
  This paper proposes a new paradigm that dynamically removes redundant filters by embedding the manifold information of all instances into the space of pruned networks. The effectiveness of the proposed method is verified on several benchmarks, which shows better performance in terms of both accuracy and computational cost.
  arXiv  Detail & Related papers  (2021-03-10T03:59:03Z)
• Inception Convolution with Efficient Dilation Search [121.41030859447487]
  Dilation convolution is a critical mutant of standard convolution neural network to control effective receptive fields and handle large scale variance of objects. We propose a new mutant of dilated convolution, namely inception (dilated) convolution where the convolutions have independent dilation among different axes, channels and layers. We explore a practical method for fitting the complex inception convolution to the data, a simple while effective dilation search algorithm(EDO) based on statistical optimization is developed.
  arXiv  Detail & Related papers  (2020-12-25T14:58:35Z)
• When Residual Learning Meets Dense Aggregation: Rethinking the Aggregation of Deep Neural Networks [57.0502745301132]
  We propose Micro-Dense Nets, a novel architecture with global residual learning and local micro-dense aggregations. Our micro-dense block can be integrated with neural architecture search based models to boost their performance.
  arXiv  Detail & Related papers  (2020-04-19T08:34:52Z)
• XSepConv: Extremely Separated Convolution [60.90871656244126]
  We propose a novel extremely separated convolutional block (XSepConv) It fuses spatially separable convolutions into depthwise convolution to reduce both the computational cost and parameter size of large kernels. XSepConv is designed to be an efficient alternative to vanilla depthwise convolution with large kernel sizes.
  arXiv  Detail & Related papers  (2020-02-27T11:46:17Z)
• Widening and Squeezing: Towards Accurate and Efficient QNNs [125.172220129257]
  Quantization neural networks (QNNs) are very attractive to the industry because their extremely cheap calculation and storage overhead, but their performance is still worse than that of networks with full-precision parameters. Most of existing methods aim to enhance performance of QNNs especially binary neural networks by exploiting more effective training techniques. We address this problem by projecting features in original full-precision networks to high-dimensional quantization features.
  arXiv  Detail & Related papers  (2020-02-03T04:11:13Z)
• Depth-wise Decomposition for Accelerating Separable Convolutions in Efficient Convolutional Neural Networks [36.64158994999578]
  Deep convolutional neural networks (CNNs) have been established as the primary methods for many computer vision tasks. Recently, depth-wise separable convolution has been proposed for image recognition tasks on computationally limited platforms. We propose a novel decomposition approach based on SVD, namely depth-wise decomposition, for expanding regular convolutions into depthwise separable convolutions.
  arXiv  Detail & Related papers  (2019-10-21T15:37:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.