When Residual Learning Meets Dense Aggregation: Rethinking the Aggregation of Deep Neural Networks

• URL: http://arxiv.org/abs/2004.08796v2
• Date: Fri, 24 Apr 2020 05:21:07 GMT
• Title: When Residual Learning Meets Dense Aggregation: Rethinking the Aggregation of Deep Neural Networks
• Authors: Zhiyu Zhu, Zhen-Peng Bian, Junhui Hou, Yi Wang, Lap-Pui Chau
• Abstract summary: We propose Micro-Dense Nets, a novel architecture with global residual learning and local micro-dense aggregations. Our micro-dense block can be integrated with neural architecture search based models to boost their performance.
• Score: 57.0502745301132
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Various architectures (such as GoogLeNets, ResNets, and DenseNets) have been proposed. However, the existing networks usually suffer from either redundancy of convolutional layers or insufficient utilization of parameters. To handle these challenging issues, we propose Micro-Dense Nets, a novel architecture with global residual learning and local micro-dense aggregations. Specifically, residual learning aims to efficiently retrieve features from different convolutional blocks, while the micro-dense aggregation is able to enhance each block and avoid redundancy of convolutional layers by lessening residual aggregations. Moreover, the proposed micro-dense architecture has two characteristics: pyramidal multi-level feature learning which can widen the deeper layer in a block progressively, and dimension cardinality adaptive convolution which can balance each layer using linearly increasing dimension cardinality. The experimental results over three datasets (i.e., CIFAR-10, CIFAR-100, and ImageNet-1K) demonstrate that the proposed Micro-Dense Net with only 4M parameters can achieve higher classification accuracy than state-of-the-art networks, while being 12.1$\times$ smaller depends on the number of parameters. In addition, our micro-dense block can be integrated with neural architecture search based models to boost their performance, validating the advantage of our architecture. We believe our design and findings will be beneficial to the DNN community.

Related papers

• DiTMoS: Delving into Diverse Tiny-Model Selection on Microcontrollers [34.282971510732736]
  We introduce DiTMoS, a novel DNN training and inference framework with a selector-classifiers architecture. A composition of weak models can exhibit high diversity and the union of them can significantly boost the accuracy upper bound. We deploy DiTMoS on the Neucleo STM32F767ZI board and evaluate it based on three time-series datasets for human activity recognition, keywords spotting, and emotion recognition.
  arXiv  Detail & Related papers  (2024-03-14T02:11:38Z)
• Neural Attentive Circuits [93.95502541529115]
  We introduce a general purpose, yet modular neural architecture called Neural Attentive Circuits (NACs) NACs learn the parameterization and a sparse connectivity of neural modules without using domain knowledge. NACs achieve an 8x speedup at inference time while losing less than 3% performance.
  arXiv  Detail & Related papers  (2022-10-14T18:00:07Z)
• ME-CapsNet: A Multi-Enhanced Capsule Networks with Routing Mechanism [0.0]
  This research focuses on bringing in a novel solution that uses sophisticated optimization for enhancing both the spatial and channel components inside each layer's receptive field. We have proposed ME-CapsNet by introducing deeper convolutional layers to extract important features before passing through modules of capsule layers strategically. The deeper convolutional layer includes blocks of Squeeze-Excitation networks which use a sampling approach for reconstructing their interdependencies without much loss of important feature information.
  arXiv  Detail & Related papers  (2022-03-29T13:29:38Z)
• Over-and-Under Complete Convolutional RNN for MRI Reconstruction [57.95363471940937]
  Recent deep learning-based methods for MR image reconstruction usually leverage a generic auto-encoder architecture. We propose an Over-and-Under Complete Convolu?tional Recurrent Neural Network (OUCR), which consists of an overcomplete and an undercomplete Convolutional Recurrent Neural Network(CRNN) The proposed method achieves significant improvements over the compressed sensing and popular deep learning-based methods with less number of trainable parameters.
  arXiv  Detail & Related papers  (2021-06-16T15:56:34Z)
• Efficient Micro-Structured Weight Unification and Pruning for Neural Network Compression [56.83861738731913]
  Deep Neural Network (DNN) models are essential for practical applications, especially for resource limited devices. Previous unstructured or structured weight pruning methods can hardly truly accelerate inference. We propose a generalized weight unification framework at a hardware compatible micro-structured level to achieve high amount of compression and acceleration.
  arXiv  Detail & Related papers  (2021-06-15T17:22:59Z)
• A Deeper Look into Convolutions via Pruning [9.89901717499058]
  Modern architectures contain a very small number of fully-connected layers, often at the end, after multiple layers of convolutions. Although this strategy already reduces the number of parameters, most of the convolutions can be eliminated as well, without suffering any loss in recognition performance. In this work, we use the matrix characteristics based on eigenvalues in addition to the classical weight-based importance assignment approach for pruning to shed light on the internal mechanisms of a widely used family of CNNs.
  arXiv  Detail & Related papers  (2021-02-04T18:55:03Z)
• Dual-constrained Deep Semi-Supervised Coupled Factorization Network with Enriched Prior [80.5637175255349]
  We propose a new enriched prior based Dual-constrained Deep Semi-Supervised Coupled Factorization Network, called DS2CF-Net. To ex-tract hidden deep features, DS2CF-Net is modeled as a deep-structure and geometrical structure-constrained neural network. Our network can obtain state-of-the-art performance for representation learning and clustering.
  arXiv  Detail & Related papers  (2020-09-08T13:10:21Z)
• Res-CR-Net, a residual network with a novel architecture optimized for the semantic segmentation of microscopy images [0.5363346028859919]
  Res-CR-Net is a type of Deep Neural Network (DNN) that features residual blocks with either a bundle of separable atrous convolutions with different dilation rates or a convolutional LSTM. The number of filters used in each residual block and the number of blocks are the only hyper parameters that need to be modified in order to optimize the network training for a variety of different microscopy images.
  arXiv  Detail & Related papers  (2020-04-14T21:21:01Z)
• Dense Residual Network: Enhancing Global Dense Feature Flow for Character Recognition [75.4027660840568]
  This paper explores how to enhance the local and global dense feature flow by exploiting hierarchical features fully from all the convolution layers. Technically, we propose an efficient and effective CNN framework, i.e., Fast Dense Residual Network (FDRN) for text recognition.
  arXiv  Detail & Related papers  (2020-01-23T06:55:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.