Rethinking Bottleneck Structure for Efficient Mobile Network Design

• URL: http://arxiv.org/abs/2007.02269v4
• Date: Fri, 27 Nov 2020 16:02:39 GMT
• Title: Rethinking Bottleneck Structure for Efficient Mobile Network Design
• Authors: Zhou Daquan, Qibin Hou, Yunpeng Chen, Jiashi Feng, Shuicheng Yan
• Abstract summary: The inverted residual block is dominating architecture design for mobile networks recently. We propose to flip the structure and present a novel bottleneck design, called the sandglass block, that performs identity mapping and spatial transformation at higher dimensions. In ImageNet classification, by simply replacing the inverted residual block with our sandglass block without increasing parameters and computation, the classification accuracy can be improved by more than 1.7% over MobileNetV2.
• Score: 154.47657111869552
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The inverted residual block is dominating architecture design for mobile networks recently. It changes the classic residual bottleneck by introducing two design rules: learning inverted residuals and using linear bottlenecks. In this paper, we rethink the necessity of such design changes and find it may bring risks of information loss and gradient confusion. We thus propose to flip the structure and present a novel bottleneck design, called the sandglass block, that performs identity mapping and spatial transformation at higher dimensions and thus alleviates information loss and gradient confusion effectively. Extensive experiments demonstrate that, different from the common belief, such bottleneck structure is more beneficial than the inverted ones for mobile networks. In ImageNet classification, by simply replacing the inverted residual block with our sandglass block without increasing parameters and computation, the classification accuracy can be improved by more than 1.7% over MobileNetV2. On Pascal VOC 2007 test set, we observe that there is also 0.9% mAP improvement in object detection. We further verify the effectiveness of the sandglass block by adding it into the search space of neural architecture search method DARTS. With 25% parameter reduction, the classification accuracy is improved by 0.13% over previous DARTS models. Code can be found at: https://github.com/zhoudaquan/rethinking_bottleneck_design.

Related papers

• Lightweight Diffusion Models with Distillation-Based Block Neural Architecture Search [55.41583104734349]
  We propose to automatically remove structural redundancy in diffusion models with our proposed Diffusion Distillation-based Block-wise Neural Architecture Search (NAS) Given a larger pretrained teacher, we leverage DiffNAS to search for the smallest architecture which can achieve on-par or even better performance than the teacher. Different from previous block-wise NAS methods, DiffNAS contains a block-wise local search strategy and a retraining strategy with a joint dynamic loss.
  arXiv  Detail & Related papers  (2023-11-08T12:56:59Z)
• Pseudo-Inverted Bottleneck Convolution for DARTS Search Space [35.50068534514941]
  We introduce the Pseudo-Inverted Bottleneck (PIBConv) block intending to reduce the computational footprint of the inverted bottleneck block proposed in ConvNeXt. Our proposed architecture is much less sensitive to evaluation layer count and outperforms a DARTS network with similar size significantly, at layer counts as small as 2.
  arXiv  Detail & Related papers  (2022-12-31T22:56:04Z)
• FOSTER: Feature Boosting and Compression for Class-Incremental Learning [52.603520403933985]
  Deep neural networks suffer from catastrophic forgetting when learning new categories. We propose a novel two-stage learning paradigm FOSTER, empowering the model to learn new categories adaptively.
  arXiv  Detail & Related papers  (2022-04-10T11:38:33Z)
• EResFD: Rediscovery of the Effectiveness of Standard Convolution for Lightweight Face Detection [13.357235715178584]
  We re-examine the effectiveness of the standard convolutional block as a lightweight backbone architecture for face detection. We show that heavily channel-pruned standard convolution layers can achieve better accuracy and inference speed. Our proposed detector EResFD obtained 80.4% mAP on WIDER FACE Hard subset which only takes 37.7 ms for VGA image inference on CPU.
  arXiv  Detail & Related papers  (2022-04-04T02:30:43Z)
• a novel attention-based network for fast salient object detection [14.246237737452105]
  In the current salient object detection network, the most popular method is using U-shape structure. We propose a new deep convolution network architecture with three contributions. Results demonstrate that the proposed method can compress the model to 1/3 of the original size nearly without losing the accuracy.
  arXiv  Detail & Related papers  (2021-12-20T12:30:20Z)
• MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning [72.80896338009579]
  We find that the memory bottleneck is due to the imbalanced memory distribution in convolutional neural network (CNN) designs. We propose a generic patch-by-patch inference scheduling, which significantly cuts down the peak memory. We automate the process with neural architecture search to jointly optimize the neural architecture and inference scheduling, leading to MCUNetV2.
  arXiv  Detail & Related papers  (2021-10-28T17:58:45Z)
• AsymmNet: Towards ultralight convolution neural networks using asymmetrical bottlenecks [7.736874685602911]
  Deep convolutional neural networks (CNN) have achieved astonishing results in a large variety of applications. However, using these models on mobile or embedded devices is difficult due to the limited memory and computation resources. We propose a novel design, called asymmetrical bottlenecks. Precisely, we adjust the first pointwise convolution, enrich the information flow by feature reuse, and migrate saved computations to the second pointwise convolution.
  arXiv  Detail & Related papers  (2021-04-15T20:58:39Z)
• Looking for change? Roll the Dice and demand Attention [0.0]
  We propose a reliable deep learning framework for the task of semantic change detection in high-resolution aerial images. Our framework consists of a new loss function, new attention modules, new feature extraction building blocks, and a new backbone architecture. We validate our approach by showing excellent performance and achieving state of the art score (F1 and Intersection over Union-hereafter IoU) on two building change detection datasets.
  arXiv  Detail & Related papers  (2020-09-04T08:30:25Z)
• When Residual Learning Meets Dense Aggregation: Rethinking the Aggregation of Deep Neural Networks [57.0502745301132]
  We propose Micro-Dense Nets, a novel architecture with global residual learning and local micro-dense aggregations. Our micro-dense block can be integrated with neural architecture search based models to boost their performance.
  arXiv  Detail & Related papers  (2020-04-19T08:34:52Z)
• FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions [70.59851564292828]
  Differentiable Neural Architecture Search (DNAS) has demonstrated great success in designing state-of-the-art, efficient neural networks. We propose a memory and computationally efficient DNAS variant: DMaskingNAS. This algorithm expands the search space by up to $1014times$ over conventional DNAS.
  arXiv  Detail & Related papers  (2020-04-12T08:52:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.