Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio

• URL: http://arxiv.org/abs/2004.02767v1
• Date: Mon, 6 Apr 2020 15:51:00 GMT
• Title: Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio
• Authors: Zhengsu Chen, Jianwei Niu, Lingxi Xie, Xuefeng Liu, Longhui Wei, Qi Tian
• Abstract summary: This paper presents a new framework named network adjustment, which considers network accuracy as a function of FLOPs. Experiments on standard image classification datasets and a wide range of base networks demonstrate the effectiveness of our approach.
• Score: 101.84651388520584
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automatic designing computationally efficient neural networks has received much attention in recent years. Existing approaches either utilize network pruning or leverage the network architecture search methods. This paper presents a new framework named network adjustment, which considers network accuracy as a function of FLOPs, so that under each network configuration, one can estimate the FLOPs utilization ratio (FUR) for each layer and use it to determine whether to increase or decrease the number of channels on the layer. Note that FUR, like the gradient of a non-linear function, is accurate only in a small neighborhood of the current network. Hence, we design an iterative mechanism so that the initial network undergoes a number of steps, each of which has a small `adjusting rate' to control the changes to the network. The computational overhead of the entire search process is reasonable, i.e., comparable to that of re-training the final model from scratch. Experiments on standard image classification datasets and a wide range of base networks demonstrate the effectiveness of our approach, which consistently outperforms the pruning counterpart. The code is available at https://github.com/danczs/NetworkAdjustment.

Related papers

• CONetV2: Efficient Auto-Channel Size Optimization for CNNs [35.951376988552695]
  This work introduces a method that is efficient in computationally constrained environments by examining the micro-search space of channel size. In tackling channel-size optimization, we design an automated algorithm to extract the dependencies within different connected layers of the network. We also introduce a novel metric that highly correlates with test accuracy and enables analysis of individual network layers.
  arXiv  Detail & Related papers  (2021-10-13T16:17:19Z)
• CONet: Channel Optimization for Convolutional Neural Networks [33.58529066005248]
  We study channel size optimization in convolutional neural networks (CNN) We introduce an efficient dynamic scaling algorithm -- CONet -- that automatically optimize channel sizes across network layers for a given CNN. We conduct experiments on CIFAR10/100 and ImageNet datasets and show that CONet can find efficient and accurate architectures searched in ResNet, DARTS, and DARTS+ spaces.
  arXiv  Detail & Related papers  (2021-08-15T21:48:25Z)
• Group Fisher Pruning for Practical Network Compression [58.25776612812883]
  We present a general channel pruning approach that can be applied to various complicated structures. We derive a unified metric based on Fisher information to evaluate the importance of a single channel and coupled channels. Our method can be used to prune any structures including those with coupled channels.
  arXiv  Detail & Related papers  (2021-08-02T08:21:44Z)
• Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization [16.85167651136133]
  We take a broader view of training sparse networks and consider the role of regularization, optimization and architecture choices on sparse models. We show that gradient flow in sparse networks can be improved by reconsidering aspects of the architecture design and the training regime.
  arXiv  Detail & Related papers  (2021-02-02T18:40:26Z)
• Channel Planting for Deep Neural Networks using Knowledge Distillation [3.0165431987188245]
  We present a novel incremental training algorithm for deep neural networks called planting. Our planting can search the optimal network architecture with smaller number of parameters for improving the network performance. We evaluate the effectiveness of the proposed method on different datasets such as CIFAR-10/100 and STL-10.
  arXiv  Detail & Related papers  (2020-11-04T16:29:59Z)
• AutoPruning for Deep Neural Network with Dynamic Channel Masking [28.018077874687343]
  We propose a learning based auto pruning algorithm for deep neural network. A two objectives' problem that aims for the the weights and the best channels for each layer is first formulated. An alternative optimization approach is then proposed to derive the optimal channel numbers and weights simultaneously.
  arXiv  Detail & Related papers  (2020-10-22T20:12:46Z)
• Dynamic Graph: Learning Instance-aware Connectivity for Neural Networks [78.65792427542672]
  Dynamic Graph Network (DG-Net) is a complete directed acyclic graph, where the nodes represent convolutional blocks and the edges represent connection paths. Instead of using the same path of the network, DG-Net aggregates features dynamically in each node, which allows the network to have more representation ability.
  arXiv  Detail & Related papers  (2020-10-02T16:50:26Z)
• The Heterogeneity Hypothesis: Finding Layer-Wise Differentiated Network Architectures [179.66117325866585]
  We investigate a design space that is usually overlooked, i.e. adjusting the channel configurations of predefined networks. We find that this adjustment can be achieved by shrinking widened baseline networks and leads to superior performance. Experiments are conducted on various networks and datasets for image classification, visual tracking and image restoration.
  arXiv  Detail & Related papers  (2020-06-29T17:59:26Z)
• Fitting the Search Space of Weight-sharing NAS with Graph Convolutional Networks [100.14670789581811]
  We train a graph convolutional network to fit the performance of sampled sub-networks. With this strategy, we achieve a higher rank correlation coefficient in the selected set of candidates.
  arXiv  Detail & Related papers  (2020-04-17T19:12:39Z)
• DHP: Differentiable Meta Pruning via HyperNetworks [158.69345612783198]
  This paper introduces a differentiable pruning method via hypernetworks for automatic network pruning. Latent vectors control the output channels of the convolutional layers in the backbone network and act as a handle for the pruning of the layers. Experiments are conducted on various networks for image classification, single image super-resolution, and denoising.
  arXiv  Detail & Related papers  (2020-03-30T17:59:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.