FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions

• URL: http://arxiv.org/abs/2004.05565v1
• Date: Sun, 12 Apr 2020 08:52:15 GMT
• Title: FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions
• Authors: Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, Peter Vajda, Joseph E. Gonzalez
• Abstract summary: Differentiable Neural Architecture Search (DNAS) has demonstrated great success in designing state-of-the-art, efficient neural networks. We propose a memory and computationally efficient DNAS variant: DMaskingNAS. This algorithm expands the search space by up to $1014times$ over conventional DNAS.
• Score: 70.59851564292828
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Differentiable Neural Architecture Search (DNAS) has demonstrated great success in designing state-of-the-art, efficient neural networks. However, DARTS-based DNAS's search space is small when compared to other search methods', since all candidate network layers must be explicitly instantiated in memory. To address this bottleneck, we propose a memory and computationally efficient DNAS variant: DMaskingNAS. This algorithm expands the search space by up to $10^{14}\times$ over conventional DNAS, supporting searches over spatial and channel dimensions that are otherwise prohibitively expensive: input resolution and number of filters. We propose a masking mechanism for feature map reuse, so that memory and computational costs stay nearly constant as the search space expands. Furthermore, we employ effective shape propagation to maximize per-FLOP or per-parameter accuracy. The searched FBNetV2s yield state-of-the-art performance when compared with all previous architectures. With up to 421$\times$ less search cost, DMaskingNAS finds models with 0.9% higher accuracy, 15% fewer FLOPs than MobileNetV3-Small; and with similar accuracy but 20% fewer FLOPs than Efficient-B0. Furthermore, our FBNetV2 outperforms MobileNetV3 by 2.6% in accuracy, with equivalent model size. FBNetV2 models are open-sourced at https://github.com/facebookresearch/mobile-vision.

Related papers

• Multi-Objective Neural Architecture Search by Learning Search Space Partitions [8.4553113915588]
  We implement a novel meta-algorithm called LaMOO on neural architecture search (NAS) tasks. LaMOO speedups the search process by learning a model from observed samples to partition the search space and then focusing on promising regions. For real-world tasks, LaMOO achieves 97.36% accuracy with only 1.62M #Params on CIFAR10 in only 600 search samples.
  arXiv  Detail & Related papers  (2024-06-01T03:51:34Z)
• MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning [72.80896338009579]
  We find that the memory bottleneck is due to the imbalanced memory distribution in convolutional neural network (CNN) designs. We propose a generic patch-by-patch inference scheduling, which significantly cuts down the peak memory. We automate the process with neural architecture search to jointly optimize the neural architecture and inference scheduling, leading to MCUNetV2.
  arXiv  Detail & Related papers  (2021-10-28T17:58:45Z)
• OPANAS: One-Shot Path Aggregation Network Architecture Search for Object Detection [82.04372532783931]
  Recently, neural architecture search (NAS) has been exploited to design feature pyramid networks (FPNs) We propose a novel One-Shot Path Aggregation Network Architecture Search (OPANAS) algorithm, which significantly improves both searching efficiency and detection accuracy.
  arXiv  Detail & Related papers  (2021-03-08T01:48:53Z)
• Distilling Optimal Neural Networks: Rapid Search in Diverse Spaces [16.920328058816338]
  DONNA (Distilling Optimal Neural Network Architectures) is a novel pipeline for rapid neural architecture search and search space exploration. In ImageNet classification, architectures found by DONNA are 20% faster than EfficientNet-B0 and MobileNetV2 on a Nvidia V100 GPU at similar accuracy and 10% faster with 0.5% higher accuracy than MobileNetV2-1.4x on a Samsung S20 smartphone.
  arXiv  Detail & Related papers  (2020-12-16T11:00:19Z)
• FP-NAS: Fast Probabilistic Neural Architecture Search [49.21560787752714]
  Probabilistic NAS, such as PARSEC, learns a distribution over high-performing architectures, and uses only as much memory as needed to train a single model. We propose a sampling method adaptive to the distribution entropy, drawing more samples to encourage explorations at the beginning, and reducing samples as learning proceeds. We show that Fast Probabilistic NAS (FP-NAS) can sample 64% fewer architectures and search 2.1x faster than PARSEC.
  arXiv  Detail & Related papers  (2020-11-22T06:10:05Z)
• PV-NAS: Practical Neural Architecture Search for Video Recognition [83.77236063613579]
  Deep neural networks for video tasks is highly customized and the design of such networks requires domain experts and costly trial and error tests. Recent advance in network architecture search has boosted the image recognition performance in a large margin. In this study, we propose a practical solution, namely Practical Video Neural Architecture Search (PV-NAS)
  arXiv  Detail & Related papers  (2020-11-02T08:50:23Z)
• Accuracy Prediction with Non-neural Model for Neural Architecture Search [185.0651567642238]
  We study an alternative approach which uses non-neural model for accuracy prediction. We leverage gradient boosting decision tree (GBDT) as the predictor for Neural architecture search (NAS) Experiments on NASBench-101 and ImageNet demonstrate the effectiveness of using GBDT as predictor for NAS.
  arXiv  Detail & Related papers  (2020-07-09T13:28:49Z)
• Fine-Grained Stochastic Architecture Search [6.277767522867666]
  Fine-Grained Architecture Search (FiGS) is a differentiable search method that searches over a much larger set of candidate architectures. FiGS simultaneously selects and modifies operators in the search space by applying a structured sparse regularization penalty. We show results across 3 existing search spaces, matching or outperforming the original search algorithms.
  arXiv  Detail & Related papers  (2020-06-17T01:04:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.