FP-NAS: Fast Probabilistic Neural Architecture Search

• URL: http://arxiv.org/abs/2011.10949v3
• Date: Wed, 31 Mar 2021 17:21:49 GMT
• Title: FP-NAS: Fast Probabilistic Neural Architecture Search
• Authors: Zhicheng Yan, Xiaoliang Dai, Peizhao Zhang, Yuandong Tian, Bichen Wu, Matt Feiszli
• Abstract summary: Probabilistic NAS, such as PARSEC, learns a distribution over high-performing architectures, and uses only as much memory as needed to train a single model. We propose a sampling method adaptive to the distribution entropy, drawing more samples to encourage explorations at the beginning, and reducing samples as learning proceeds. We show that Fast Probabilistic NAS (FP-NAS) can sample 64% fewer architectures and search 2.1x faster than PARSEC.
• Score: 49.21560787752714
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Differential Neural Architecture Search (NAS) requires all layer choices to be held in memory simultaneously; this limits the size of both search space and final architecture. In contrast, Probabilistic NAS, such as PARSEC, learns a distribution over high-performing architectures, and uses only as much memory as needed to train a single model. Nevertheless, it needs to sample many architectures, making it computationally expensive for searching in an extensive space. To solve these problems, we propose a sampling method adaptive to the distribution entropy, drawing more samples to encourage explorations at the beginning, and reducing samples as learning proceeds. Furthermore, to search fast in the multi-variate space, we propose a coarse-to-fine strategy by using a factorized distribution at the beginning which can reduce the number of architecture parameters by over an order of magnitude. We call this method Fast Probabilistic NAS (FP-NAS). Compared with PARSEC, it can sample 64% fewer architectures and search 2.1x faster. Compared with FBNetV2, FP-NAS is 1.9x - 3.5x faster, and the searched models outperform FBNetV2 models on ImageNet. FP-NAS allows us to expand the giant FBNetV2 space to be wider (i.e. larger channel choices) and deeper (i.e. more blocks), while adding Split-Attention block and enabling the search over the number of splits. When searching a model of size 0.4G FLOPS, FP-NAS is 132x faster than EfficientNet, and the searched FP-NAS-L0 model outperforms EfficientNet-B0 by 0.7% accuracy. Without using any architecture surrogate or scaling tricks, we directly search large models up to 1.0G FLOPS. Our FP-NAS-L2 model with simple distillation outperforms BigNAS-XL with advanced in-place distillation by 0.7% accuracy using similar FLOPS.

Related papers

• $\alpha$NAS: Neural Architecture Search using Property Guided Synthesis [1.2746672439030722]
  We develop techniques that enable efficient neural architecture search (NAS) in a significantly larger design space. Our key insights are as follows: (1) the abstract search space is significantly smaller than the original search space, and (2) architectures with similar program properties also have similar performance. We implement our approach, $alpha$NAS, within an evolutionary framework, where the mutations are guided by the program properties.
  arXiv  Detail & Related papers  (2022-05-08T21:48:03Z)
• When NAS Meets Trees: An Efficient Algorithm for Neural Architecture Search [117.89827740405694]
  Key challenge in neural architecture search (NAS) is designing how to explore wisely in the huge search space. We propose a new NAS method called TNAS (NAS with trees), which improves search efficiency by exploring only a small number of architectures. TNAS finds the global optimal architecture on CIFAR-10 with test accuracy of 94.37% in four GPU hours in NAS-Bench-201.
  arXiv  Detail & Related papers  (2022-04-11T07:34:21Z)
• OPANAS: One-Shot Path Aggregation Network Architecture Search for Object Detection [82.04372532783931]
  Recently, neural architecture search (NAS) has been exploited to design feature pyramid networks (FPNs) We propose a novel One-Shot Path Aggregation Network Architecture Search (OPANAS) algorithm, which significantly improves both searching efficiency and detection accuracy.
  arXiv  Detail & Related papers  (2021-03-08T01:48:53Z)
• Memory-Efficient Hierarchical Neural Architecture Search for Image Restoration [68.6505473346005]
  We propose a memory-efficient hierarchical NAS HiNAS (HiNAS) for image denoising and image super-resolution tasks. With a single GTX1080Ti GPU, it takes only about 1 hour for searching for denoising network on BSD 500 and 3.5 hours for searching for the super-resolution structure on DIV2K.
  arXiv  Detail & Related papers  (2020-12-24T12:06:17Z)
• FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions [70.59851564292828]
  Differentiable Neural Architecture Search (DNAS) has demonstrated great success in designing state-of-the-art, efficient neural networks. We propose a memory and computationally efficient DNAS variant: DMaskingNAS. This algorithm expands the search space by up to $1014times$ over conventional DNAS.
  arXiv  Detail & Related papers  (2020-04-12T08:52:15Z)
• Probabilistic Dual Network Architecture Search on Graphs [13.140262952697746]
  We present the first differentiable Network Architecture Search (NAS) for Graph Neural Networks (GNNs) GNNs show promising performance on a wide range of tasks, but require a large amount of architecture engineering. We use a fully gradient-based search approach to update architectural parameters, making it the first differentiable graph NAS method.
  arXiv  Detail & Related papers  (2020-03-21T15:06:47Z)
• DDPNAS: Efficient Neural Architecture Search via Dynamic Distribution Pruning [135.27931587381596]
  We propose an efficient and unified NAS framework termed DDPNAS via dynamic distribution pruning. In particular, we first sample architectures from a joint categorical distribution. Then the search space is dynamically pruned and its distribution is updated every few epochs. With the proposed efficient network generation method, we directly obtain the optimal neural architectures on given constraints.
  arXiv  Detail & Related papers  (2019-05-28T06:35:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.