Efficient Sampling for Predictor-Based Neural Architecture Search

• URL: http://arxiv.org/abs/2011.12043v1
• Date: Tue, 24 Nov 2020 11:36:36 GMT
• Title: Efficient Sampling for Predictor-Based Neural Architecture Search
• Authors: Lukas Mauch, Stephen Tiedemann, Javier Alonso Garcia, Bac Nguyen Cong, Kazuki Yoshiyama, Fabien Cardinaux, Thomas Kemp
• Abstract summary: We study predictor-based NAS algorithms for neural architecture search. We show that the sample efficiency of predictor-based algorithms decreases dramatically if the proxy is only computed for a subset of the search space. This is an important step to make predictor-based NAS algorithms useful, in practice.
• Score: 3.287802528135173
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, predictor-based algorithms emerged as a promising approach for neural architecture search (NAS). For NAS, we typically have to calculate the validation accuracy of a large number of Deep Neural Networks (DNNs), what is computationally complex. Predictor-based NAS algorithms address this problem. They train a proxy model that can infer the validation accuracy of DNNs directly from their network structure. During optimization, the proxy can be used to narrow down the number of architectures for which the true validation accuracy must be computed, what makes predictor-based algorithms sample efficient. Usually, we compute the proxy for all DNNs in the network search space and pick those that maximize the proxy as candidates for optimization. However, that is intractable in practice, because the search spaces are often very large and contain billions of network architectures. The contributions of this paper are threefold: 1) We define a sample efficiency gain to compare different predictor-based NAS algorithms. 2) We conduct experiments on the NASBench-101 dataset and show that the sample efficiency of predictor-based algorithms decreases dramatically if the proxy is only computed for a subset of the search space. 3) We show that if we choose the subset of the search space on which the proxy is evaluated in a smart way, the sample efficiency of the original predictor-based algorithm that has access to the full search space can be regained. This is an important step to make predictor-based NAS algorithms useful, in practice.

Related papers

• Multi-Predict: Few Shot Predictors For Efficient Neural Architecture Search [10.538869116366415]
  We introduce a novel search-space independent NN encoding based on zero-cost proxies that achieves sample-efficient prediction on multiple tasks and NAS search spaces. Our NN encoding enables multi-search-space transfer of latency predictors from NASBench-201 to FBNet in under 85 HW measurements.
  arXiv  Detail & Related papers  (2023-06-04T20:22:14Z)
• Approximate Neural Architecture Search via Operation Distribution Learning [4.358626952482686]
  We show that given an architectural cell, its performance largely depends on the ratio of used operations. This intuition is to any specific search strategy and can be applied to a diverse set of NAS algorithms.
  arXiv  Detail & Related papers  (2021-11-08T17:38:29Z)
• A Data-driven Approach to Neural Architecture Search Initialization [12.901952926144258]
  We propose a data-driven technique to initialize a population-based NAS algorithm. We benchmark our proposed approach against random and Latin hypercube sampling.
  arXiv  Detail & Related papers  (2021-11-05T14:30:19Z)
• IQNAS: Interpretable Integer Quadratic Programming Neural Architecture Search [40.77061519007659]
  A popular approach to find fitting networks is through constrained Neural Architecture Search (NAS) Previous methods use complicated predictors for the accuracy of the network. We introduce Interpretable Quadratic programming Neural Architecture Search (IQNAS)
  arXiv  Detail & Related papers  (2021-10-24T09:45:00Z)
• Learning to Hash Robustly, with Guarantees [79.68057056103014]
  In this paper, we design an NNS algorithm for the Hamming space that has worst-case guarantees essentially matching that of theoretical algorithms. We evaluate the algorithm's ability to optimize for a given dataset both theoretically and practically. Our algorithm has a 1.8x and 2.1x better recall on the worst-performing queries to the MNIST and ImageNet datasets.
  arXiv  Detail & Related papers  (2021-08-11T20:21:30Z)
• OPANAS: One-Shot Path Aggregation Network Architecture Search for Object Detection [82.04372532783931]
  Recently, neural architecture search (NAS) has been exploited to design feature pyramid networks (FPNs) We propose a novel One-Shot Path Aggregation Network Architecture Search (OPANAS) algorithm, which significantly improves both searching efficiency and detection accuracy.
  arXiv  Detail & Related papers  (2021-03-08T01:48:53Z)
• Towards Optimally Efficient Tree Search with Deep Learning [76.64632985696237]
  This paper investigates the classical integer least-squares problem which estimates signals integer from linear models. The problem is NP-hard and often arises in diverse applications such as signal processing, bioinformatics, communications and machine learning. We propose a general hyper-accelerated tree search (HATS) algorithm by employing a deep neural network to estimate the optimal estimation for the underlying simplified memory-bounded A* algorithm.
  arXiv  Detail & Related papers  (2021-01-07T08:00:02Z)
• BRP-NAS: Prediction-based NAS using GCNs [21.765796576990137]
  BRP-NAS is an efficient hardware-aware NAS enabled by an accurate performance predictor-based on graph convolutional network (GCN) We show that our proposed method outperforms all prior methods on NAS-Bench-101 and NAS-Bench-201. We also release LatBench -- a latency dataset of NAS-Bench-201 models running on a broad range of devices.
  arXiv  Detail & Related papers  (2020-07-16T21:58:43Z)
• Accuracy Prediction with Non-neural Model for Neural Architecture Search [185.0651567642238]
  We study an alternative approach which uses non-neural model for accuracy prediction. We leverage gradient boosting decision tree (GBDT) as the predictor for Neural architecture search (NAS) Experiments on NASBench-101 and ImageNet demonstrate the effectiveness of using GBDT as predictor for NAS.
  arXiv  Detail & Related papers  (2020-07-09T13:28:49Z)
• FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining [65.39532971991778]
  We present an accuracy predictor that scores architecture and training recipes jointly, guiding both sample selection and ranking. We run fast evolutionary searches in just CPU minutes to generate architecture-recipe pairs for a variety of resource constraints. FBNetV3 makes up a family of state-of-the-art compact neural networks that outperform both automatically and manually-designed competitors.
  arXiv  Detail & Related papers  (2020-06-03T05:20:21Z)
• DA-NAS: Data Adapted Pruning for Efficient Neural Architecture Search [76.9225014200746]
  Efficient search is a core issue in Neural Architecture Search (NAS) We present DA-NAS that can directly search the architecture for large-scale target tasks while allowing a large candidate set in a more efficient manner. It is 2x faster than previous methods while the accuracy is currently state-of-the-art, at 76.2% under small FLOPs constraint.
  arXiv  Detail & Related papers  (2020-03-27T17:55:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.