A Survey on Surrogate-assisted Efficient Neural Architecture Search

• URL: http://arxiv.org/abs/2206.01520v1
• Date: Fri, 3 Jun 2022 12:02:20 GMT
• Title: A Survey on Surrogate-assisted Efficient Neural Architecture Search
• Authors: Shiqing Liu, Haoyu Zhang and Yaochu Jin
• Abstract summary: Neural architecture search (NAS) has become increasingly popular in the deep learning community recently. NAS is still laborious and time-consuming because a large number of performance estimations are required during the search process of NAS. To solve the major limitation of NAS, improving the efficiency of NAS is essential in the design of NAS.
• Score: 18.914781707473296
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural architecture search (NAS) has become increasingly popular in the deep learning community recently, mainly because it can provide an opportunity to allow interested users without rich expertise to benefit from the success of deep neural networks (DNNs). However, NAS is still laborious and time-consuming because a large number of performance estimations are required during the search process of NAS, and training DNNs is computationally intensive. To solve the major limitation of NAS, improving the efficiency of NAS is essential in the design of NAS. This paper begins with a brief introduction to the general framework of NAS. Then, the methods for evaluating network candidates under the proxy metrics are systematically discussed. This is followed by a description of surrogate-assisted NAS, which is divided into three different categories, namely Bayesian optimization for NAS, surrogate-assisted evolutionary algorithms for NAS, and MOP for NAS. Finally, remaining challenges and open research questions are discussed, and promising research topics are suggested in this emerging field.

Related papers

• How Much Is Hidden in the NAS Benchmarks? Few-Shot Adaptation of a NAS Predictor [22.87207410692821]
  We borrow from the rich field of meta-learning for few-shot adaptation and study applicability of those methods to NAS. Our meta-learning approach not only shows superior (or matching) performance in the cross-validation experiments but also successful extrapolation to a new search space and tasks.
  arXiv  Detail & Related papers  (2023-11-30T10:51:46Z)
• DiffusionNAG: Predictor-guided Neural Architecture Generation with Diffusion Models [56.584561770857306]
  We propose a novel conditional Neural Architecture Generation (NAG) framework based on diffusion models, dubbed DiffusionNAG. Specifically, we consider the neural architectures as directed graphs and propose a graph diffusion model for generating them. We validate the effectiveness of DiffusionNAG through extensive experiments in two predictor-based NAS scenarios: Transferable NAS and Bayesian Optimization (BO)-based NAS. When integrated into a BO-based algorithm, DiffusionNAG outperforms existing BO-based NAS approaches, particularly in the large MobileNetV3 search space on the ImageNet 1K dataset.
  arXiv  Detail & Related papers  (2023-05-26T13:58:18Z)
• Generalization Properties of NAS under Activation and Skip Connection Search [66.8386847112332]
  We study the generalization properties of Neural Architecture Search (NAS) under a unifying framework. We derive the lower (and upper) bounds of the minimum eigenvalue of the Neural Tangent Kernel (NTK) under the (in)finite-width regime. We show how the derived results can guide NAS to select the top-performing architectures, even in the case without training.
  arXiv  Detail & Related papers  (2022-09-15T12:11:41Z)
• UnrealNAS: Can We Search Neural Architectures with Unreal Data? [84.78460976605425]
  Neural architecture search (NAS) has shown great success in the automatic design of deep neural networks (DNNs) Previous work has analyzed the necessity of having ground-truth labels in NAS and inspired broad interest. We take a further step to question whether real data is necessary for NAS to be effective.
  arXiv  Detail & Related papers  (2022-05-04T16:30:26Z)
• NAS-Bench-360: Benchmarking Diverse Tasks for Neural Architecture Search [18.9676056830197]
  Most existing neural architecture search (NAS) benchmarks and algorithms prioritize performance on well-studied tasks. We present NAS-Bench-360, a benchmark suite for evaluating state-of-the-art NAS methods for convolutional neural networks (CNNs)
  arXiv  Detail & Related papers  (2021-10-12T01:13:18Z)
• Lessons from the Clustering Analysis of a Search Space: A Centroid-based Approach to Initializing NAS [12.901952926144258]
  Recent availability of NAS benchmarks have enabled low computational resources prototyping. A calibrated clustering analysis of the search space is performed. Second, the centroids are extracted and used to initialize a NAS algorithm.
  arXiv  Detail & Related papers  (2021-08-20T11:46:33Z)
• TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Search [98.22779489340869]
  We propose TransNAS-Bench-101, a benchmark dataset containing network performance across seven vision tasks. We explore two fundamentally different types of search space: cell-level search space and macro-level search space. With 7,352 backbones evaluated on seven tasks, 51,464 trained models with detailed training information are provided.
  arXiv  Detail & Related papers  (2021-05-25T12:15:21Z)
• A Comprehensive Survey on Hardware-Aware Neural Architecture Search [6.23453131728063]
  Neural Architecture Search (NAS) methods have been growing in popularity. NAS has been extensively studied in the past few years. Applying NAS to real-world problems still poses significant challenges and is not widely practical. One solution growing in popularity is to use multi-objective optimization algorithms in the NAS search strategy by taking into account execution latency, energy consumption, memory footprint, etc. This kind of NAS, called hardware-aware NAS (HW-NAS), makes searching the most efficient architecture more complicated and opens several questions.
  arXiv  Detail & Related papers  (2021-01-22T21:13:46Z)
• AdvantageNAS: Efficient Neural Architecture Search with Credit Assignment [23.988393741948485]
  We propose a novel search strategy for one-shot and sparse propagation NAS, namely AdvantageNAS. AdvantageNAS is a gradient-based approach that improves the search efficiency by introducing credit assignment in gradient estimation for architecture updates. Experiments on the NAS-Bench-201 and PTB dataset show that AdvantageNAS discovers an architecture with higher performance under a limited time budget.
  arXiv  Detail & Related papers  (2020-12-11T05:45:03Z)
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions [48.76705090826339]
  Neural Architecture Search (NAS) is a revolutionary algorithm, and the related research work is complicated and rich. We provide a new perspective: beginning with an overview of the characteristics of the earliest NAS algorithms, summarizing the problems in these early NAS algorithms. Besides, we conduct a detailed and comprehensive analysis, comparison, and summary of these works.
  arXiv  Detail & Related papers  (2020-06-01T13:08:03Z)
• NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search [55.12928953187342]
  We propose an extension to NAS-Bench-101: NAS-Bench-201 with a different search space, results on multiple datasets, and more diagnostic information. NAS-Bench-201 has a fixed search space and provides a unified benchmark for almost any up-to-date NAS algorithms. We provide additional diagnostic information such as fine-grained loss and accuracy, which can give inspirations to new designs of NAS algorithms.
  arXiv  Detail & Related papers  (2020-01-02T05:28:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.