A Data-driven Approach to Neural Architecture Search Initialization

• URL: http://arxiv.org/abs/2111.03524v1
• Date: Fri, 5 Nov 2021 14:30:19 GMT
• Title: A Data-driven Approach to Neural Architecture Search Initialization
• Authors: Kalifou Ren\'e Traor\'e, Andr\'es Camero and Xiao Xiang Zhu
• Abstract summary: We propose a data-driven technique to initialize a population-based NAS algorithm. We benchmark our proposed approach against random and Latin hypercube sampling.
• Score: 12.901952926144258
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Algorithmic design in neural architecture search (NAS) has received a lot of attention, aiming to improve performance and reduce computational cost. Despite the great advances made, few authors have proposed to tailor initialization techniques for NAS. However, literature shows that a good initial set of solutions facilitate finding the optima. Therefore, in this study, we propose a data-driven technique to initialize a population-based NAS algorithm. Particularly, we proposed a two-step methodology. First, we perform a calibrated clustering analysis of the search space, and second, we extract the centroids and use them to initialize a NAS algorithm. We benchmark our proposed approach against random and Latin hypercube sampling initialization using three population-based algorithms, namely a genetic algorithm, evolutionary algorithm, and aging evolution, on CIFAR-10. More specifically, we use NAS-Bench-101 to leverage the availability of NAS benchmarks. The results show that compared to random and Latin hypercube sampling, the proposed initialization technique enables achieving significant long-term improvements for two of the search baselines, and sometimes in various search scenarios (various training budgets). Moreover, we analyze the distributions of solutions obtained and find that that the population provided by the data-driven initialization technique enables retrieving local optima (maxima) of high fitness and similar configurations.

Related papers

• A Pairwise Comparison Relation-assisted Multi-objective Evolutionary Neural Architecture Search Method with Multi-population Mechanism [58.855741970337675]
  Neural architecture search (NAS) enables re-searchers to automatically explore vast search spaces and find efficient neural networks. NAS suffers from a key bottleneck, i.e., numerous architectures need to be evaluated during the search process. We propose the SMEM-NAS, a pairwise com-parison relation-assisted multi-objective evolutionary algorithm based on a multi-population mechanism.
  arXiv  Detail & Related papers  (2024-07-22T12:46:22Z)
• DCP-NAS: Discrepant Child-Parent Neural Architecture Search for 1-bit CNNs [53.82853297675979]
  1-bit convolutional neural networks (CNNs) with binary weights and activations show their potential for resource-limited embedded devices. One natural approach is to use 1-bit CNNs to reduce the computation and memory cost of NAS. We introduce Discrepant Child-Parent Neural Architecture Search (DCP-NAS) to efficiently search 1-bit CNNs.
  arXiv  Detail & Related papers  (2023-06-27T11:28:29Z)
• Speeding up NAS with Adaptive Subset Selection [21.31075249079979]
  We present an adaptive subset selection approach to neural architecture search (NAS) We devise an algorithm that makes use of state-of-the-art techniques from both areas. Our results are consistent across multiple datasets.
  arXiv  Detail & Related papers  (2022-11-02T19:48:42Z)
• Going Beyond Neural Architecture Search with Sampling-based Neural Ensemble Search [31.059040393415003]
  We present two novel sampling algorithms under our Neural Ensemble Search via Sampling (NESS) framework. Our NESS algorithms are shown to be able to achieve improved performance in both classification and adversarial defense tasks.
  arXiv  Detail & Related papers  (2021-09-06T15:18:37Z)
• NASI: Label- and Data-agnostic Neural Architecture Search at Initialization [35.18069719489172]
  We propose a novel NAS algorithm called NAS at Initialization (NASI) NASI exploits the capability of a Neural Tangent Kernel in being able to characterize the converged performance of candidate architectures. NASI also achieves competitive search effectiveness on various datasets like CIFAR-10/100 and ImageNet.
  arXiv  Detail & Related papers  (2021-09-02T09:49:28Z)
• Lessons from the Clustering Analysis of a Search Space: A Centroid-based Approach to Initializing NAS [12.901952926144258]
  Recent availability of NAS benchmarks have enabled low computational resources prototyping. A calibrated clustering analysis of the search space is performed. Second, the centroids are extracted and used to initialize a NAS algorithm.
  arXiv  Detail & Related papers  (2021-08-20T11:46:33Z)
• Trilevel Neural Architecture Search for Efficient Single Image Super-Resolution [127.92235484598811]
  This paper proposes a trilevel neural architecture search (NAS) method for efficient single image super-resolution (SR) For modeling the discrete search space, we apply a new continuous relaxation on the discrete search spaces to build a hierarchical mixture of network-path, cell-operations, and kernel-width. An efficient search algorithm is proposed to perform optimization in a hierarchical supernet manner.
  arXiv  Detail & Related papers  (2021-01-17T12:19:49Z)
• Efficient Sampling for Predictor-Based Neural Architecture Search [3.287802528135173]
  We study predictor-based NAS algorithms for neural architecture search. We show that the sample efficiency of predictor-based algorithms decreases dramatically if the proxy is only computed for a subset of the search space. This is an important step to make predictor-based NAS algorithms useful, in practice.
  arXiv  Detail & Related papers  (2020-11-24T11:36:36Z)
• Hyperparameter Optimization in Neural Networks via Structured Sparse Recovery [54.60327265077322]
  We study two important problems in the automated design of neural networks through the lens of sparse recovery methods. In the first part of this paper, we establish a novel connection between HPO and structured sparse recovery. In the second part of this paper, we establish a connection between NAS and structured sparse recovery.
  arXiv  Detail & Related papers  (2020-07-07T00:57:09Z)
• DrNAS: Dirichlet Neural Architecture Search [88.56953713817545]
  We treat the continuously relaxed architecture mixing weight as random variables, modeled by Dirichlet distribution. With recently developed pathwise derivatives, the Dirichlet parameters can be easily optimized with gradient-based generalization. To alleviate the large memory consumption of differentiable NAS, we propose a simple yet effective progressive learning scheme.
  arXiv  Detail & Related papers  (2020-06-18T08:23:02Z)
• DA-NAS: Data Adapted Pruning for Efficient Neural Architecture Search [76.9225014200746]
  Efficient search is a core issue in Neural Architecture Search (NAS) We present DA-NAS that can directly search the architecture for large-scale target tasks while allowing a large candidate set in a more efficient manner. It is 2x faster than previous methods while the accuracy is currently state-of-the-art, at 76.2% under small FLOPs constraint.
  arXiv  Detail & Related papers  (2020-03-27T17:55:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.