Analyzing the Expected Hitting Time of Evolutionary Computation-based Neural Architecture Search Algorithms

• URL: http://arxiv.org/abs/2210.05397v3
• Date: Fri, 15 Mar 2024 06:51:04 GMT
• Title: Analyzing the Expected Hitting Time of Evolutionary Computation-based Neural Architecture Search Algorithms
• Authors: Zeqiong Lv, Chao Qian, Gary G. Yen, Yanan Sun,
• Abstract summary: The expected hitting time (EHT) is one of the most important theoretical issues, since it implies the average computational time complexity. This paper proposes a general method by integrating theory and experiment for estimating the EHT of ENAS algorithms. To the best of our knowledge, this work is the first attempt to establish a theoretical foundation for ENAS algorithms.
• Score: 29.385876073356044
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evolutionary computation-based neural architecture search (ENAS) is a popular technique for automating architecture design of deep neural networks. Despite its groundbreaking applications, there is no theoretical study for ENAS. The expected hitting time (EHT) is one of the most important theoretical issues, since it implies the average computational time complexity. This paper proposes a general method by integrating theory and experiment for estimating the EHT of ENAS algorithms, which includes common configuration, search space partition, transition probability estimation, population distribution fitting, and hitting time analysis. By exploiting the proposed method, we consider the ($\lambda$+$\lambda$)-ENAS algorithms with different mutation operators and estimate the lower bounds of the EHT. Furthermore, we study the EHT on the NAS-Bench-101 problem, and the results demonstrate the validity of the proposed method. To the best of our knowledge, this work is the first attempt to establish a theoretical foundation for ENAS algorithms.

Related papers

• A Pairwise Comparison Relation-assisted Multi-objective Evolutionary Neural Architecture Search Method with Multi-population Mechanism [58.855741970337675]
  Neural architecture search (NAS) enables re-searchers to automatically explore vast search spaces and find efficient neural networks. NAS suffers from a key bottleneck, i.e., numerous architectures need to be evaluated during the search process. We propose the SMEM-NAS, a pairwise com-parison relation-assisted multi-objective evolutionary algorithm based on a multi-population mechanism.
  arXiv  Detail & Related papers  (2024-07-22T12:46:22Z)
• GPT-NAS: Evolutionary Neural Architecture Search with the Generative Pre-Trained Model [25.187467297581073]
  This work presents a novel architecture search algorithm, called GPT-NAS, that optimize neural architectures by Generative Pre-Trained (GPT) model. In GPT-NAS, we assume that a generative model pre-trained on a large-scale corpus could learn the fundamental law of building neural architectures. Our GPT-NAS method significantly outperforms seven manually designed neural architectures and thirteen architectures provided by competing NAS methods.
  arXiv  Detail & Related papers  (2023-05-09T11:29:42Z)
• PRE-NAS: Predictor-assisted Evolutionary Neural Architecture Search [34.06028035262884]
  We propose a novel evolutionary-based NAS strategy, Predictor-assisted E-NAS (PRE-NAS) PRE-NAS leverages new evolutionary search strategies and integrates high-fidelity weight inheritance over generations. Experiments on NAS-Bench-201 and DARTS search spaces show that PRE-NAS can outperform state-of-the-art NAS methods.
  arXiv  Detail & Related papers  (2022-04-27T06:40:39Z)
• $\beta$-DARTS: Beta-Decay Regularization for Differentiable Architecture Search [85.84110365657455]
  We propose a simple-but-efficient regularization method, termed as Beta-Decay, to regularize the DARTS-based NAS searching process. Experimental results on NAS-Bench-201 show that our proposed method can help to stabilize the searching process and makes the searched network more transferable across different datasets.
  arXiv  Detail & Related papers  (2022-03-03T11:47:14Z)
• Lessons from the Clustering Analysis of a Search Space: A Centroid-based Approach to Initializing NAS [12.901952926144258]
  Recent availability of NAS benchmarks have enabled low computational resources prototyping. A calibrated clustering analysis of the search space is performed. Second, the centroids are extracted and used to initialize a NAS algorithm.
  arXiv  Detail & Related papers  (2021-08-20T11:46:33Z)
• iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients [75.41173109807735]
  Differentiable ARchiTecture Search (DARTS) has recently become the mainstream of neural architecture search (NAS) We tackle the hypergradient computation in DARTS based on the implicit function theorem. We show that the architecture optimisation with the proposed method, named iDARTS, is expected to converge to a stationary point.
  arXiv  Detail & Related papers  (2021-06-21T00:44:11Z)
• On the performance of deep learning for numerical optimization: an application to protein structure prediction [0.0]
  We present a study on the performance of the deep learning models to deal with global optimization problems. The proposed approach adopts the idea of the neural architecture search (NAS) to generate efficient neural networks. Experiments reveal that the generated learning models can achieve competitive results when compared to hand-designed algorithms.
  arXiv  Detail & Related papers  (2020-12-17T17:01:30Z)
• Neural Architecture Search of SPD Manifold Networks [79.45110063435617]
  We propose a new neural architecture search (NAS) problem of Symmetric Positive Definite (SPD) manifold networks. We first introduce a geometrically rich and diverse SPD neural architecture search space for an efficient SPD cell design. We exploit a differentiable NAS algorithm on our relaxed continuous search space for SPD neural architecture search.
  arXiv  Detail & Related papers  (2020-10-27T18:08:57Z)
• DrNAS: Dirichlet Neural Architecture Search [88.56953713817545]
  We treat the continuously relaxed architecture mixing weight as random variables, modeled by Dirichlet distribution. With recently developed pathwise derivatives, the Dirichlet parameters can be easily optimized with gradient-based generalization. To alleviate the large memory consumption of differentiable NAS, we propose a simple yet effective progressive learning scheme.
  arXiv  Detail & Related papers  (2020-06-18T08:23:02Z)
• Sampled Training and Node Inheritance for Fast Evolutionary Neural Architecture Search [22.483917379706725]
  evolutionary neural architecture search (ENAS) has received increasing attention due to the attractive global optimization capability of evolutionary algorithms. This paper proposes a new framework for fast ENAS based on directed acyclic graph, in which parents are randomly sampled and trained on each mini-batch of training data. We evaluate the proposed algorithm on the widely used datasets, in comparison with 26 state-of-the-art peer algorithms.
  arXiv  Detail & Related papers  (2020-03-07T12:33:01Z)
• DDPNAS: Efficient Neural Architecture Search via Dynamic Distribution Pruning [135.27931587381596]
  We propose an efficient and unified NAS framework termed DDPNAS via dynamic distribution pruning. In particular, we first sample architectures from a joint categorical distribution. Then the search space is dynamically pruned and its distribution is updated every few epochs. With the proposed efficient network generation method, we directly obtain the optimal neural architectures on given constraints.
  arXiv  Detail & Related papers  (2019-05-28T06:35:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.