Neural Architecture Search using Covariance Matrix Adaptation Evolution Strategy

• URL: http://arxiv.org/abs/2107.07266v1
• Date: Thu, 15 Jul 2021 11:41:23 GMT
• Title: Neural Architecture Search using Covariance Matrix Adaptation Evolution Strategy
• Authors: Nilotpal Sinha, Kuan-Wen Chen
• Abstract summary: We propose a framework of applying the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to the neural architecture search problem called CMANAS. The architecture are modelled using a normal distribution, which is updated using CMA-ES based on the fitness of the sampled population. CMANAS finished the architecture search on CIFAR-10 with the top-1 test accuracy of 97.44% in 0.45 GPU day and on CIFAR-100 with the top-1 test accuracy of 83.24% for 0.6 GPU day on a single GPU.
• Score: 6.8129169853808795
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Evolution-based neural architecture search requires high computational resources, resulting in long search time. In this work, we propose a framework of applying the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to the neural architecture search problem called CMANAS, which achieves better results than previous evolution-based methods while reducing the search time significantly. The architectures are modelled using a normal distribution, which is updated using CMA-ES based on the fitness of the sampled population. We used the accuracy of a trained one shot model (OSM) on the validation data as a prediction of the fitness of an individual architecture to reduce the search time. We also used an architecture-fitness table (AF table) for keeping record of the already evaluated architecture, thus further reducing the search time. CMANAS finished the architecture search on CIFAR-10 with the top-1 test accuracy of 97.44% in 0.45 GPU day and on CIFAR-100 with the top-1 test accuracy of 83.24% for 0.6 GPU day on a single GPU. The top architectures from the searches on CIFAR-10 and CIFAR-100 were then transferred to ImageNet, achieving the top-5 accuracy of 92.6% and 92.1%, respectively.

Related papers

• A Pairwise Comparison Relation-assisted Multi-objective Evolutionary Neural Architecture Search Method with Multi-population Mechanism [58.855741970337675]
  Neural architecture search (NAS) enables re-searchers to automatically explore vast search spaces and find efficient neural networks. NAS suffers from a key bottleneck, i.e., numerous architectures need to be evaluated during the search process. We propose the SMEM-NAS, a pairwise com-parison relation-assisted multi-objective evolutionary algorithm based on a multi-population mechanism.
  arXiv  Detail & Related papers  (2024-07-22T12:46:22Z)
• NAAP-440 Dataset and Baseline for Neural Architecture Accuracy Prediction [1.2183405753834562]
  We introduce the NAAP-440 dataset of 440 neural architectures, which were trained on CIFAR10 using a fixed recipe. Experiments indicate that by using off-the-shelf regression algorithms and running up to 10% of the training process, not only is it possible to predict an architecture's accuracy rather precisely. This approach may serve as a powerful tool for accelerating NAS-based studies and thus dramatically increase their efficiency.
  arXiv  Detail & Related papers  (2022-09-14T13:21:39Z)
• ZARTS: On Zero-order Optimization for Neural Architecture Search [94.41017048659664]
  Differentiable architecture search (DARTS) has been a popular one-shot paradigm for NAS due to its high efficiency. This work turns to zero-order optimization and proposes a novel NAS scheme, called ZARTS, to search without enforcing the above approximation. In particular, results on 12 benchmarks verify the outstanding robustness of ZARTS, where the performance of DARTS collapses due to its known instability issue.
  arXiv  Detail & Related papers  (2021-10-10T09:35:15Z)
• Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search [84.4140192638394]
  Most differentiable neural architecture search methods construct a super-net for search and derive a target-net as its sub-graph for evaluation. In this paper, we introduce EnTranNAS that is composed of Engine-cells and Transit-cells. Our method also spares much memory and computation cost, which speeds up the search process.
  arXiv  Detail & Related papers  (2021-01-27T12:16:47Z)
• Evolving Neural Architecture Using One Shot Model [5.188825486231326]
  We propose a novel way of applying a simple genetic algorithm to the NAS problem called EvNAS (Evolving Neural Architecture using One Shot Model) EvNAS searches for the architecture on the proxy dataset i.e. CIFAR-10 for 4.4 GPU day on a single GPU and achieves top-1 test error of 2.47%. Results show the potential of evolutionary methods in solving the architecture search problem.
  arXiv  Detail & Related papers  (2020-12-23T08:40:53Z)
• ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding [86.40042104698792]
  We formulate neural architecture search as a sparse coding problem. In experiments, our two-stage method on CIFAR-10 requires only 0.05 GPU-day for search. Our one-stage method produces state-of-the-art performances on both CIFAR-10 and ImageNet at the cost of only evaluation time.
  arXiv  Detail & Related papers  (2020-10-13T04:34:24Z)
• Off-Policy Reinforcement Learning for Efficient and Effective GAN Architecture Search [50.40004966087121]
  We introduce a new reinforcement learning based neural architecture search (NAS) methodology for generative adversarial network (GAN) architecture search. The key idea is to formulate the GAN architecture search problem as a Markov decision process (MDP) for smoother architecture sampling. We exploit an off-policy GAN architecture search algorithm that makes efficient use of the samples generated by previous policies.
  arXiv  Detail & Related papers  (2020-07-17T18:29:17Z)
• Multi-Objective Neural Architecture Search Based on Diverse Structures and Adaptive Recommendation [4.595675084986132]
  The search space of neural architecture search (NAS) for convolutional neural network (CNN) is huge. We propose MoARR algorithm, which utilizes the existing research results and historical information to quickly find architectures that are both lightweight and accurate. Experimental results show that our MoARR can achieve a powerful and lightweight model (with 1.9% error rate and 2.3M parameters) on CIFAR-10 in 6 GPU hours.
  arXiv  Detail & Related papers  (2020-07-06T13:42:33Z)
• Cyclic Differentiable Architecture Search [99.12381460261841]
  Differentiable ARchiTecture Search, i.e., DARTS, has drawn great attention in neural architecture search. We propose new joint objectives and a novel Cyclic Differentiable ARchiTecture Search framework, dubbed CDARTS. In the DARTS search space, we achieve 97.52% top-1 accuracy on CIFAR10 and 76.3% top-1 accuracy on ImageNet.
  arXiv  Detail & Related papers  (2020-06-18T17:55:19Z)
• ADWPNAS: Architecture-Driven Weight Prediction for Neural Architecture Search [6.458169480971417]
  We propose an Architecture-Driven Weight Prediction (ADWP) approach for neural architecture search (NAS) In our approach, we first design an architecture-intensive search space and then train a HyperNetwork by inputting encoding architecture parameters. Results show that one search procedure can be completed in 4.0 GPU hours on CIFAR-10.
  arXiv  Detail & Related papers  (2020-03-03T05:06:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.