Asynchronous Evolution of Deep Neural Network Architectures
- URL: http://arxiv.org/abs/2308.04102v3
- Date: Mon, 1 Jan 2024 14:16:18 GMT
- Title: Asynchronous Evolution of Deep Neural Network Architectures
- Authors: Jason Liang, Hormoz Shahrzad, Risto Miikkulainen
- Abstract summary: Many evolutionary algorithms (EAs) take advantage of parallel evaluation of candidates.
If evaluation times vary significantly, many worker nodes (i.e., compute clients) are idle much of the time, waiting for the next generation to be created.
This paper proposes a generic asynchronous evaluation strategy (AES) that is then adapted to work with ENAS.
- Score: 10.60691612679966
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Many evolutionary algorithms (EAs) take advantage of parallel evaluation of
candidates. However, if evaluation times vary significantly, many worker nodes
(i.e.,\ compute clients) are idle much of the time, waiting for the next
generation to be created. Evolutionary neural architecture search (ENAS), a
class of EAs that optimizes the architecture and hyperparameters of deep neural
networks, is particularly vulnerable to this issue. This paper proposes a
generic asynchronous evaluation strategy (AES) that is then adapted to work
with ENAS. AES increases throughput by maintaining a queue of up to $K$
individuals ready to be sent to the workers for evaluation and proceeding to
the next generation as soon as $M<<K$ individuals have been evaluated. A
suitable value for $M$ is determined experimentally, balancing diversity and
efficiency. To showcase the generality and power of AES, it was first evaluated
in eight-line sorting network design (a single-population optimization task
with limited evaluation-time variability), achieving an over two-fold speedup.
Next, it was evaluated in 11-bit multiplexer design (a single-population
discovery task with extended variability), where a 14-fold speedup was
observed. It was then scaled up to ENAS for image captioning (a
multi-population open-ended-optimization task), resulting in an over two-fold
speedup. In all problems, a multifold performance improvement was observed,
suggesting that AES is a promising method for parallelizing the evolution of
complex systems with long and variable evaluation times, such as those in ENAS.
Related papers
- Runtime Analysis of Evolutionary NAS for Multiclass Classification [25.67863839453669]
We consider (1+1)-ENAS algorithms with one-bit and bit-wise mutations, and analyze their upper and lower bounds on the expected runtime.<n>We prove that the algorithm using both mutations can find the optimum with the expected runtime upper bound of $O(rMlnrM)$ and lower bound of $Omega(rMlnM)$.
arXiv Detail & Related papers (2025-06-06T12:09:30Z) - A Pairwise Comparison Relation-assisted Multi-objective Evolutionary Neural Architecture Search Method with Multi-population Mechanism [58.855741970337675]
Neural architecture search (NAS) enables re-searchers to automatically explore vast search spaces and find efficient neural networks.
NAS suffers from a key bottleneck, i.e., numerous architectures need to be evaluated during the search process.
We propose the SMEM-NAS, a pairwise com-parison relation-assisted multi-objective evolutionary algorithm based on a multi-population mechanism.
arXiv Detail & Related papers (2024-07-22T12:46:22Z) - Towards Running Time Analysis of Interactive Multi-objective
Evolutionary Algorithms [23.815981112784552]
This paper provides the first running time analysis (the essential theoretical aspect of EAs) for practical iMOEAs.
We prove that the expected running time of the well-developed interactive NSGA-II for solving the OneMinMax and OneJumpZeroJump problems is $O(n log n)$ and $O(nk)$, respectively.
arXiv Detail & Related papers (2023-10-12T14:57:47Z) - Neural Architecture Search as Multiobjective Optimization Benchmarks:
Problem Formulation and Performance Assessment [30.264524448340406]
We formulate neural architecture search (NAS) tasks into general multi-objective optimization problems.
We analyze the complex characteristics from an optimization point of view.
We present an end-to-end pipeline, dubbed $texttEvoXBench$, to generate benchmark test problems for EMO algorithms to run efficiently.
arXiv Detail & Related papers (2022-08-08T02:07:49Z) - B2EA: An Evolutionary Algorithm Assisted by Two Bayesian Optimization
Modules for Neural Architecture Search [3.126118485851773]
We develop Btextsuperscript2EA that is a surrogate assisted EA with two BO surrogate models and a mutation step in between.
Btextsuperscript2EA is robust and efficient over the 14 benchmarks for three difficulty levels of target performance.
arXiv Detail & Related papers (2022-02-07T08:50:21Z) - Elastic Architecture Search for Diverse Tasks with Different Resources [87.23061200971912]
We study a new challenging problem of efficient deployment for diverse tasks with different resources, where the resource constraint and task of interest corresponding to a group of classes are dynamically specified at testing time.
Previous NAS approaches seek to design architectures for all classes simultaneously, which may not be optimal for some individual tasks.
We present a novel and general framework, called Elastic Architecture Search (EAS), permitting instant specializations at runtime for diverse tasks with various resource constraints.
arXiv Detail & Related papers (2021-08-03T00:54:27Z) - An Efficient Asynchronous Method for Integrating Evolutionary and
Gradient-based Policy Search [76.73477450555046]
We introduce an Asynchronous Evolution Strategy-Reinforcement Learning (AES-RL) that maximizes the parallel efficiency of ES and integrates it with policy gradient methods.
Specifically, we propose 1) a novel framework to merge ES and DRL asynchronously and 2) various asynchronous update methods that can take all advantages of asynchronism, ES, and DRL.
arXiv Detail & Related papers (2020-12-10T02:30:48Z) - Multi-objective Neural Architecture Search with Almost No Training [9.93048700248444]
We propose an effective alternative, dubbed Random-Weight Evaluation (RWE), to rapidly estimate the performance of network architectures.
RWE reduces the computational cost of evaluating an architecture from hours to seconds.
When integrated within an evolutionary multi-objective algorithm, RWE obtains a set of efficient architectures with state-of-the-art performance on CIFAR-10 with less than two hours' searching on a single GPU card.
arXiv Detail & Related papers (2020-11-27T07:39:17Z) - Rethinking Performance Estimation in Neural Architecture Search [191.08960589460173]
We provide a novel yet systematic rethinking of performance estimation (PE) in a resource constrained regime.
By combining BPE with various search algorithms including reinforcement learning, evolution algorithm, random search, and differentiable architecture search, we achieve 1, 000x of NAS speed up with a negligible performance drop.
arXiv Detail & Related papers (2020-05-20T09:01:44Z) - Communication-Efficient Distributed Stochastic AUC Maximization with
Deep Neural Networks [50.42141893913188]
We study a distributed variable for large-scale AUC for a neural network as with a deep neural network.
Our model requires a much less number of communication rounds and still a number of communication rounds in theory.
Our experiments on several datasets show the effectiveness of our theory and also confirm our theory.
arXiv Detail & Related papers (2020-05-05T18:08:23Z) - MTL-NAS: Task-Agnostic Neural Architecture Search towards
General-Purpose Multi-Task Learning [71.90902837008278]
We propose to incorporate neural architecture search (NAS) into general-purpose multi-task learning (GP-MTL)
In order to adapt to different task combinations, we disentangle the GP-MTL networks into single-task backbones.
We also propose a novel single-shot gradient-based search algorithm that closes the performance gap between the searched architectures.
arXiv Detail & Related papers (2020-03-31T09:49:14Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.