ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxies

• URL: http://arxiv.org/abs/2110.10423v1
• Date: Wed, 20 Oct 2021 08:18:16 GMT
• Title: ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxies
• Authors: Yu Shen, Yang Li, Jian Zheng, Wentao Zhang, Peng Yao, Jixiang Li, Sen Yang, Ji Liu, Cui Bin
• Abstract summary: We present ProxyBO, an efficient Bayesian optimization framework that utilizes zero-cost proxies to accelerate neural architecture search. We show that ProxyBO consistently outperforms competitive baselines on five tasks from three public benchmarks.
• Score: 30.059154132130207
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Designing neural architectures requires immense manual efforts. This has promoted the development of neural architecture search (NAS) to automate this design. While previous NAS methods achieve promising results but run slowly and zero-cost proxies run extremely fast but are less promising, recent work considers utilizing zero-cost proxies via a simple warm-up. The existing method has two limitations, which are unforeseeable reliability and one-shot usage. To address the limitations, we present ProxyBO, an efficient Bayesian optimization framework that utilizes the zero-cost proxies to accelerate neural architecture search. We propose the generalization ability measurement to estimate the fitness of proxies on the task during each iteration and then combine BO with zero-cost proxies via dynamic influence combination. Extensive empirical studies show that ProxyBO consistently outperforms competitive baselines on five tasks from three public benchmarks. Concretely, ProxyBO achieves up to 5.41x and 3.83x speedups over the state-of-the-art approach REA and BRP-NAS, respectively.

Related papers

• TG-NAS: Leveraging Zero-Cost Proxies with Transformer and Graph Convolution Networks for Efficient Neural Architecture Search [1.30891455653235]
  TG-NAS aims to create training-free proxies for architecture performance prediction. We introduce TG-NAS, a novel model-based universal proxy that leverages a transformer-based operator embedding generator and a graph convolution network (GCN) to predict architecture performance. TG-NAS achieves up to 300X improvements in search efficiency compared to previous SOTA ZC proxy methods.
  arXiv  Detail & Related papers  (2024-03-30T07:25:30Z)
• AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search [30.64117903216323]
  Training-free network architecture search (NAS) aims to discover high-performing networks with zero-cost proxies. We propose AZ-NAS, a novel approach that leverages the ensemble of various zero-cost proxies to enhance the correlation between a predicted ranking of networks and the ground truth. Results conclusively demonstrate the efficacy and efficiency of AZ-NAS, outperforming state-of-the-art methods on standard benchmarks.
  arXiv  Detail & Related papers  (2024-03-28T08:44:36Z)
• Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunities [58.67514819895494]
  Key idea behind zero-shot NAS approaches is to design proxies that can predict the accuracy of some given networks without training the network parameters. This paper aims to comprehensively review and compare the state-of-the-art (SOTA) zero-shot NAS approaches.
  arXiv  Detail & Related papers  (2023-07-05T03:07:00Z)
• ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradients [17.139381064317778]
  We propose a new zero-shot proxy, ZiCo, that works consistently better than #Params. ZiCo-based NAS can find optimal architectures with 78.1%, 79.4%, and 80.4% test accuracy under inference budgets of 450M, 600M, and 1000M FLOPs, respectively.
  arXiv  Detail & Related papers  (2023-01-26T18:38:56Z)
• $\beta$-DARTS++: Bi-level Regularization for Proxy-robust Differentiable Architecture Search [96.99525100285084]
  Regularization method, Beta-Decay, is proposed to regularize the DARTS-based NAS searching process (i.e., $beta$-DARTS) In-depth theoretical analyses on how it works and why it works are provided.
  arXiv  Detail & Related papers  (2023-01-16T12:30:32Z)
• Extensible Proxy for Efficient NAS [38.124755703499886]
  We propose a new approach to design deep neural networks (DNNs) called Neural Architecture Search (NAS) NAS proxies are proposed to address the demanding computational issues of NAS, where each candidate architecture network only requires one iteration of backpropagation. Our experiments confirm the effectiveness of both Eproxy and Eproxy+DPS.
  arXiv  Detail & Related papers  (2022-10-17T22:18:22Z)
• FNAS: Uncertainty-Aware Fast Neural Architecture Search [54.49650267859032]
  Reinforcement learning (RL)-based neural architecture search (NAS) generally guarantees better convergence yet suffers from the requirement of huge computational resources. We propose a general pipeline to accelerate the convergence of the rollout process as well as the RL process in NAS. Experiments on the Mobile Neural Architecture Search (MNAS) search space show the proposed Fast Neural Architecture Search (FNAS) accelerates standard RL-based NAS process by 10x.
  arXiv  Detail & Related papers  (2021-05-25T06:32:52Z)
• Speedy Performance Estimation for Neural Architecture Search [47.683124540824515]
  We propose to estimate the final test performance based on a simple measure of training speed. Our estimator is theoretically motivated by the connection between generalisation and training speed.
  arXiv  Detail & Related papers  (2020-06-08T11:48:09Z)
• BNAS:An Efficient Neural Architecture Search Approach Using Broad Scalable Architecture [62.587982139871976]
  We propose Broad Neural Architecture Search (BNAS) where we elaborately design broad scalable architecture dubbed Broad Convolutional Neural Network (BCNN) BNAS delivers 0.19 days which is 2.37x less expensive than ENAS who ranks the best in reinforcement learning-based NAS approaches.
  arXiv  Detail & Related papers  (2020-01-18T15:07:55Z)
• EcoNAS: Finding Proxies for Economical Neural Architecture Search [130.59673917196994]
  In this paper, we observe that most existing proxies exhibit different behaviors in maintaining the rank consistency among network candidates. Inspired by these observations, we present a reliable proxy and further formulate a hierarchical proxy strategy. The strategy spends more computations on candidate networks that are potentially more accurate, while discards unpromising ones in early stage with a fast proxy.
  arXiv  Detail & Related papers  (2020-01-05T13:29:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.