Extensible Proxy for Efficient NAS
- URL: http://arxiv.org/abs/2210.09459v1
- Date: Mon, 17 Oct 2022 22:18:22 GMT
- Title: Extensible Proxy for Efficient NAS
- Authors: Yuhong Li, Jiajie Li, Cong Han, Pan Li, Jinjun Xiong, Deming Chen
- Abstract summary: We propose a new approach to design deep neural networks (DNNs) called Neural Architecture Search (NAS)
NAS proxies are proposed to address the demanding computational issues of NAS, where each candidate architecture network only requires one iteration of backpropagation.
Our experiments confirm the effectiveness of both Eproxy and Eproxy+DPS.
- Score: 38.124755703499886
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Neural Architecture Search (NAS) has become a de facto approach in the recent
trend of AutoML to design deep neural networks (DNNs). Efficient or
near-zero-cost NAS proxies are further proposed to address the demanding
computational issues of NAS, where each candidate architecture network only
requires one iteration of backpropagation. The values obtained from the proxies
are considered the predictions of architecture performance on downstream tasks.
However, two significant drawbacks hinder the extended usage of Efficient NAS
proxies. (1) Efficient proxies are not adaptive to various search spaces. (2)
Efficient proxies are not extensible to multi-modality downstream tasks. Based
on the observations, we design a Extensible proxy (Eproxy) that utilizes
self-supervised, few-shot training (i.e., 10 iterations of backpropagation)
which yields near-zero costs. The key component that makes Eproxy efficient is
an untrainable convolution layer termed barrier layer that add the
non-linearities to the optimization spaces so that the Eproxy can discriminate
the performance of architectures in the early stage. Furthermore, to make
Eproxy adaptive to different downstream tasks/search spaces, we propose a
Discrete Proxy Search (DPS) to find the optimized training settings for Eproxy
with only handful of benchmarked architectures on the target tasks. Our
extensive experiments confirm the effectiveness of both Eproxy and Eproxy+DPS.
Code is available at https://github.com/leeyeehoo/GenNAS-Zero.
Related papers
- Zero-Shot NAS via the Suppression of Local Entropy Decrease [21.100745856699277]
Architecture performance evaluation is the most time-consuming part of neural architecture search (NAS)
Zero-Shot NAS accelerates the evaluation by utilizing zero-cost proxies instead of training.
architectural topologies are used to evaluate the performance of networks in this study.
arXiv Detail & Related papers (2024-11-09T17:36:53Z) - TG-NAS: Leveraging Zero-Cost Proxies with Transformer and Graph Convolution Networks for Efficient Neural Architecture Search [1.30891455653235]
TG-NAS aims to create training-free proxies for architecture performance prediction.
We introduce TG-NAS, a novel model-based universal proxy that leverages a transformer-based operator embedding generator and a graph convolution network (GCN) to predict architecture performance.
TG-NAS achieves up to 300X improvements in search efficiency compared to previous SOTA ZC proxy methods.
arXiv Detail & Related papers (2024-03-30T07:25:30Z) - AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search [30.64117903216323]
Training-free network architecture search (NAS) aims to discover high-performing networks with zero-cost proxies.
We propose AZ-NAS, a novel approach that leverages the ensemble of various zero-cost proxies to enhance the correlation between a predicted ranking of networks and the ground truth.
Results conclusively demonstrate the efficacy and efficiency of AZ-NAS, outperforming state-of-the-art methods on standard benchmarks.
arXiv Detail & Related papers (2024-03-28T08:44:36Z) - Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunities [58.67514819895494]
Key idea behind zero-shot NAS approaches is to design proxies that can predict the accuracy of some given networks without training the network parameters.
This paper aims to comprehensively review and compare the state-of-the-art (SOTA) zero-shot NAS approaches.
arXiv Detail & Related papers (2023-07-05T03:07:00Z) - $\beta$-DARTS++: Bi-level Regularization for Proxy-robust Differentiable
Architecture Search [96.99525100285084]
Regularization method, Beta-Decay, is proposed to regularize the DARTS-based NAS searching process (i.e., $beta$-DARTS)
In-depth theoretical analyses on how it works and why it works are provided.
arXiv Detail & Related papers (2023-01-16T12:30:32Z) - Non-isotropy Regularization for Proxy-based Deep Metric Learning [78.18860829585182]
We propose non-isotropy regularization ($mathbbNIR$) for proxy-based Deep Metric Learning.
This allows us to explicitly induce a non-isotropic distribution of samples around a proxy to optimize for.
Experiments highlight consistent generalization benefits of $mathbbNIR$ while achieving competitive and state-of-the-art performance.
arXiv Detail & Related papers (2022-03-16T11:13:20Z) - ProxyBO: Accelerating Neural Architecture Search via Bayesian
Optimization with Zero-cost Proxies [30.059154132130207]
We present ProxyBO, an efficient Bayesian optimization framework that utilizes zero-cost proxies to accelerate neural architecture search.
We show that ProxyBO consistently outperforms competitive baselines on five tasks from three public benchmarks.
arXiv Detail & Related papers (2021-10-20T08:18:16Z) - OPANAS: One-Shot Path Aggregation Network Architecture Search for Object
Detection [82.04372532783931]
Recently, neural architecture search (NAS) has been exploited to design feature pyramid networks (FPNs)
We propose a novel One-Shot Path Aggregation Network Architecture Search (OPANAS) algorithm, which significantly improves both searching efficiency and detection accuracy.
arXiv Detail & Related papers (2021-03-08T01:48:53Z) - EcoNAS: Finding Proxies for Economical Neural Architecture Search [130.59673917196994]
In this paper, we observe that most existing proxies exhibit different behaviors in maintaining the rank consistency among network candidates.
Inspired by these observations, we present a reliable proxy and further formulate a hierarchical proxy strategy.
The strategy spends more computations on candidate networks that are potentially more accurate, while discards unpromising ones in early stage with a fast proxy.
arXiv Detail & Related papers (2020-01-05T13:29:02Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.