Prior-Guided One-shot Neural Architecture Search
- URL: http://arxiv.org/abs/2206.13329v1
- Date: Mon, 27 Jun 2022 14:19:56 GMT
- Title: Prior-Guided One-shot Neural Architecture Search
- Authors: Peijie Dong, Xin Niu, Lujun Li, Linzhen Xie, Wenbin Zou, Tian Ye,
Zimian Wei, Hengyue Pan
- Abstract summary: We present Prior-Guided One-shot NAS (PGONAS) to strengthen the ranking correlation of supernets.
Our PGONAS ranks 3rd place in the supernet Track Track of CVPR2022 Second lightweight NAS challenge.
- Score: 11.609732776776982
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Neural architecture search methods seek optimal candidates with efficient
weight-sharing supernet training. However, recent studies indicate poor ranking
consistency about the performance between stand-alone architectures and
shared-weight networks. In this paper, we present Prior-Guided One-shot NAS
(PGONAS) to strengthen the ranking correlation of supernets. Specifically, we
first explore the effect of activation functions and propose a balanced
sampling strategy based on the Sandwich Rule to alleviate weight coupling in
the supernet. Then, FLOPs and Zen-Score are adopted to guide the training of
supernet with ranking correlation loss. Our PGONAS ranks 3rd place in the
supernet Track Track of CVPR2022 Second lightweight NAS challenge. Code is
available in
https://github.com/pprp/CVPR2022-NAS?competition-Track1-3th-solution.
Related papers
- Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Experts [55.470959564665705]
Weight-sharing supernets are crucial for performance estimation in cutting-edge neural search frameworks.
The proposed method attains state-of-the-art (SoTA) performance in NAS for fast machine translation models.
It excels in NAS for building memory-efficient task-agnostic BERT models.
arXiv Detail & Related papers (2023-06-08T00:35:36Z) - Improve Ranking Correlation of Super-net through Training Scheme from
One-shot NAS to Few-shot NAS [13.390484379343908]
We propose a step-by-step training super-net scheme from one-shot NAS to few-shot NAS.
In the training scheme, we firstly train super-net in a one-shot way, and then we disentangle the weights of super-net.
Our method ranks 4th place in the CVPR2022 3rd Lightweight NAS Challenge Track1.
arXiv Detail & Related papers (2022-06-13T04:02:12Z) - An Analysis of Super-Net Heuristics in Weight-Sharing NAS [70.57382341642418]
We show that simple random search achieves competitive performance to complex state-of-the-art NAS algorithms when the super-net is properly trained.
We show that simple random search achieves competitive performance to complex state-of-the-art NAS algorithms when the super-net is properly trained.
arXiv Detail & Related papers (2021-10-04T02:18:44Z) - Pi-NAS: Improving Neural Architecture Search by Reducing Supernet
Training Consistency Shift [128.32670289503025]
Recently proposed neural architecture search (NAS) methods co-train billions of architectures in a supernet and estimate their potential accuracy.
The ranking correlation between the architectures' predicted accuracy and their actual capability is incorrect, which causes the existing NAS methods' dilemma.
We attribute this ranking correlation problem to the supernet training consistency shift, including feature shift and parameter shift.
We address these two shifts simultaneously using a nontrivial supernet-Pi model, called Pi-NAS.
arXiv Detail & Related papers (2021-08-22T09:08:48Z) - Improving Ranking Correlation of Supernet with Candidates Enhancement
and Progressive Training [8.373420721376739]
One-shot neural architecture search (NAS) applies weight-sharing supernet to reduce the unaffordable computation overhead of automated architecture designing.
We propose a candidates enhancement method and progressive training pipeline to improve the ranking correlation of supernet.
Our method ranks the 1st place in the Supernet Track of CVPR 2021 1st Lightweight NAS Challenge.
arXiv Detail & Related papers (2021-08-12T17:27:10Z) - K-shot NAS: Learnable Weight-Sharing for NAS with K-shot Supernets [52.983810997539486]
We introduce $K$-shot supernets and take their weights for each operation as a dictionary.
A textitsimplex-net is introduced to produce architecture-customized code for each path.
Experiments on benchmark datasets validate that K-shot NAS significantly improves the evaluation accuracy of paths.
arXiv Detail & Related papers (2021-06-11T14:57:36Z) - Landmark Regularization: Ranking Guided Super-Net Training in Neural
Architecture Search [70.57382341642418]
Weight sharing has become a de facto standard in neural architecture search because it enables the search to be done on commodity hardware.
Recent works have empirically shown a ranking disorder between the performance of stand-alone architectures and that of the corresponding shared-weight networks.
We propose a regularization term that aims to maximize the correlation between the performance rankings of the shared-weight network and that of the standalone architectures.
arXiv Detail & Related papers (2021-04-12T09:32:33Z) - AlphaNet: Improved Training of Supernet with Alpha-Divergence [28.171262066145616]
We propose to improve the supernet training with a more generalized alpha-divergence.
We apply the proposed alpha-divergence based supernet training to both slimmable neural networks and weight-sharing NAS.
Specifically, our discovered model family, AlphaNet, outperforms prior-art models on a wide range of FLOPs regimes.
arXiv Detail & Related papers (2021-02-16T04:23:55Z) - GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet [63.96959854429752]
GreedyNAS is easy-to-follow, and experimental results on ImageNet dataset indicate that it can achieve better Top-1 accuracy under same search space and FLOPs or latency level.
By searching on a larger space, our GreedyNAS can also obtain new state-of-the-art architectures.
arXiv Detail & Related papers (2020-03-25T06:54:10Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.