Neural Architecture Ranker
- URL: http://arxiv.org/abs/2201.12725v1
- Date: Sun, 30 Jan 2022 04:54:59 GMT
- Title: Neural Architecture Ranker
- Authors: Bicheng Guo, Shibo He, Tao Chen, Jiming Chen, Peng Ye
- Abstract summary: Architecture ranking has recently been advocated to design an efficient and effective performance predictor for Neural Architecture Search (NAS)
Inspired by the stratification stratification, we propose a predictor, namely Neural Ranker (NAR)
- Score: 19.21631623578852
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Architecture ranking has recently been advocated to design an efficient and
effective performance predictor for Neural Architecture Search (NAS). The
previous contrastive method solves the ranking problem by comparing pairs of
architectures and predicting their relative performance, which may suffer
generalization issues due to local pair-wise comparison. Inspired by the
quality stratification phenomenon in the search space, we propose a predictor,
namely Neural Architecture Ranker (NAR), from a new and global perspective by
exploiting the quality distribution of the whole search space. The NAR learns
the similar characteristics of the same quality tier (i.e., level) and
distinguishes among different individuals by first matching architectures with
the representation of tiers, and then classifying and scoring them. It can
capture the features of different quality tiers and thus generalize its ranking
ability to the entire search space. Besides, distributions of different quality
tiers are also beneficial to guide the sampling procedure, which is free of
training a search algorithm and thus simplifies the NAS pipeline. The proposed
NAR achieves better performance than the state-of-the-art methods on two widely
accepted datasets. On NAS-Bench-101, it finds the architectures with top
0.01$\unicode{x2030}$ performance among the search space and stably focuses on
the top architectures. On NAS-Bench-201, it identifies the optimal
architectures on CIFAR-10, CIFAR-100 and, ImageNet-16-120. We expand and
release these two datasets covering detailed cell computational information to
boost the study of NAS.
Related papers
- A General-Purpose Transferable Predictor for Neural Architecture Search [22.883809911265445]
We propose a general-purpose neural predictor for Neural Architecture Search (NAS) that can transfer across search spaces.
Experimental results on NAS-Bench-101, 201 and 301 demonstrate the efficacy of our scheme.
arXiv Detail & Related papers (2023-02-21T17:28:05Z) - BaLeNAS: Differentiable Architecture Search via the Bayesian Learning
Rule [95.56873042777316]
Differentiable Architecture Search (DARTS) has received massive attention in recent years, mainly because it significantly reduces the computational cost.
This paper formulates the neural architecture search as a distribution learning problem through relaxing the architecture weights into Gaussian distributions.
We demonstrate how the differentiable NAS benefits from Bayesian principles, enhancing exploration and improving stability.
arXiv Detail & Related papers (2021-11-25T18:13:42Z) - ZARTS: On Zero-order Optimization for Neural Architecture Search [94.41017048659664]
Differentiable architecture search (DARTS) has been a popular one-shot paradigm for NAS due to its high efficiency.
This work turns to zero-order optimization and proposes a novel NAS scheme, called ZARTS, to search without enforcing the above approximation.
In particular, results on 12 benchmarks verify the outstanding robustness of ZARTS, where the performance of DARTS collapses due to its known instability issue.
arXiv Detail & Related papers (2021-10-10T09:35:15Z) - RankNAS: Efficient Neural Architecture Search by Pairwise Ranking [30.890612901949307]
We propose a performance ranking method (RankNAS) via pairwise ranking.
It enables efficient architecture search using much fewer training examples.
It can design high-performance architectures while being orders of magnitude faster than state-of-the-art NAS systems.
arXiv Detail & Related papers (2021-09-15T15:43:08Z) - AceNAS: Learning to Rank Ace Neural Architectures with Weak Supervision
of Weight Sharing [6.171090327531059]
We introduce Learning to Rank methods to select the best (ace) architectures from a space.
We also propose to leverage weak supervision from weight sharing by pretraining architecture representation on weak labels obtained from the super-net.
Experiments on NAS benchmarks and large-scale search spaces demonstrate that our approach outperforms SOTA with a significantly reduced search cost.
arXiv Detail & Related papers (2021-08-06T08:31:42Z) - Contrastive Neural Architecture Search with Neural Architecture
Comparators [46.45102111497492]
One of the key steps in Neural Architecture Search (NAS) is to estimate the performance of candidate architectures.
Existing methods either directly use the validation performance or learn a predictor to estimate the performance.
We propose a novel Contrastive Neural Architecture Search (CTNAS) method which performs architecture search by taking the comparison results between architectures as the reward.
arXiv Detail & Related papers (2021-03-08T11:24:07Z) - Weak NAS Predictors Are All You Need [91.11570424233709]
Recent predictor-based NAS approaches attempt to solve the problem with two key steps: sampling some architecture-performance pairs and fitting a proxy accuracy predictor.
We shift the paradigm from finding a complicated predictor that covers the whole architecture space to a set of weaker predictors that progressively move towards the high-performance sub-space.
Our method costs fewer samples to find the top-performance architectures on NAS-Bench-101 and NAS-Bench-201, and it achieves the state-of-the-art ImageNet performance on the NASNet search space.
arXiv Detail & Related papers (2021-02-21T01:58:43Z) - Hierarchical Neural Architecture Search for Deep Stereo Matching [131.94481111956853]
We propose the first end-to-end hierarchical NAS framework for deep stereo matching.
Our framework incorporates task-specific human knowledge into the neural architecture search framework.
It is ranked at the top 1 accuracy on KITTI stereo 2012, 2015 and Middlebury benchmarks, as well as the top 1 on SceneFlow dataset.
arXiv Detail & Related papers (2020-10-26T11:57:37Z) - DC-NAS: Divide-and-Conquer Neural Architecture Search [108.57785531758076]
We present a divide-and-conquer (DC) approach to effectively and efficiently search deep neural architectures.
We achieve a $75.1%$ top-1 accuracy on the ImageNet dataset, which is higher than that of state-of-the-art methods using the same search space.
arXiv Detail & Related papers (2020-05-29T09:02:16Z) - A Semi-Supervised Assessor of Neural Architectures [157.76189339451565]
We employ an auto-encoder to discover meaningful representations of neural architectures.
A graph convolutional neural network is introduced to predict the performance of architectures.
arXiv Detail & Related papers (2020-05-14T09:02:33Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.