DAS: Neural Architecture Search via Distinguishing Activation Score
- URL: http://arxiv.org/abs/2212.12132v1
- Date: Fri, 23 Dec 2022 04:02:46 GMT
- Title: DAS: Neural Architecture Search via Distinguishing Activation Score
- Authors: Yuqiao Liu, Haipeng Li, Yanan Sun, Shuaicheng Liu
- Abstract summary: Neural Architecture Search (NAS) is an automatic technique that can search for well-performed architectures for a specific task.
We propose a dataset called Darts-training-bench (DTB), which fills the gap that no training states of architecture in existing datasets.
Our proposed method has 1.04$times$ - 1.56$times$ improvements on NAS-Bench-101, Network Design Spaces, and the proposed DTB.
- Score: 21.711985665733653
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Neural Architecture Search (NAS) is an automatic technique that can search
for well-performed architectures for a specific task. Although NAS surpasses
human-designed architecture in many fields, the high computational cost of
architecture evaluation it requires hinders its development. A feasible
solution is to directly evaluate some metrics in the initial stage of the
architecture without any training. NAS without training (WOT) score is such a
metric, which estimates the final trained accuracy of the architecture through
the ability to distinguish different inputs in the activation layer. However,
WOT score is not an atomic metric, meaning that it does not represent a
fundamental indicator of the architecture. The contributions of this paper are
in three folds. First, we decouple WOT into two atomic metrics which represent
the distinguishing ability of the network and the number of activation units,
and explore better combination rules named (Distinguishing Activation Score)
DAS. We prove the correctness of decoupling theoretically and confirmed the
effectiveness of the rules experimentally. Second, in order to improve the
prediction accuracy of DAS to meet practical search requirements, we propose a
fast training strategy. When DAS is used in combination with the fast training
strategy, it yields more improvements. Third, we propose a dataset called
Darts-training-bench (DTB), which fills the gap that no training states of
architecture in existing datasets. Our proposed method has 1.04$\times$ -
1.56$\times$ improvements on NAS-Bench-101, Network Design Spaces, and the
proposed DTB.
Related papers
- BaLeNAS: Differentiable Architecture Search via the Bayesian Learning
Rule [95.56873042777316]
Differentiable Architecture Search (DARTS) has received massive attention in recent years, mainly because it significantly reduces the computational cost.
This paper formulates the neural architecture search as a distribution learning problem through relaxing the architecture weights into Gaussian distributions.
We demonstrate how the differentiable NAS benefits from Bayesian principles, enhancing exploration and improving stability.
arXiv Detail & Related papers (2021-11-25T18:13:42Z) - AceNAS: Learning to Rank Ace Neural Architectures with Weak Supervision
of Weight Sharing [6.171090327531059]
We introduce Learning to Rank methods to select the best (ace) architectures from a space.
We also propose to leverage weak supervision from weight sharing by pretraining architecture representation on weak labels obtained from the super-net.
Experiments on NAS benchmarks and large-scale search spaces demonstrate that our approach outperforms SOTA with a significantly reduced search cost.
arXiv Detail & Related papers (2021-08-06T08:31:42Z) - Rapid Neural Architecture Search by Learning to Generate Graphs from
Datasets [42.993720854755736]
We propose an efficient Neural Search (NAS) framework that is trained once on a database consisting of datasets and pretrained networks.
We show that our model meta-learned on subsets of ImageNet-1K and architectures from NAS-Bench 201 search space successfully generalizes to multiple unseen datasets.
arXiv Detail & Related papers (2021-07-02T06:33:59Z) - iDARTS: Differentiable Architecture Search with Stochastic Implicit
Gradients [75.41173109807735]
Differentiable ARchiTecture Search (DARTS) has recently become the mainstream of neural architecture search (NAS)
We tackle the hypergradient computation in DARTS based on the implicit function theorem.
We show that the architecture optimisation with the proposed method, named iDARTS, is expected to converge to a stationary point.
arXiv Detail & Related papers (2021-06-21T00:44:11Z) - The Nonlinearity Coefficient -- A Practical Guide to Neural Architecture
Design [3.04585143845864]
We develop methods that can predict, without any training, whether an architecture will achieve a relatively high test or training error on a task after training.
We then go on to explain the error in terms of the architecture definition itself and develop tools for modifying the architecture.
Our first major contribution is to show that the 'degree of nonlinearity' of a neural architecture is a key causal driver behind its performance.
arXiv Detail & Related papers (2021-05-25T20:47:43Z) - Contrastive Neural Architecture Search with Neural Architecture
Comparators [46.45102111497492]
One of the key steps in Neural Architecture Search (NAS) is to estimate the performance of candidate architectures.
Existing methods either directly use the validation performance or learn a predictor to estimate the performance.
We propose a novel Contrastive Neural Architecture Search (CTNAS) method which performs architecture search by taking the comparison results between architectures as the reward.
arXiv Detail & Related papers (2021-03-08T11:24:07Z) - Weak NAS Predictors Are All You Need [91.11570424233709]
Recent predictor-based NAS approaches attempt to solve the problem with two key steps: sampling some architecture-performance pairs and fitting a proxy accuracy predictor.
We shift the paradigm from finding a complicated predictor that covers the whole architecture space to a set of weaker predictors that progressively move towards the high-performance sub-space.
Our method costs fewer samples to find the top-performance architectures on NAS-Bench-101 and NAS-Bench-201, and it achieves the state-of-the-art ImageNet performance on the NASNet search space.
arXiv Detail & Related papers (2021-02-21T01:58:43Z) - Stage-Wise Neural Architecture Search [65.03109178056937]
Modern convolutional networks such as ResNet and NASNet have achieved state-of-the-art results in many computer vision applications.
These networks consist of stages, which are sets of layers that operate on representations in the same resolution.
It has been demonstrated that increasing the number of layers in each stage improves the prediction ability of the network.
However, the resulting architecture becomes computationally expensive in terms of floating point operations, memory requirements and inference time.
arXiv Detail & Related papers (2020-04-23T14:16:39Z) - PONAS: Progressive One-shot Neural Architecture Search for Very
Efficient Deployment [9.442139459221783]
We propose Progressive One-shot Neural Architecture Search (PONAS) that combines advantages of progressive NAS and one-shot methods.
PONAS is able to find architecture of a specialized network in around 10 seconds.
In ImageNet classification, 75.2% top-1 accuracy can be obtained, which is comparable with the state of the arts.
arXiv Detail & Related papers (2020-03-11T05:00:31Z) - Semi-Supervised Neural Architecture Search [185.0651567642238]
SemiNAS is a semi-supervised Neural architecture search (NAS) approach that leverages numerous unlabeled architectures (without evaluation and thus nearly no cost)
It achieves 94.02% test accuracy on NASBench-101, outperforming all the baselines when using the same number of architectures.
It achieves 97% intelligibility rate in the low-resource setting and 15% test error rate in the robustness setting, with 9%, 7% improvements over the baseline respectively.
arXiv Detail & Related papers (2020-02-24T17:23:00Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.