Evolving Zero Cost Proxies For Neural Architecture Scoring
- URL: http://arxiv.org/abs/2209.07413v1
- Date: Thu, 15 Sep 2022 16:10:16 GMT
- Title: Evolving Zero Cost Proxies For Neural Architecture Scoring
- Authors: Yash Akhauri, J. Pablo Munoz, Nilesh Jain, Ravi Iyer
- Abstract summary: We propose a genetic programming framework to automate the discovery of zero-cost proxies for neural architecture scoring.
Our methodology efficiently discovers an interpretable and generalizable zero-cost proxy that gives state of the art score-accuracy correlation.
- Score: 3.441021278275805
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Neural Architecture Search (NAS) has significantly improved productivity in
the design and deployment of neural networks (NN). As NAS typically evaluates
multiple models by training them partially or completely, the improved
productivity comes at the cost of significant carbon footprint. To alleviate
this expensive training routine, zero-shot/cost proxies analyze an NN at
initialization to generate a score, which correlates highly with its true
accuracy. Zero-cost proxies are currently designed by experts conducting
multiple cycles of empirical testing on possible algorithms, data-sets, and
neural architecture design spaces. This lowers productivity and is an
unsustainable approach towards zero-cost proxy design as deep learning
use-cases diversify in nature. Additionally, existing zero-cost proxies fail to
generalize across neural architecture design spaces. In this paper, we propose
a genetic programming framework to automate the discovery of zero-cost proxies
for neural architecture scoring. Our methodology efficiently discovers an
interpretable and generalizable zero-cost proxy that gives state of the art
score-accuracy correlation on all data-sets and search spaces of NASBench-201
and Network Design Spaces (NDS). We believe that this research indicates a
promising direction towards automatically discovering zero-cost proxies that
can work across network architecture design spaces, data-sets, and tasks.
Related papers
- ZeroLM: Data-Free Transformer Architecture Search for Language Models [54.83882149157548]
Current automated proxy discovery approaches suffer from extended search times, susceptibility to data overfitting, and structural complexity.
This paper introduces a novel zero-cost proxy methodology that quantifies model capacity through efficient weight statistics.
Our evaluation demonstrates the superiority of this approach, achieving a Spearman's rho of 0.76 and Kendall's tau of 0.53 on the FlexiBERT benchmark.
arXiv Detail & Related papers (2025-03-24T13:11:22Z) - GreenMachine: Automatic Design of Zero-Cost Proxies for Energy-Efficient NAS [0.8192907805418583]
This paper addresses the challenges of model evaluation by automatically designing zero-cost proxies to assess Deep Neural Networks (DNNs) efficiently.
Our method begins with a randomly generated set of zero-cost proxies, which are evolved and tested using the NATS-Bench benchmark.
Results show our method outperforms existing approaches on the stratified sampling strategy.
arXiv Detail & Related papers (2024-11-22T17:24:19Z) - TG-NAS: Leveraging Zero-Cost Proxies with Transformer and Graph Convolution Networks for Efficient Neural Architecture Search [1.30891455653235]
TG-NAS aims to create training-free proxies for architecture performance prediction.
We introduce TG-NAS, a novel model-based universal proxy that leverages a transformer-based operator embedding generator and a graph convolution network (GCN) to predict architecture performance.
TG-NAS achieves up to 300X improvements in search efficiency compared to previous SOTA ZC proxy methods.
arXiv Detail & Related papers (2024-03-30T07:25:30Z) - Auto-Train-Once: Controller Network Guided Automatic Network Pruning from Scratch [72.26822499434446]
Auto-Train-Once (ATO) is an innovative network pruning algorithm designed to automatically reduce the computational and storage costs of DNNs.
We provide a comprehensive convergence analysis as well as extensive experiments, and the results show that our approach achieves state-of-the-art performance across various model architectures.
arXiv Detail & Related papers (2024-03-21T02:33:37Z) - DNA Family: Boosting Weight-Sharing NAS with Block-Wise Supervisions [121.05720140641189]
We develop a family of models with the distilling neural architecture (DNA) techniques.
Our proposed DNA models can rate all architecture candidates, as opposed to previous works that can only access a sub- search space using algorithms.
Our models achieve state-of-the-art top-1 accuracy of 78.9% and 83.6% on ImageNet for a mobile convolutional network and a small vision transformer, respectively.
arXiv Detail & Related papers (2024-03-02T22:16:47Z) - Principled Architecture-aware Scaling of Hyperparameters [69.98414153320894]
Training a high-quality deep neural network requires choosing suitable hyperparameters, which is a non-trivial and expensive process.
In this work, we precisely characterize the dependence of initializations and maximal learning rates on the network architecture.
We demonstrate that network rankings can be easily changed by better training networks in benchmarks.
arXiv Detail & Related papers (2024-02-27T11:52:49Z) - A General-Purpose Transferable Predictor for Neural Architecture Search [22.883809911265445]
We propose a general-purpose neural predictor for Neural Architecture Search (NAS) that can transfer across search spaces.
Experimental results on NAS-Bench-101, 201 and 301 demonstrate the efficacy of our scheme.
arXiv Detail & Related papers (2023-02-21T17:28:05Z) - FreeREA: Training-Free Evolution-based Architecture Search [17.202375422110553]
FreeREA is a custom cell-based evolution NAS algorithm that exploits an optimised combination of training-free metrics to rank architectures.
Our experiments, carried out on the common benchmarks NAS-Bench-101 and NATS-Bench, demonstrate that i) FreeREA is a fast, efficient, and effective search method for models automatic design.
arXiv Detail & Related papers (2022-06-17T11:16:28Z) - Automating Neural Architecture Design without Search [3.651848964235307]
We study the automated architecture design from a new perspective that eliminates the need to sequentially evaluate each neural architecture generated during algorithm execution.
We implement the proposed approach by using a graph neural network for link prediction and acquired the knowledge from NAS-Bench-101.
In addition, we also utilized the learned knowledge from NAS-Bench-101 to automate architecture design in the DARTS search space, and achieved 97.82% accuracy on CIFAR10, and 76.51% top-1 accuracy on ImageNet consuming only $2times10-4$ GPU days.
arXiv Detail & Related papers (2022-04-21T14:41:05Z) - Neural Architecture Search of SPD Manifold Networks [79.45110063435617]
We propose a new neural architecture search (NAS) problem of Symmetric Positive Definite (SPD) manifold networks.
We first introduce a geometrically rich and diverse SPD neural architecture search space for an efficient SPD cell design.
We exploit a differentiable NAS algorithm on our relaxed continuous search space for SPD neural architecture search.
arXiv Detail & Related papers (2020-10-27T18:08:57Z) - MS-RANAS: Multi-Scale Resource-Aware Neural Architecture Search [94.80212602202518]
We propose Multi-Scale Resource-Aware Neural Architecture Search (MS-RANAS)
We employ a one-shot architecture search approach in order to obtain a reduced search cost.
We achieve state-of-the-art results in terms of accuracy-speed trade-off.
arXiv Detail & Related papers (2020-09-29T11:56:01Z) - A Semi-Supervised Assessor of Neural Architectures [157.76189339451565]
We employ an auto-encoder to discover meaningful representations of neural architectures.
A graph convolutional neural network is introduced to predict the performance of architectures.
arXiv Detail & Related papers (2020-05-14T09:02:33Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.