Efficacy of Neural Prediction-Based Zero-Shot NAS
- URL: http://arxiv.org/abs/2308.16775v3
- Date: Fri, 22 Sep 2023 07:15:05 GMT
- Title: Efficacy of Neural Prediction-Based Zero-Shot NAS
- Authors: Minh Le, Nhan Nguyen, and Ngoc Hoang Luong
- Abstract summary: We propose a novel approach for zero-shot Neural Architecture Search (NAS) using deep learning.
Our method employs Fourier sum of sines encoding for convolutional kernels, enabling the construction of a computational feed-forward graph with a structure similar to the architecture under evaluation.
Experimental results show that our approach surpasses previous methods using graph convolutional networks in terms of correlation on the NAS-Bench-201 dataset and exhibits a higher convergence rate.
- Score: 0.04096453902709291
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In prediction-based Neural Architecture Search (NAS), performance indicators
derived from graph convolutional networks have shown remarkable success. These
indicators, achieved by representing feed-forward structures as component
graphs through one-hot encoding, face a limitation: their inability to evaluate
architecture performance across varying search spaces. In contrast, handcrafted
performance indicators (zero-shot NAS), which use the same architecture with
random initialization, can generalize across multiple search spaces. Addressing
this limitation, we propose a novel approach for zero-shot NAS using deep
learning. Our method employs Fourier sum of sines encoding for convolutional
kernels, enabling the construction of a computational feed-forward graph with a
structure similar to the architecture under evaluation. These encodings are
learnable and offer a comprehensive view of the architecture's topological
information. An accompanying multi-layer perceptron (MLP) then ranks these
architectures based on their encodings. Experimental results show that our
approach surpasses previous methods using graph convolutional networks in terms
of correlation on the NAS-Bench-201 dataset and exhibits a higher convergence
rate. Moreover, our extracted feature representation trained on each NAS
benchmark is transferable to other NAS benchmarks, showing promising
generalizability across multiple search spaces. The code is available at:
https://github.com/minh1409/DFT-NPZS-NAS
Related papers
- DiffusionNAG: Predictor-guided Neural Architecture Generation with Diffusion Models [56.584561770857306]
We propose a novel conditional Neural Architecture Generation (NAG) framework based on diffusion models, dubbed DiffusionNAG.
Specifically, we consider the neural architectures as directed graphs and propose a graph diffusion model for generating them.
We validate the effectiveness of DiffusionNAG through extensive experiments in two predictor-based NAS scenarios: Transferable NAS and Bayesian Optimization (BO)-based NAS.
When integrated into a BO-based algorithm, DiffusionNAG outperforms existing BO-based NAS approaches, particularly in the large MobileNetV3 search space on the ImageNet 1K dataset.
arXiv Detail & Related papers (2023-05-26T13:58:18Z) - GeNAS: Neural Architecture Search with Better Generalization [14.92869716323226]
Recent neural architecture search (NAS) approaches rely on validation loss or accuracy to find the superior network for the target data.
In this paper, we investigate a new neural architecture search measure for excavating architectures with better generalization.
arXiv Detail & Related papers (2023-05-15T12:44:54Z) - Are Neural Architecture Search Benchmarks Well Designed? A Deeper Look
Into Operation Importance [5.065947993017157]
We conduct an empirical analysis of the widely used NAS-Bench-101, NAS-Bench-201 and TransNAS-Bench-101 benchmarks.
We found that only a subset of the operation pool is required to generate architectures close to the upper-bound of the performance range.
We consistently found convolution layers to have the highest impact on the architecture's performance.
arXiv Detail & Related papers (2023-03-29T18:03:28Z) - A General-Purpose Transferable Predictor for Neural Architecture Search [22.883809911265445]
We propose a general-purpose neural predictor for Neural Architecture Search (NAS) that can transfer across search spaces.
Experimental results on NAS-Bench-101, 201 and 301 demonstrate the efficacy of our scheme.
arXiv Detail & Related papers (2023-02-21T17:28:05Z) - NASiam: Efficient Representation Learning using Neural Architecture
Search for Siamese Networks [76.8112416450677]
Siamese networks are one of the most trending methods to achieve self-supervised visual representation learning (SSL)
NASiam is a novel approach that uses for the first time differentiable NAS to improve the multilayer perceptron projector and predictor (encoder/predictor pair)
NASiam reaches competitive performance in both small-scale (i.e., CIFAR-10/CIFAR-100) and large-scale (i.e., ImageNet) image classification datasets while costing only a few GPU hours.
arXiv Detail & Related papers (2023-01-31T19:48:37Z) - NAS-Bench-Graph: Benchmarking Graph Neural Architecture Search [55.75621026447599]
We propose NAS-Bench-Graph, a tailored benchmark that supports unified, reproducible, and efficient evaluations for GraphNAS.
Specifically, we construct a unified, expressive yet compact search space, covering 26,206 unique graph neural network (GNN) architectures.
Based on our proposed benchmark, the performance of GNN architectures can be directly obtained by a look-up table without any further computation.
arXiv Detail & Related papers (2022-06-18T10:17:15Z) - UnrealNAS: Can We Search Neural Architectures with Unreal Data? [84.78460976605425]
Neural architecture search (NAS) has shown great success in the automatic design of deep neural networks (DNNs)
Previous work has analyzed the necessity of having ground-truth labels in NAS and inspired broad interest.
We take a further step to question whether real data is necessary for NAS to be effective.
arXiv Detail & Related papers (2022-05-04T16:30:26Z) - BaLeNAS: Differentiable Architecture Search via the Bayesian Learning
Rule [95.56873042777316]
Differentiable Architecture Search (DARTS) has received massive attention in recent years, mainly because it significantly reduces the computational cost.
This paper formulates the neural architecture search as a distribution learning problem through relaxing the architecture weights into Gaussian distributions.
We demonstrate how the differentiable NAS benefits from Bayesian principles, enhancing exploration and improving stability.
arXiv Detail & Related papers (2021-11-25T18:13:42Z) - Weak NAS Predictors Are All You Need [91.11570424233709]
Recent predictor-based NAS approaches attempt to solve the problem with two key steps: sampling some architecture-performance pairs and fitting a proxy accuracy predictor.
We shift the paradigm from finding a complicated predictor that covers the whole architecture space to a set of weaker predictors that progressively move towards the high-performance sub-space.
Our method costs fewer samples to find the top-performance architectures on NAS-Bench-101 and NAS-Bench-201, and it achieves the state-of-the-art ImageNet performance on the NASNet search space.
arXiv Detail & Related papers (2021-02-21T01:58:43Z) - NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language
Processing [12.02718579660613]
We step outside the computer vision domain by leveraging the language modeling task, which is the core of natural language processing (NLP)
We have provided search space of recurrent neural networks on the text datasets and trained 14k architectures within it.
We have conducted both intrinsic and extrinsic evaluation of the trained models using datasets for semantic relatedness and language understanding evaluation.
arXiv Detail & Related papers (2020-06-12T12:19:06Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.