FR-NAS: Forward-and-Reverse Graph Predictor for Efficient Neural Architecture Search

• URL: http://arxiv.org/abs/2404.15622v1
• Date: Wed, 24 Apr 2024 03:22:49 GMT
• Title: FR-NAS: Forward-and-Reverse Graph Predictor for Efficient Neural Architecture Search
• Authors: Haoming Zhang, Ran Cheng,
• Abstract summary: We introduce a novel Graph Neural Networks (GNN) predictor for Neural Architecture Search (NAS) This predictor renders neural architectures into vector representations by combining both the conventional and inverse graph views. The experimental results showcase a significant improvement in prediction accuracy, with a 3%--16% increase in Kendall-tau correlation.
• Score: 10.699485270006601
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural Architecture Search (NAS) has emerged as a key tool in identifying optimal configurations of deep neural networks tailored to specific tasks. However, training and assessing numerous architectures introduces considerable computational overhead. One method to mitigating this is through performance predictors, which offer a means to estimate the potential of an architecture without exhaustive training. Given that neural architectures fundamentally resemble Directed Acyclic Graphs (DAGs), Graph Neural Networks (GNNs) become an apparent choice for such predictive tasks. Nevertheless, the scarcity of training data can impact the precision of GNN-based predictors. To address this, we introduce a novel GNN predictor for NAS. This predictor renders neural architectures into vector representations by combining both the conventional and inverse graph views. Additionally, we incorporate a customized training loss within the GNN predictor to ensure efficient utilization of both types of representations. We subsequently assessed our method through experiments on benchmark datasets including NAS-Bench-101, NAS-Bench-201, and the DARTS search space, with a training dataset ranging from 50 to 400 samples. Benchmarked against leading GNN predictors, the experimental results showcase a significant improvement in prediction accuracy, with a 3%--16% increase in Kendall-tau correlation. Source codes are available at https://github.com/EMI-Group/fr-nas.

Related papers

• Do Not Train It: A Linear Neural Architecture Search of Graph Neural Networks [15.823247346294089]
  We develop a novel NAS-GNNs method, namely neural architecture coding (NAC) Our approach leads to state-of-the-art performance, which is up to $200times$ faster and $18.8%$ more accurate than the strong baselines.
  arXiv  Detail & Related papers  (2023-05-23T13:44:04Z)
• A General-Purpose Transferable Predictor for Neural Architecture Search [22.883809911265445]
  We propose a general-purpose neural predictor for Neural Architecture Search (NAS) that can transfer across search spaces. Experimental results on NAS-Bench-101, 201 and 301 demonstrate the efficacy of our scheme.
  arXiv  Detail & Related papers  (2023-02-21T17:28:05Z)
• NAR-Former: Neural Architecture Representation Learning towards Holistic Attributes Prediction [37.357949900603295]
  We propose a neural architecture representation model that can be used to estimate attributes holistically. Experiment results show that our proposed framework can be used to predict the latency and accuracy attributes of both cell architectures and whole deep neural networks.
  arXiv  Detail & Related papers  (2022-11-15T10:15:21Z)
• A Comprehensive Study on Large-Scale Graph Training: Benchmarking and Rethinking [124.21408098724551]
  Large-scale graph training is a notoriously challenging problem for graph neural networks (GNNs) We present a new ensembling training manner, named EnGCN, to address the existing issues. Our proposed method has achieved new state-of-the-art (SOTA) performance on large-scale datasets.
  arXiv  Detail & Related papers  (2022-10-14T03:43:05Z)
• NAS-Bench-Graph: Benchmarking Graph Neural Architecture Search [55.75621026447599]
  We propose NAS-Bench-Graph, a tailored benchmark that supports unified, reproducible, and efficient evaluations for GraphNAS. Specifically, we construct a unified, expressive yet compact search space, covering 26,206 unique graph neural network (GNN) architectures. Based on our proposed benchmark, the performance of GNN architectures can be directly obtained by a look-up table without any further computation.
  arXiv  Detail & Related papers  (2022-06-18T10:17:15Z)
• Self-supervised Representation Learning for Evolutionary Neural Architecture Search [9.038625856798227]
  Recently proposed neural architecture search (NAS) algorithms adopt neural predictors to accelerate the architecture search. How to obtain a neural predictor with high prediction accuracy using a small amount of training data is a central problem to neural predictor-based NAS. We devise two self-supervised learning methods to pre-train the architecture embedding part of neural predictors. We achieve state-of-the-art performance on the NASBench-101 and NASBench201 benchmarks when integrating the pre-trained neural predictors with an evolutionary NAS algorithm.
  arXiv  Detail & Related papers  (2020-10-31T04:57:16Z)
• Accuracy Prediction with Non-neural Model for Neural Architecture Search [185.0651567642238]
  We study an alternative approach which uses non-neural model for accuracy prediction. We leverage gradient boosting decision tree (GBDT) as the predictor for Neural architecture search (NAS) Experiments on NASBench-101 and ImageNet demonstrate the effectiveness of using GBDT as predictor for NAS.
  arXiv  Detail & Related papers  (2020-07-09T13:28:49Z)
• FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining [65.39532971991778]
  We present an accuracy predictor that scores architecture and training recipes jointly, guiding both sample selection and ranking. We run fast evolutionary searches in just CPU minutes to generate architecture-recipe pairs for a variety of resource constraints. FBNetV3 makes up a family of state-of-the-art compact neural networks that outperform both automatically and manually-designed competitors.
  arXiv  Detail & Related papers  (2020-06-03T05:20:21Z)
• A Semi-Supervised Assessor of Neural Architectures [157.76189339451565]
  We employ an auto-encoder to discover meaningful representations of neural architectures. A graph convolutional neural network is introduced to predict the performance of architectures.
  arXiv  Detail & Related papers  (2020-05-14T09:02:33Z)
• Binarized Graph Neural Network [65.20589262811677]
  We develop a binarized graph neural network to learn the binary representations of the nodes with binary network parameters. Our proposed method can be seamlessly integrated into the existing GNN-based embedding approaches. Experiments indicate that the proposed binarized graph neural network, namely BGN, is orders of magnitude more efficient in terms of both time and space.
  arXiv  Detail & Related papers  (2020-04-19T09:43:14Z)
• A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS [18.409809742204896]
  This work proposes a novel Graph-based neural ArchiTecture Scheme, a.k.a. a GATES, to improve the predictor-based neural architecture search. Gates models the operations as the transformation of the propagating information, which mimics the actual data processing of neural architecture.
  arXiv  Detail & Related papers  (2020-04-04T09:54:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.