DiffusionNAG: Predictor-guided Neural Architecture Generation with Diffusion Models

• URL: http://arxiv.org/abs/2305.16943v4
• Date: Sun, 24 Mar 2024 22:00:04 GMT
• Title: DiffusionNAG: Predictor-guided Neural Architecture Generation with Diffusion Models
• Authors: Sohyun An, Hayeon Lee, Jaehyeong Jo, Seanie Lee, Sung Ju Hwang,
• Abstract summary: We propose a novel conditional Neural Architecture Generation (NAG) framework based on diffusion models, dubbed DiffusionNAG. Specifically, we consider the neural architectures as directed graphs and propose a graph diffusion model for generating them. We validate the effectiveness of DiffusionNAG through extensive experiments in two predictor-based NAS scenarios: Transferable NAS and Bayesian Optimization (BO)-based NAS. When integrated into a BO-based algorithm, DiffusionNAG outperforms existing BO-based NAS approaches, particularly in the large MobileNetV3 search space on the ImageNet 1K dataset.
• Score: 56.584561770857306
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Existing NAS methods suffer from either an excessive amount of time for repetitive sampling and training of many task-irrelevant architectures. To tackle such limitations of existing NAS methods, we propose a paradigm shift from NAS to a novel conditional Neural Architecture Generation (NAG) framework based on diffusion models, dubbed DiffusionNAG. Specifically, we consider the neural architectures as directed graphs and propose a graph diffusion model for generating them. Moreover, with the guidance of parameterized predictors, DiffusionNAG can flexibly generate task-optimal architectures with the desired properties for diverse tasks, by sampling from a region that is more likely to satisfy the properties. This conditional NAG scheme is significantly more efficient than previous NAS schemes which sample the architectures and filter them using the property predictors. We validate the effectiveness of DiffusionNAG through extensive experiments in two predictor-based NAS scenarios: Transferable NAS and Bayesian Optimization (BO)-based NAS. DiffusionNAG achieves superior performance with speedups of up to 35 times when compared to the baselines on Transferable NAS benchmarks. Furthermore, when integrated into a BO-based algorithm, DiffusionNAG outperforms existing BO-based NAS approaches, particularly in the large MobileNetV3 search space on the ImageNet 1K dataset. Code is available at https://github.com/CownowAn/DiffusionNAG.

Related papers

• POMONAG: Pareto-Optimal Many-Objective Neural Architecture Generator [4.09225917049674]
  Transferable NAS has emerged, generalizing the search process from dataset-dependent to task-dependent. This paper introduces POMONAG, extending DiffusionNAG via a many-optimal diffusion process. Results were validated on two search spaces -- NAS201 and MobileNetV3 -- and evaluated across 15 image classification datasets.
  arXiv  Detail & Related papers  (2024-09-30T16:05:29Z)
• Efficacy of Neural Prediction-Based Zero-Shot NAS [0.04096453902709291]
  We propose a novel approach for zero-shot Neural Architecture Search (NAS) using deep learning. Our method employs Fourier sum of sines encoding for convolutional kernels, enabling the construction of a computational feed-forward graph with a structure similar to the architecture under evaluation. Experimental results show that our approach surpasses previous methods using graph convolutional networks in terms of correlation on the NAS-Bench-201 dataset and exhibits a higher convergence rate.
  arXiv  Detail & Related papers  (2023-08-31T14:54:06Z)
• Speeding up NAS with Adaptive Subset Selection [21.31075249079979]
  We present an adaptive subset selection approach to neural architecture search (NAS) We devise an algorithm that makes use of state-of-the-art techniques from both areas. Our results are consistent across multiple datasets.
  arXiv  Detail & Related papers  (2022-11-02T19:48:42Z)
• PredNAS: A Universal and Sample Efficient Neural Architecture Search Framework [20.59478264338981]
  We present a general and effective framework for Neural Architecture Search (NAS) named PredNAS. We adopt a neural predictor as the performance predictor. Surprisingly, PredNAS can achieve state-of-the-art performances on NAS benchmarks with only a few training samples.
  arXiv  Detail & Related papers  (2022-10-26T04:15:58Z)
• Generalization Properties of NAS under Activation and Skip Connection Search [66.8386847112332]
  We study the generalization properties of Neural Architecture Search (NAS) under a unifying framework. We derive the lower (and upper) bounds of the minimum eigenvalue of the Neural Tangent Kernel (NTK) under the (in)finite-width regime. We show how the derived results can guide NAS to select the top-performing architectures, even in the case without training.
  arXiv  Detail & Related papers  (2022-09-15T12:11:41Z)
• UnrealNAS: Can We Search Neural Architectures with Unreal Data? [84.78460976605425]
  Neural architecture search (NAS) has shown great success in the automatic design of deep neural networks (DNNs) Previous work has analyzed the necessity of having ground-truth labels in NAS and inspired broad interest. We take a further step to question whether real data is necessary for NAS to be effective.
  arXiv  Detail & Related papers  (2022-05-04T16:30:26Z)
• U-Boost NAS: Utilization-Boosted Differentiable Neural Architecture Search [50.33956216274694]
  optimizing resource utilization in target platforms is key to achieving high performance during DNN inference. We propose a novel hardware-aware NAS framework that does not only optimize for task accuracy and inference latency, but also for resource utilization. We achieve 2.8 - 4x speedup for DNN inference compared to prior hardware-aware NAS methods.
  arXiv  Detail & Related papers  (2022-03-23T13:44:15Z)
• Weak NAS Predictors Are All You Need [91.11570424233709]
  Recent predictor-based NAS approaches attempt to solve the problem with two key steps: sampling some architecture-performance pairs and fitting a proxy accuracy predictor. We shift the paradigm from finding a complicated predictor that covers the whole architecture space to a set of weaker predictors that progressively move towards the high-performance sub-space. Our method costs fewer samples to find the top-performance architectures on NAS-Bench-101 and NAS-Bench-201, and it achieves the state-of-the-art ImageNet performance on the NASNet search space.
  arXiv  Detail & Related papers  (2021-02-21T01:58:43Z)
• BRP-NAS: Prediction-based NAS using GCNs [21.765796576990137]
  BRP-NAS is an efficient hardware-aware NAS enabled by an accurate performance predictor-based on graph convolutional network (GCN) We show that our proposed method outperforms all prior methods on NAS-Bench-101 and NAS-Bench-201. We also release LatBench -- a latency dataset of NAS-Bench-201 models running on a broad range of devices.
  arXiv  Detail & Related papers  (2020-07-16T21:58:43Z)
• DDPNAS: Efficient Neural Architecture Search via Dynamic Distribution Pruning [135.27931587381596]
  We propose an efficient and unified NAS framework termed DDPNAS via dynamic distribution pruning. In particular, we first sample architectures from a joint categorical distribution. Then the search space is dynamically pruned and its distribution is updated every few epochs. With the proposed efficient network generation method, we directly obtain the optimal neural architectures on given constraints.
  arXiv  Detail & Related papers  (2019-05-28T06:35:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.