NAS-HPO-Bench-II: A Benchmark Dataset on Joint Optimization of Convolutional Neural Network Architecture and Training Hyperparameters

• URL: http://arxiv.org/abs/2110.10165v1
• Date: Tue, 19 Oct 2021 18:00:01 GMT
• Title: NAS-HPO-Bench-II: A Benchmark Dataset on Joint Optimization of Convolutional Neural Network Architecture and Training Hyperparameters
• Authors: Yoichi Hirose, Nozomu Yoshinari, Shinichi Shirakawa
• Abstract summary: This paper introduces the first benchmark dataset for joint optimization of network connections and training hyperparameters, which we call NAS-HPO-Bench-II. We collect the performance data of 4K cell-based convolutional neural network architectures trained on the CIFAR-10 dataset with different learning rate and batch size settings. We build a surrogate model predicting the accuracies after 200 epoch training to provide the performance data of longer training epoch.
• Score: 4.039245878626346
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The benchmark datasets for neural architecture search (NAS) have been developed to alleviate the computationally expensive evaluation process and ensure a fair comparison. Recent NAS benchmarks only focus on architecture optimization, although the training hyperparameters affect the obtained model performances. Building the benchmark dataset for joint optimization of architecture and training hyperparameters is essential to further NAS research. The existing NAS-HPO-Bench is a benchmark for joint optimization, but it does not consider the network connectivity design as done in modern NAS algorithms. This paper introduces the first benchmark dataset for joint optimization of network connections and training hyperparameters, which we call NAS-HPO-Bench-II. We collect the performance data of 4K cell-based convolutional neural network architectures trained on the CIFAR-10 dataset with different learning rate and batch size settings, resulting in the data of 192K configurations. The dataset includes the exact data for 12 epoch training. We further build the surrogate model predicting the accuracies after 200 epoch training to provide the performance data of longer training epoch. By analyzing NAS-HPO-Bench-II, we confirm the dependency between architecture and training hyperparameters and the necessity of joint optimization. Finally, we demonstrate the benchmarking of the baseline optimization algorithms using NAS-HPO-Bench-II.

Related papers

• Accel-NASBench: Sustainable Benchmarking for Accelerator-Aware NAS [3.598880812393792]
  We present a technique that allows searching for training proxies that reduce the cost of benchmark construction by significant margins. We show that the benchmark is accurate and allows searching for state-of-the-art hardware-aware models at zero cost.
  arXiv  Detail & Related papers  (2024-04-09T06:23:41Z)
• DiffusionNAG: Predictor-guided Neural Architecture Generation with Diffusion Models [56.584561770857306]
  We propose a novel conditional Neural Architecture Generation (NAG) framework based on diffusion models, dubbed DiffusionNAG. Specifically, we consider the neural architectures as directed graphs and propose a graph diffusion model for generating them. We validate the effectiveness of DiffusionNAG through extensive experiments in two predictor-based NAS scenarios: Transferable NAS and Bayesian Optimization (BO)-based NAS. When integrated into a BO-based algorithm, DiffusionNAG outperforms existing BO-based NAS approaches, particularly in the large MobileNetV3 search space on the ImageNet 1K dataset.
  arXiv  Detail & Related papers  (2023-05-26T13:58:18Z)
• Lightweight Neural Architecture Search for Temporal Convolutional Networks at the Edge [21.72253397805102]
  This work focuses in particular on Temporal Convolutional Networks (TCNs), a convolutional model for time-series processing. We propose the first NAS tool that explicitly targets the optimization of the most peculiar architectural parameters of TCNs. We test the proposed NAS on four real-world, edge-relevant tasks, involving audio and bio-signals.
  arXiv  Detail & Related papers  (2023-01-24T19:47:40Z)
• U-Boost NAS: Utilization-Boosted Differentiable Neural Architecture Search [50.33956216274694]
  optimizing resource utilization in target platforms is key to achieving high performance during DNN inference. We propose a novel hardware-aware NAS framework that does not only optimize for task accuracy and inference latency, but also for resource utilization. We achieve 2.8 - 4x speedup for DNN inference compared to prior hardware-aware NAS methods.
  arXiv  Detail & Related papers  (2022-03-23T13:44:15Z)
• BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule [95.56873042777316]
  Differentiable Architecture Search (DARTS) has received massive attention in recent years, mainly because it significantly reduces the computational cost. This paper formulates the neural architecture search as a distribution learning problem through relaxing the architecture weights into Gaussian distributions. We demonstrate how the differentiable NAS benefits from Bayesian principles, enhancing exploration and improving stability.
  arXiv  Detail & Related papers  (2021-11-25T18:13:42Z)
• Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimization [29.80410614305851]
  Neural architecture search (NAS) and hyper parameter optimization (HPO) make deep learning accessible to non-experts. We propose a set of methods that extend current approaches to jointly optimize neural architectures and hyper parameters with respect to multiple objectives. These methods will serve as simple baselines for future research on multi-objective joint NAS + HPO.
  arXiv  Detail & Related papers  (2021-05-03T17:04:56Z)
• Weak NAS Predictors Are All You Need [91.11570424233709]
  Recent predictor-based NAS approaches attempt to solve the problem with two key steps: sampling some architecture-performance pairs and fitting a proxy accuracy predictor. We shift the paradigm from finding a complicated predictor that covers the whole architecture space to a set of weaker predictors that progressively move towards the high-performance sub-space. Our method costs fewer samples to find the top-performance architectures on NAS-Bench-101 and NAS-Bench-201, and it achieves the state-of-the-art ImageNet performance on the NASNet search space.
  arXiv  Detail & Related papers  (2021-02-21T01:58:43Z)
• Binarized Neural Architecture Search for Efficient Object Recognition [120.23378346337311]
  Binarized neural architecture search (BNAS) produces extremely compressed models to reduce huge computational cost on embedded devices for edge computing. An accuracy of $96.53%$ vs. $97.22%$ is achieved on the CIFAR-10 dataset, but with a significantly compressed model, and a $40%$ faster search than the state-of-the-art PC-DARTS.
  arXiv  Detail & Related papers  (2020-09-08T15:51:23Z)
• BRP-NAS: Prediction-based NAS using GCNs [21.765796576990137]
  BRP-NAS is an efficient hardware-aware NAS enabled by an accurate performance predictor-based on graph convolutional network (GCN) We show that our proposed method outperforms all prior methods on NAS-Bench-101 and NAS-Bench-201. We also release LatBench -- a latency dataset of NAS-Bench-201 models running on a broad range of devices.
  arXiv  Detail & Related papers  (2020-07-16T21:58:43Z)
• FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining [65.39532971991778]
  We present an accuracy predictor that scores architecture and training recipes jointly, guiding both sample selection and ranking. We run fast evolutionary searches in just CPU minutes to generate architecture-recipe pairs for a variety of resource constraints. FBNetV3 makes up a family of state-of-the-art compact neural networks that outperform both automatically and manually-designed competitors.
  arXiv  Detail & Related papers  (2020-06-03T05:20:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.