AdvantageNAS: Efficient Neural Architecture Search with Credit Assignment

• URL: http://arxiv.org/abs/2012.06138v2
• Date: Wed, 10 Mar 2021 03:02:38 GMT
• Title: AdvantageNAS: Efficient Neural Architecture Search with Credit Assignment
• Authors: Rei Sato, Jun Sakuma, Youhei Akimoto
• Abstract summary: We propose a novel search strategy for one-shot and sparse propagation NAS, namely AdvantageNAS. AdvantageNAS is a gradient-based approach that improves the search efficiency by introducing credit assignment in gradient estimation for architecture updates. Experiments on the NAS-Bench-201 and PTB dataset show that AdvantageNAS discovers an architecture with higher performance under a limited time budget.
• Score: 23.988393741948485
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural architecture search (NAS) is an approach for automatically designing a neural network architecture without human effort or expert knowledge. However, the high computational cost of NAS limits its use in commercial applications. Two recent NAS paradigms, namely one-shot and sparse propagation, which reduce the time and space complexities, respectively, provide clues for solving this problem. In this paper, we propose a novel search strategy for one-shot and sparse propagation NAS, namely AdvantageNAS, which further reduces the time complexity of NAS by reducing the number of search iterations. AdvantageNAS is a gradient-based approach that improves the search efficiency by introducing credit assignment in gradient estimation for architecture updates. Experiments on the NAS-Bench-201 and PTB dataset show that AdvantageNAS discovers an architecture with higher performance under a limited time budget compared to existing sparse propagation NAS. To further reveal the reliabilities of AdvantageNAS, we investigate it theoretically and find that it monotonically improves the expected loss and thus converges.

Related papers

• Delta-NAS: Difference of Architecture Encoding for Predictor-based Evolutionary Neural Architecture Search [5.1331676121360985]
  We craft an algorithm with the capability to perform fine-grain NAS at a low cost. We propose projecting the problem to a lower dimensional space through predicting the difference in accuracy of a pair of similar networks.
  arXiv  Detail & Related papers  (2024-11-21T02:43:32Z)
• SiGeo: Sub-One-Shot NAS via Information Theory and Geometry of Loss Landscape [14.550053893504764]
  We introduce a "sub-one-shot" paradigm that serves as a bridge between zero-shot and one-shot NAS. In sub-one-shot NAS, the supernet is trained using only a small subset of the training data, a phase we refer to as "warm-up" We present SiGeo, a proxy founded on a novel theoretical framework that connects the supernet warm-up with the efficacy of the proxy.
  arXiv  Detail & Related papers  (2023-11-22T05:25:24Z)
• DiffusionNAG: Predictor-guided Neural Architecture Generation with Diffusion Models [56.584561770857306]
  We propose a novel conditional Neural Architecture Generation (NAG) framework based on diffusion models, dubbed DiffusionNAG. Specifically, we consider the neural architectures as directed graphs and propose a graph diffusion model for generating them. We validate the effectiveness of DiffusionNAG through extensive experiments in two predictor-based NAS scenarios: Transferable NAS and Bayesian Optimization (BO)-based NAS. When integrated into a BO-based algorithm, DiffusionNAG outperforms existing BO-based NAS approaches, particularly in the large MobileNetV3 search space on the ImageNet 1K dataset.
  arXiv  Detail & Related papers  (2023-05-26T13:58:18Z)
• Generalization Properties of NAS under Activation and Skip Connection Search [66.8386847112332]
  We study the generalization properties of Neural Architecture Search (NAS) under a unifying framework. We derive the lower (and upper) bounds of the minimum eigenvalue of the Neural Tangent Kernel (NTK) under the (in)finite-width regime. We show how the derived results can guide NAS to select the top-performing architectures, even in the case without training.
  arXiv  Detail & Related papers  (2022-09-15T12:11:41Z)
• U-Boost NAS: Utilization-Boosted Differentiable Neural Architecture Search [50.33956216274694]
  optimizing resource utilization in target platforms is key to achieving high performance during DNN inference. We propose a novel hardware-aware NAS framework that does not only optimize for task accuracy and inference latency, but also for resource utilization. We achieve 2.8 - 4x speedup for DNN inference compared to prior hardware-aware NAS methods.
  arXiv  Detail & Related papers  (2022-03-23T13:44:15Z)
• TND-NAS: Towards Non-differentiable Objectives in Progressive Differentiable NAS Framework [6.895590095853327]
  Differentiable architecture search has gradually become the mainstream research topic in the field of Neural Architecture Search (NAS) Recent differentiable NAS also aims at further improving the search performance and reducing the GPU-memory consumption. We propose the TND-NAS, which is with the merits of the high efficiency in differentiable NAS framework and the compatibility among non-differentiable metrics in Multi-objective NAS.
  arXiv  Detail & Related papers  (2021-11-06T14:19:36Z)
• Searching Efficient Model-guided Deep Network for Image Denoising [61.65776576769698]
  We present a novel approach by connecting model-guided design with NAS (MoD-NAS) MoD-NAS employs a highly reusable width search strategy and a densely connected search block to automatically select the operations of each layer. Experimental results on several popular datasets show that our MoD-NAS has achieved even better PSNR performance than current state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-04-06T14:03:01Z)
• Efficient Neural Architecture Search for End-to-end Speech Recognition via Straight-Through Gradients [17.501966450686282]
  We develop an efficient Neural Architecture Search (NAS) method via Straight-Through (ST) gradients, called ST-NAS. Experiments over the widely benchmarked 80-hour WSJ and 300-hour Switchboard datasets show that ST-NAS induced architectures significantly outperform the human-designed architecture across the two datasets. Strengths of ST-NAS such as architecture transferability and low computation cost in memory and time are also reported.
  arXiv  Detail & Related papers  (2020-11-11T09:18:58Z)
• Binarized Neural Architecture Search for Efficient Object Recognition [120.23378346337311]
  Binarized neural architecture search (BNAS) produces extremely compressed models to reduce huge computational cost on embedded devices for edge computing. An accuracy of $96.53%$ vs. $97.22%$ is achieved on the CIFAR-10 dataset, but with a significantly compressed model, and a $40%$ faster search than the state-of-the-art PC-DARTS.
  arXiv  Detail & Related papers  (2020-09-08T15:51:23Z)
• NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search [55.12928953187342]
  We propose an extension to NAS-Bench-101: NAS-Bench-201 with a different search space, results on multiple datasets, and more diagnostic information. NAS-Bench-201 has a fixed search space and provides a unified benchmark for almost any up-to-date NAS algorithms. We provide additional diagnostic information such as fine-grained loss and accuracy, which can give inspirations to new designs of NAS algorithms.
  arXiv  Detail & Related papers  (2020-01-02T05:28:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.