Bonsai-Net: One-Shot Neural Architecture Search via Differentiable Pruners

• URL: http://arxiv.org/abs/2006.09264v3
• Date: Fri, 4 Jun 2021 15:40:29 GMT
• Title: Bonsai-Net: One-Shot Neural Architecture Search via Differentiable Pruners
• Authors: Rob Geada, Dennis Prangle, Andrew Stephen McGough
• Abstract summary: One-shot Neural Architecture Search (NAS) aims to minimize the computational expense of discovering state-of-the-art models. We present Bonsai-Net, an efficient one-shot NAS method to explore our relaxed search space.
• Score: 1.4180331276028662
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: One-shot Neural Architecture Search (NAS) aims to minimize the computational expense of discovering state-of-the-art models. However, in the past year attention has been drawn to the comparable performance of naive random search across the same search spaces used by leading NAS algorithms. To address this, we explore the effects of drastically relaxing the NAS search space, and we present Bonsai-Net, an efficient one-shot NAS method to explore our relaxed search space. Bonsai-Net is built around a modified differential pruner and can consistently discover state-of-the-art architectures that are significantly better than random search with fewer parameters than other state-of-the-art methods. Additionally, Bonsai-Net performs simultaneous model search and training, dramatically reducing the total time it takes to generate fully-trained models from scratch.

Related papers

• DNA Family: Boosting Weight-Sharing NAS with Block-Wise Supervisions [121.05720140641189]
  We develop a family of models with the distilling neural architecture (DNA) techniques. Our proposed DNA models can rate all architecture candidates, as opposed to previous works that can only access a sub- search space using algorithms. Our models achieve state-of-the-art top-1 accuracy of 78.9% and 83.6% on ImageNet for a mobile convolutional network and a small vision transformer, respectively.
  arXiv  Detail & Related papers  (2024-03-02T22:16:47Z)
• When NAS Meets Trees: An Efficient Algorithm for Neural Architecture Search [117.89827740405694]
  Key challenge in neural architecture search (NAS) is designing how to explore wisely in the huge search space. We propose a new NAS method called TNAS (NAS with trees), which improves search efficiency by exploring only a small number of architectures. TNAS finds the global optimal architecture on CIFAR-10 with test accuracy of 94.37% in four GPU hours in NAS-Bench-201.
  arXiv  Detail & Related papers  (2022-04-11T07:34:21Z)
• $\beta$-DARTS: Beta-Decay Regularization for Differentiable Architecture Search [85.84110365657455]
  We propose a simple-but-efficient regularization method, termed as Beta-Decay, to regularize the DARTS-based NAS searching process. Experimental results on NAS-Bench-201 show that our proposed method can help to stabilize the searching process and makes the searched network more transferable across different datasets.
  arXiv  Detail & Related papers  (2022-03-03T11:47:14Z)
• Lightweight Monocular Depth with a Novel Neural Architecture Search Method [46.97673710849343]
  This paper presents a novel neural architecture search method, called LiDNAS, for generating lightweight monocular depth estimation models. We construct the search space on a pre-defined backbone network to balance layer diversity and search space size. The LiDNAS optimized models achieve results superior to compact depth estimation state-of-the-art on NYU-Depth-v2, KITTI, and ScanNet.
  arXiv  Detail & Related papers  (2021-08-25T08:06:28Z)
• Searching Efficient Model-guided Deep Network for Image Denoising [61.65776576769698]
  We present a novel approach by connecting model-guided design with NAS (MoD-NAS) MoD-NAS employs a highly reusable width search strategy and a densely connected search block to automatically select the operations of each layer. Experimental results on several popular datasets show that our MoD-NAS has achieved even better PSNR performance than current state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-04-06T14:03:01Z)
• BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search [100.28980854978768]
  We present Block-wisely Self-supervised Neural Architecture Search (BossNAS) We factorize the search space into blocks and utilize a novel self-supervised training scheme, named ensemble bootstrapping, to train each block separately. We also present HyTra search space, a fabric-like hybrid CNN-transformer search space with searchable down-sampling positions.
  arXiv  Detail & Related papers  (2021-03-23T10:05:58Z)
• AdvantageNAS: Efficient Neural Architecture Search with Credit Assignment [23.988393741948485]
  We propose a novel search strategy for one-shot and sparse propagation NAS, namely AdvantageNAS. AdvantageNAS is a gradient-based approach that improves the search efficiency by introducing credit assignment in gradient estimation for architecture updates. Experiments on the NAS-Bench-201 and PTB dataset show that AdvantageNAS discovers an architecture with higher performance under a limited time budget.
  arXiv  Detail & Related papers  (2020-12-11T05:45:03Z)
• Progressive Automatic Design of Search Space for One-Shot Neural Architecture Search [15.017964136568061]
  It has been observed that a model with higher one-shot model accuracy does not necessarily perform better when stand-alone trained. We propose Progressive Automatic Design of search space, named PAD-NAS. In this way, PAD-NAS can automatically design the operations for each layer and achieve a trade-off between search space quality and model diversity.
  arXiv  Detail & Related papers  (2020-05-15T14:21:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.