BS-NAS: Broadening-and-Shrinking One-Shot NAS with Searchable Numbers of Channels

• URL: http://arxiv.org/abs/2003.09821v1
• Date: Sun, 22 Mar 2020 06:32:47 GMT
• Title: BS-NAS: Broadening-and-Shrinking One-Shot NAS with Searchable Numbers of Channels
• Authors: Zan Shen, Jiang Qian, Bojin Zhuang, Shaojun Wang, Jing Xiao
• Abstract summary: One-Shot methods have evolved into one of the most popular methods in Neural Architecture Search (NAS) One-Shot methods have evolved into one of the most popular methods in Neural Architecture Search (NAS) due to weight sharing and single training of a supernet. Existing methods generally suffer from two issues: predetermined number of channels in each layer which is suboptimal; and model averaging effects and poor ranking correlation caused by weight coupling and continuously expanding search space. A Broadening-and-Shrinking One-Shot NAS (BS-NAS) framework is proposed, in which broadening' refers to broadening the search
• Score: 25.43631259260473
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: One-Shot methods have evolved into one of the most popular methods in Neural Architecture Search (NAS) due to weight sharing and single training of a supernet. However, existing methods generally suffer from two issues: predetermined number of channels in each layer which is suboptimal; and model averaging effects and poor ranking correlation caused by weight coupling and continuously expanding search space. To explicitly address these issues, in this paper, a Broadening-and-Shrinking One-Shot NAS (BS-NAS) framework is proposed, in which `broadening' refers to broadening the search space with a spring block enabling search for numbers of channels during training of the supernet; while `shrinking' refers to a novel shrinking strategy gradually turning off those underperforming operations. The above innovations broaden the search space for wider representation and then shrink it by gradually removing underperforming operations, followed by an evolutionary algorithm to efficiently search for the optimal architecture. Extensive experiments on ImageNet illustrate the effectiveness of the proposed BS-NAS as well as the state-of-the-art performance.

Related papers

• TopoNAS: Boosting Search Efficiency of Gradient-based NAS via Topological Simplification [11.08910129925713]
  TopoNAS is a model-agnostic approach for gradient-based one-shot NAS. It significantly reduces searching time and memory usage by topological simplification of searchable paths.
  arXiv  Detail & Related papers  (2024-08-02T15:01:29Z)
• Lightweight Diffusion Models with Distillation-Based Block Neural Architecture Search [55.41583104734349]
  We propose to automatically remove structural redundancy in diffusion models with our proposed Diffusion Distillation-based Block-wise Neural Architecture Search (NAS) Given a larger pretrained teacher, we leverage DiffNAS to search for the smallest architecture which can achieve on-par or even better performance than the teacher. Different from previous block-wise NAS methods, DiffNAS contains a block-wise local search strategy and a retraining strategy with a joint dynamic loss.
  arXiv  Detail & Related papers  (2023-11-08T12:56:59Z)
• Generalizing Few-Shot NAS with Gradient Matching [165.5690495295074]
  One-Shot methods train one supernet to approximate the performance of every architecture in the search space via weight-sharing. Few-Shot NAS reduces the level of weight-sharing by splitting the One-Shot supernet into multiple separated sub-supernets. It significantly outperforms its Few-Shot counterparts while surpassing previous comparable methods in terms of the accuracy of derived architectures.
  arXiv  Detail & Related papers  (2022-03-29T03:06:16Z)
• $\beta$-DARTS: Beta-Decay Regularization for Differentiable Architecture Search [85.84110365657455]
  We propose a simple-but-efficient regularization method, termed as Beta-Decay, to regularize the DARTS-based NAS searching process. Experimental results on NAS-Bench-201 show that our proposed method can help to stabilize the searching process and makes the searched network more transferable across different datasets.
  arXiv  Detail & Related papers  (2022-03-03T11:47:14Z)
• Searching Efficient Model-guided Deep Network for Image Denoising [61.65776576769698]
  We present a novel approach by connecting model-guided design with NAS (MoD-NAS) MoD-NAS employs a highly reusable width search strategy and a densely connected search block to automatically select the operations of each layer. Experimental results on several popular datasets show that our MoD-NAS has achieved even better PSNR performance than current state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-04-06T14:03:01Z)
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap [90.93522795555724]
  Neural architecture search (NAS) has attracted increasing attentions in both academia and industry. Weight-sharing methods were proposed in which exponentially many architectures share weights in the same super-network. This paper provides a literature review on NAS, in particular the weight-sharing methods.
  arXiv  Detail & Related papers  (2020-08-04T11:57:03Z)
• Bonsai-Net: One-Shot Neural Architecture Search via Differentiable Pruners [1.4180331276028662]
  One-shot Neural Architecture Search (NAS) aims to minimize the computational expense of discovering state-of-the-art models. We present Bonsai-Net, an efficient one-shot NAS method to explore our relaxed search space.
  arXiv  Detail & Related papers  (2020-06-12T14:44:00Z)
• Angle-based Search Space Shrinking for Neural Architecture Search [78.49722661000442]
  Angle-Based search space Shrinking (ABS) for Neural Architecture Search (NAS) Our approach progressively simplifies the original search space by dropping unpromising candidates. ABS can dramatically enhance existing NAS approaches by providing a promising shrunk search space.
  arXiv  Detail & Related papers  (2020-04-28T11:26:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.