BNAS-v2: Memory-efficient and Performance-collapse-prevented Broad Neural Architecture Search

• URL: http://arxiv.org/abs/2009.08886v4
• Date: Mon, 25 Jan 2021 09:05:02 GMT
• Title: BNAS-v2: Memory-efficient and Performance-collapse-prevented Broad Neural Architecture Search
• Authors: Zixiang Ding, Yaran Chen, Nannan Li and Dongbin Zhao
• Abstract summary: BNAS-v2 embodying both superiorities of BCNN simultaneously. continuous relaxation strategy to make each edge of cell relevant to all candidate operations. Combination of partial channel connections and edge normalization can improve the memory efficiency further.
• Score: 15.287692867984228
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we propose BNAS-v2 to further improve the efficiency of NAS, embodying both superiorities of BCNN simultaneously. To mitigate the unfair training issue of BNAS, we employ continuous relaxation strategy to make each edge of cell in BCNN relevant to all candidate operations for over-parameterized BCNN construction. Moreover, the continuous relaxation strategy relaxes the choice of a candidate operation as a softmax over all predefined operations. Consequently, BNAS-v2 employs the gradient-based optimization algorithm to simultaneously update every possible path of over-parameterized BCNN, rather than the single sampled one as BNAS. However, continuous relaxation leads to another issue named performance collapse, in which those weight-free operations are prone to be selected by the search strategy. For this consequent issue, two solutions are given: 1) we propose Confident Learning Rate (CLR) that considers the confidence of gradient for architecture weights update, increasing with the training time of over-parameterized BCNN; 2) we introduce the combination of partial channel connections and edge normalization that also can improve the memory efficiency further. Moreover, we denote differentiable BNAS (i.e. BNAS with continuous relaxation) as BNAS-D, BNAS-D with CLR as BNAS-v2-CLR, and partial-connected BNAS-D as BNAS-v2-PC. Experimental results on CIFAR-10 and ImageNet show that 1) BNAS-v2 delivers state-of-the-art search efficiency on both CIFAR-10 (0.05 GPU days that is 4x faster than BNAS) and ImageNet (0.19 GPU days); and 2) the proposed CLR is effective to alleviate the performance collapse issue in both BNAS-D and vanilla differentiable NAS framework.

Related papers

• Shrink-Perturb Improves Architecture Mixing during Population Based Training for Neural Architecture Search [62.997667081978825]
  We show that simultaneously training and mixing neural networks is a promising way to conduct Neural Architecture Search (NAS) We propose PBT-NAS, an adaptation of PBT to NAS where architectures are improved during training by replacing poorly-performing networks in a population with the result of mixing well-performing ones and inheriting the weights using the shrink-perturb technique.
  arXiv  Detail & Related papers  (2023-07-28T15:29:52Z)
• DropNAS: Grouped Operation Dropout for Differentiable Architecture Search [78.06809383150437]
  Recently, DARTS relaxes the search process with a differentiable formulation that leverages weight-sharing and SGD. This causes two problems: firstly, the operations with more parameters may never have the chance to express the desired function. We propose a novel grouped operation dropout algorithm named DropNAS to fix the problems with DARTS.
  arXiv  Detail & Related papers  (2022-01-27T17:28:23Z)
• Stacked BNAS: Rethinking Broad Convolutional Neural Network for Neural Architecture Search [16.6035648938434]
  We propose Stacked BNAS whose search space is a developed broad scalable architecture named Stacked BCNN, with better performance than BNAS. On the one hand, Stacked BCNN treats mini-BCNN as the basic block to preserve comprehensive representation and deliver powerful feature extraction ability. On the other hand, we propose Knowledge Embedding Search (KES) to learn appropriate knowledge embeddings.
  arXiv  Detail & Related papers  (2021-11-15T12:49:27Z)
• BN-NAS: Neural Architecture Search with Batch Normalization [116.47802796784386]
  We present BN-NAS, neural architecture search with Batch Normalization (BN-NAS), to accelerate neural architecture search (NAS) BN-NAS can significantly reduce the time required by model training and evaluation in NAS.
  arXiv  Detail & Related papers  (2021-08-16T23:23:21Z)
• BenchENAS: A Benchmarking Platform for Evolutionary Neural Architecture Search [10.925662100634378]
  evolutionary computation based NAS (ENAS) methods have recently gained much attention. The issues of fair comparisons and efficient evaluations have hindered the development of ENAS. This paper develops a platform named BenchENAS to address these issues.
  arXiv  Detail & Related papers  (2021-08-09T07:59:03Z)
• Binarized Neural Architecture Search for Efficient Object Recognition [120.23378346337311]
  Binarized neural architecture search (BNAS) produces extremely compressed models to reduce huge computational cost on embedded devices for edge computing. An accuracy of $96.53%$ vs. $97.22%$ is achieved on the CIFAR-10 dataset, but with a significantly compressed model, and a $40%$ faster search than the state-of-the-art PC-DARTS.
  arXiv  Detail & Related papers  (2020-09-08T15:51:23Z)
• DSNAS: Direct Neural Architecture Search without Parameter Retraining [112.02966105995641]
  We propose a new problem definition for NAS, task-specific end-to-end, based on this observation. We propose DSNAS, an efficient differentiable NAS framework that simultaneously optimize architecture and parameters with a low-biased Monte Carlo estimate. DSNAS successfully discovers networks with comparable accuracy (74.4%) on ImageNet in 420 GPU hours, reducing the total time by more than 34%.
  arXiv  Detail & Related papers  (2020-02-21T04:41:47Z)
• BNAS:An Efficient Neural Architecture Search Approach Using Broad Scalable Architecture [62.587982139871976]
  We propose Broad Neural Architecture Search (BNAS) where we elaborately design broad scalable architecture dubbed Broad Convolutional Neural Network (BCNN) BNAS delivers 0.19 days which is 2.37x less expensive than ENAS who ranks the best in reinforcement learning-based NAS approaches.
  arXiv  Detail & Related papers  (2020-01-18T15:07:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.