Related papers: BN-NAS: Neural Architecture Search with Batch Normalization

BN-NAS: Neural Architecture Search with Batch Normalization

URL: http://arxiv.org/abs/2108.07375v1
Date: Mon, 16 Aug 2021 23:23:21 GMT
Title: BN-NAS: Neural Architecture Search with Batch Normalization
Authors: Boyu Chen, Peixia Li, Baopu Li, Chen Lin, Chuming Li, Ming Sun, Junjie Yan, Wanli Ouyang
Abstract summary: We present BN-NAS, neural architecture search with Batch Normalization (BN-NAS), to accelerate neural architecture search (NAS) BN-NAS can significantly reduce the time required by model training and evaluation in NAS.
Score: 116.47802796784386
License: http://creativecommons.org/licenses/by/4.0/
Abstract: We present BN-NAS, neural architecture search with Batch Normalization (BN-NAS), to accelerate neural architecture search (NAS). BN-NAS can significantly reduce the time required by model training and evaluation in NAS. Specifically, for fast evaluation, we propose a BN-based indicator for predicting subnet performance at a very early training stage. The BN-based indicator further facilitates us to improve the training efficiency by only training the BN parameters during the supernet training. This is based on our observation that training the whole supernet is not necessary while training only BN parameters accelerates network convergence for network architecture search. Extensive experiments show that our method can significantly shorten the time of training supernet by more than 10 times and shorten the time of evaluating subnets by more than 600,000 times without losing accuracy.

Related papers

Shrink-Perturb Improves Architecture Mixing during Population Based Training for Neural Architecture Search [62.997667081978825]
We show that simultaneously training and mixing neural networks is a promising way to conduct Neural Architecture Search (NAS) We propose PBT-NAS, an adaptation of PBT to NAS where architectures are improved during training by replacing poorly-performing networks in a population with the result of mixing well-performing ones and inheriting the weights using the shrink-perturb technique.
arXiv Detail & Related papers (2023-07-28T15:29:52Z)
Training BatchNorm Only in Neural Architecture Search and Beyond [17.21663067385715]
There is no effort to understand why training BatchNorm only can find the perform-well architectures with the reduced supernet-training time. We show that train-BN-only supernet provides an advantage on convolutions over other operators, cause unfair competition between architectures. We propose a novel composite performance indicator to evaluate networks from three perspectives.
arXiv Detail & Related papers (2021-12-01T04:09:09Z)
"BNN - BN = ?": Training Binary Neural Networks without Batch Normalization [92.23297927690149]
Batch normalization (BN) is a key facilitator and considered essential for state-of-the-art binary neural networks (BNN) We extend their framework to training BNNs, and for the first time demonstrate that BNs can be completed removed from BNN training and inference regimes.
arXiv Detail & Related papers (2021-04-16T16:46:57Z)
BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search [100.28980854978768]
We present Block-wisely Self-supervised Neural Architecture Search (BossNAS) We factorize the search space into blocks and utilize a novel self-supervised training scheme, named ensemble bootstrapping, to train each block separately. We also present HyTra search space, a fabric-like hybrid CNN-transformer search space with searchable down-sampling positions.
arXiv Detail & Related papers (2021-03-23T10:05:58Z)
Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective [88.39981851247727]
We propose a novel framework called training-free neural architecture search (TE-NAS) TE-NAS ranks architectures by analyzing the spectrum of the neural tangent kernel (NTK) and the number of linear regions in the input space. We show that: (1) these two measurements imply the trainability and expressivity of a neural network; (2) they strongly correlate with the network's test accuracy.
arXiv Detail & Related papers (2021-02-23T07:50:44Z)
Revisiting Batch Normalization for Training Low-latency Deep Spiking Neural Networks from Scratch [5.511606249429581]
Spiking Neural Networks (SNNs) have emerged as an alternative to deep learning. High-accuracy and low-latency SNNs from scratch suffer from non-differentiable nature of a spiking neuron. We propose a temporal Batch Normalization Through Time (BNTT) technique for training temporal SNNs.
arXiv Detail & Related papers (2020-10-05T00:49:30Z)
S3NAS: Fast NPU-aware Neural Architecture Search Methodology [2.607400740040335]
We present a fast NPU-aware NAS methodology, called S3NAS, to find a CNN architecture with higher accuracy than the existing ones. We are able to find a network in 3 hours using TPUv3, which shows 82.72% top-1 accuracy on ImageNet with 11.66 ms latency.
arXiv Detail & Related papers (2020-09-04T04:45:50Z)
Neural Architecture Search without Training [8.067283219068832]
In this work, we examine the overlap of activations between datapoints in untrained networks. We motivate how this can give a measure which is usefully indicative of a network's trained performance. We incorporate this measure into a simple algorithm that allows us to search for powerful networks without any training in a matter of seconds on a single GPU.
arXiv Detail & Related papers (2020-06-08T14:53:56Z)
BNAS:An Efficient Neural Architecture Search Approach Using Broad Scalable Architecture [62.587982139871976]
We propose Broad Neural Architecture Search (BNAS) where we elaborately design broad scalable architecture dubbed Broad Convolutional Neural Network (BCNN) BNAS delivers 0.19 days which is 2.37x less expensive than ENAS who ranks the best in reinforcement learning-based NAS approaches.
arXiv Detail & Related papers (2020-01-18T15:07:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.