Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition

• URL: http://arxiv.org/abs/2102.01063v1
• Date: Mon, 1 Feb 2021 18:53:40 GMT
• Title: Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition
• Authors: Ming Lin, Pichao Wang, Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, Rong Jin
• Abstract summary: A key component in Neural Architecture Search (NAS) is an accuracy predictor which asserts the accuracy of a queried architecture. We propose to replace the accuracy predictor with a novel model-complexity index named Zen-score. Instead of predicting model accuracy, Zen-score directly asserts the model complexity of a network without training its parameters.
• Score: 43.97052733871721
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A key component in Neural Architecture Search (NAS) is an accuracy predictor which asserts the accuracy of a queried architecture. To build a high quality accuracy predictor, conventional NAS algorithms rely on training a mass of architectures or a big supernet. This step often consumes hundreds to thousands of GPU days, dominating the total search cost. To address this issue, we propose to replace the accuracy predictor with a novel model-complexity index named Zen-score. Instead of predicting model accuracy, Zen-score directly asserts the model complexity of a network without training its parameters. This is inspired by recent advances in deep learning theories which show that model complexity of a network positively correlates to its accuracy on the target dataset. The computation of Zen-score only takes a few forward inferences through a randomly initialized network using random Gaussian input. It is applicable to any Vanilla Convolutional Neural Networks (VCN-networks) or compatible variants, covering a majority of networks popular in real-world applications. When combining Zen-score with Evolutionary Algorithm, we obtain a novel Zero-Shot NAS algorithm named Zen-NAS. We conduct extensive experiments on CIFAR10/CIFAR100 and ImageNet. In summary, Zen-NAS is able to design high performance architectures in less than half GPU day (12 GPU hours). The resultant networks, named ZenNets, achieve up to $83.0\%$ top-1 accuracy on ImageNet. Comparing to EfficientNets-B3/B5 of the same or better accuracies, ZenNets are up to $5.6$ times faster on NVIDIA V100, $11$ times faster on NVIDIA T4, $2.6$ times faster on Google Pixel2 and uses $50\%$ less FLOPs. Our source code and pre-trained models are released on https://github.com/idstcv/ZenNAS.

Related papers

• DNA Family: Boosting Weight-Sharing NAS with Block-Wise Supervisions [121.05720140641189]
  We develop a family of models with the distilling neural architecture (DNA) techniques. Our proposed DNA models can rate all architecture candidates, as opposed to previous works that can only access a sub- search space using algorithms. Our models achieve state-of-the-art top-1 accuracy of 78.9% and 83.6% on ImageNet for a mobile convolutional network and a small vision transformer, respectively.
  arXiv  Detail & Related papers  (2024-03-02T22:16:47Z)
• Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunities [58.67514819895494]
  Key idea behind zero-shot NAS approaches is to design proxies that can predict the accuracy of some given networks without training the network parameters. This paper aims to comprehensively review and compare the state-of-the-art (SOTA) zero-shot NAS approaches.
  arXiv  Detail & Related papers  (2023-07-05T03:07:00Z)
• Neural Architecture Search via Two Constant Shared Weights Initialisations [0.0]
  We present a zero-cost metric that highly correlated with the train set accuracy across the NAS-Bench-101, NAS-Bench-201 and NAS-Bench-NLP benchmark datasets. Our method is easy to integrate within existing NAS algorithms and takes a fraction of a second to evaluate a single network.
  arXiv  Detail & Related papers  (2023-02-09T02:25:38Z)
• NASiam: Efficient Representation Learning using Neural Architecture Search for Siamese Networks [76.8112416450677]
  Siamese networks are one of the most trending methods to achieve self-supervised visual representation learning (SSL) NASiam is a novel approach that uses for the first time differentiable NAS to improve the multilayer perceptron projector and predictor (encoder/predictor pair) NASiam reaches competitive performance in both small-scale (i.e., CIFAR-10/CIFAR-100) and large-scale (i.e., ImageNet) image classification datasets while costing only a few GPU hours.
  arXiv  Detail & Related papers  (2023-01-31T19:48:37Z)
• ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradients [17.139381064317778]
  We propose a new zero-shot proxy, ZiCo, that works consistently better than #Params. ZiCo-based NAS can find optimal architectures with 78.1%, 79.4%, and 80.4% test accuracy under inference budgets of 450M, 600M, and 1000M FLOPs, respectively.
  arXiv  Detail & Related papers  (2023-01-26T18:38:56Z)
• Revisiting Efficient Object Detection Backbones from Zero-Shot Neural Architecture Search [34.88658308647129]
  In object detection models, the detection backbone consumes more than half of the overall inference cost. We propose a novel zero-shot NAS method to address this issue. The proposed method, named ZenDet, automatically designs efficient detection backbones without training network parameters.
  arXiv  Detail & Related papers  (2021-11-26T07:18:52Z)
• Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift [128.32670289503025]
  Recently proposed neural architecture search (NAS) methods co-train billions of architectures in a supernet and estimate their potential accuracy. The ranking correlation between the architectures' predicted accuracy and their actual capability is incorrect, which causes the existing NAS methods' dilemma. We attribute this ranking correlation problem to the supernet training consistency shift, including feature shift and parameter shift. We address these two shifts simultaneously using a nontrivial supernet-Pi model, called Pi-NAS.
  arXiv  Detail & Related papers  (2021-08-22T09:08:48Z)
• FOX-NAS: Fast, On-device and Explainable Neural Architecture Search [0.0]
  One-Shot approaches typically require a supernet with weight sharing and predictors that predict the performance of architecture. Our method is quantization-friendly and can be efficiently deployed to the edge. Fox-NAS is the 3rd place winner of the 2020 Low-Power Computer Vision Challenge (LPCVC), DSP classification track.
  arXiv  Detail & Related papers  (2021-08-14T16:23:13Z)
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective [88.39981851247727]
  We propose a novel framework called training-free neural architecture search (TE-NAS) TE-NAS ranks architectures by analyzing the spectrum of the neural tangent kernel (NTK) and the number of linear regions in the input space. We show that: (1) these two measurements imply the trainability and expressivity of a neural network; (2) they strongly correlate with the network's test accuracy.
  arXiv  Detail & Related papers  (2021-02-23T07:50:44Z)
• Fast Neural Network Adaptation via Parameter Remapping and Architecture Search [35.61441231491448]
  Deep neural networks achieve remarkable performance in many computer vision tasks. Most state-of-the-art (SOTA) semantic segmentation and object detection approaches reuse neural network architectures designed for image classification as the backbone. One major challenge though, is that ImageNet pre-training of the search space representation incurs huge computational cost. In this paper, we propose a Fast Neural Network Adaptation (FNA) method, which can adapt both the architecture and parameters of a seed network.
  arXiv  Detail & Related papers  (2020-01-08T13:45:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.