GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet

• URL: http://arxiv.org/abs/2003.11236v1
• Date: Wed, 25 Mar 2020 06:54:10 GMT
• Title: GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet
• Authors: Shan You, Tao Huang, Mingmin Yang, Fei Wang, Chen Qian, Changshui Zhang
• Abstract summary: GreedyNAS is easy-to-follow, and experimental results on ImageNet dataset indicate that it can achieve better Top-1 accuracy under same search space and FLOPs or latency level. By searching on a larger space, our GreedyNAS can also obtain new state-of-the-art architectures.
• Score: 63.96959854429752
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training a supernet matters for one-shot neural architecture search (NAS) methods since it serves as a basic performance estimator for different architectures (paths). Current methods mainly hold the assumption that a supernet should give a reasonable ranking over all paths. They thus treat all paths equally, and spare much effort to train paths. However, it is harsh for a single supernet to evaluate accurately on such a huge-scale search space (e.g., $7^{21}$). In this paper, instead of covering all paths, we ease the burden of supernet by encouraging it to focus more on evaluation of those potentially-good ones, which are identified using a surrogate portion of validation data. Concretely, during training, we propose a multi-path sampling strategy with rejection, and greedily filter the weak paths. The training efficiency is thus boosted since the training space has been greedily shrunk from all paths to those potentially-good ones. Moreover, we further adopt an exploration and exploitation policy by introducing an empirical candidate path pool. Our proposed method GreedyNAS is easy-to-follow, and experimental results on ImageNet dataset indicate that it can achieve better Top-1 accuracy under same search space and FLOPs or latency level, but with only $\sim$60\% of supernet training cost. By searching on a larger space, our GreedyNAS can also obtain new state-of-the-art architectures.

Related papers

• Robust and Efficient Transfer Learning via Supernet Transfer in Warm-started Neural Architecture Search [4.19475889117731]
  Neural Architecture Search (NAS) frameworks offer a useful and popular solution that helps to democratize AI. NAS frameworks are often computationally expensive to run, which limits their applicability and accessibility. We propose a novel transfer learning approach, capable of effectively transferring pretrained supernets based on Optimal Transport or multi-dataset pretaining.
  arXiv  Detail & Related papers  (2024-07-26T00:17:57Z)
• PA&DA: Jointly Sampling PAth and DAta for Consistent NAS [8.737995937682271]
  One-shot NAS methods train a supernet and then inherit the pre-trained weights to evaluate sub-models. Large gradient variance occurs during supernet training, which degrades the supernet ranking consistency. We propose to explicitly minimize the gradient variance of the supernet training by jointly optimizing the sampling distributions of PAth and DAta.
  arXiv  Detail & Related papers  (2023-02-28T17:14:24Z)
• Prior-Guided One-shot Neural Architecture Search [11.609732776776982]
  We present Prior-Guided One-shot NAS (PGONAS) to strengthen the ranking correlation of supernets. Our PGONAS ranks 3rd place in the supernet Track Track of CVPR2022 Second lightweight NAS challenge.
  arXiv  Detail & Related papers  (2022-06-27T14:19:56Z)
• GreedyNASv2: Greedier Search with a Greedy Path Filter [70.64311838369707]
  In one-shot NAS methods, the search space is usually considerably huge (eg, $1321$) In this paper, we leverage an explicit path filter to capture the characteristics of paths and directly filter those weak ones. For example, our obtained GreedyNASv2-L achieves $81.1%$ Top-1 accuracy on ImageNet dataset, significantly outperforming the ResNet-50 strong baselines.
  arXiv  Detail & Related papers  (2021-11-24T16:32:29Z)
• Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift [128.32670289503025]
  Recently proposed neural architecture search (NAS) methods co-train billions of architectures in a supernet and estimate their potential accuracy. The ranking correlation between the architectures' predicted accuracy and their actual capability is incorrect, which causes the existing NAS methods' dilemma. We attribute this ranking correlation problem to the supernet training consistency shift, including feature shift and parameter shift. We address these two shifts simultaneously using a nontrivial supernet-Pi model, called Pi-NAS.
  arXiv  Detail & Related papers  (2021-08-22T09:08:48Z)
• OPANAS: One-Shot Path Aggregation Network Architecture Search for Object Detection [82.04372532783931]
  Recently, neural architecture search (NAS) has been exploited to design feature pyramid networks (FPNs) We propose a novel One-Shot Path Aggregation Network Architecture Search (OPANAS) algorithm, which significantly improves both searching efficiency and detection accuracy.
  arXiv  Detail & Related papers  (2021-03-08T01:48:53Z)
• Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search [60.965024145243596]
  One-shot weight sharing methods have recently drawn great attention in neural architecture search due to high efficiency and competitive performance. To alleviate this problem, we present a simple yet effective architecture distillation method. We introduce the concept of prioritized path, which refers to the architecture candidates exhibiting superior performance during training. Since the prioritized paths are changed on the fly depending on their performance and complexity, the final obtained paths are the cream of the crop.
  arXiv  Detail & Related papers  (2020-10-29T17:55:05Z)
• DA-NAS: Data Adapted Pruning for Efficient Neural Architecture Search [76.9225014200746]
  Efficient search is a core issue in Neural Architecture Search (NAS) We present DA-NAS that can directly search the architecture for large-scale target tasks while allowing a large candidate set in a more efficient manner. It is 2x faster than previous methods while the accuracy is currently state-of-the-art, at 76.2% under small FLOPs constraint.
  arXiv  Detail & Related papers  (2020-03-27T17:55:21Z)
• NAS evaluation is frustratingly hard [1.7188280334580197]
  Neural Architecture Search (NAS) is an exciting new field which promises to be as much as a game-changer as Convolutional Neural Networks were in 2012. Comparison between different methods is still very much an open issue. Our first contribution is a benchmark of $8$ NAS methods on $5$ datasets.
  arXiv  Detail & Related papers  (2019-12-28T21:24:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.