Improve Ranking Correlation of Super-net through Training Scheme from One-shot NAS to Few-shot NAS

• URL: http://arxiv.org/abs/2206.05896v2
• Date: Tue, 14 Jun 2022 03:07:09 GMT
• Title: Improve Ranking Correlation of Super-net through Training Scheme from One-shot NAS to Few-shot NAS
• Authors: Jiawei Liu, Kaiyu Zhang, Weitai Hu and Qing Yang
• Abstract summary: We propose a step-by-step training super-net scheme from one-shot NAS to few-shot NAS. In the training scheme, we firstly train super-net in a one-shot way, and then we disentangle the weights of super-net. Our method ranks 4th place in the CVPR2022 3rd Lightweight NAS Challenge Track1.
• Score: 13.390484379343908
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The algorithms of one-shot neural architecture search(NAS) have been widely used to reduce computation consumption. However, because of the interference among the subnets in which weights are shared, the subnets inherited from these super-net trained by those algorithms have poor consistency in precision ranking. To address this problem, we propose a step-by-step training super-net scheme from one-shot NAS to few-shot NAS. In the training scheme, we firstly train super-net in a one-shot way, and then we disentangle the weights of super-net by splitting them into multi-subnets and training them gradually. Finally, our method ranks 4th place in the CVPR2022 3rd Lightweight NAS Challenge Track1. Our code is available at https://github.com/liujiawei2333/CVPR2022-NAS-competition-Track-1-4th-solution.

Related papers

• SiGeo: Sub-One-Shot NAS via Information Theory and Geometry of Loss Landscape [14.550053893504764]
  We introduce a "sub-one-shot" paradigm that serves as a bridge between zero-shot and one-shot NAS. In sub-one-shot NAS, the supernet is trained using only a small subset of the training data, a phase we refer to as "warm-up" We present SiGeo, a proxy founded on a novel theoretical framework that connects the supernet warm-up with the efficacy of the proxy.
  arXiv  Detail & Related papers  (2023-11-22T05:25:24Z)
• Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Experts [52.71174872516908]
  We propose mixture-of-supernets, where mixture-of-experts (MoE) is adopted to enhance the expressive power of the supernet model. Compared to existing weight-sharing supernet for NLP, our method can minimize the retraining time, greatly improving training efficiency.
  arXiv  Detail & Related papers  (2023-06-08T00:35:36Z)
• RD-NAS: Enhancing One-shot Supernet Ranking Ability via Ranking Distillation from Zero-cost Proxies [20.076610051602618]
  We propose Ranking Distillation one-shot NAS (RD-NAS) to enhance ranking consistency. Our evaluation of the NAS-Bench-201 and ResNet-based search space demonstrates that RD-NAS achieve 10.7% and 9.65% improvements in ranking ability.
  arXiv  Detail & Related papers  (2023-01-24T07:49:04Z)
• Prior-Guided One-shot Neural Architecture Search [11.609732776776982]
  We present Prior-Guided One-shot NAS (PGONAS) to strengthen the ranking correlation of supernets. Our PGONAS ranks 3rd place in the supernet Track Track of CVPR2022 Second lightweight NAS challenge.
  arXiv  Detail & Related papers  (2022-06-27T14:19:56Z)
• Evolutionary Neural Cascade Search across Supernetworks [68.8204255655161]
  We introduce ENCAS - Evolutionary Neural Cascade Search. ENCAS can be used to search over multiple pretrained supernetworks. We test ENCAS on common computer vision benchmarks.
  arXiv  Detail & Related papers  (2022-03-08T11:06:01Z)
• An Analysis of Super-Net Heuristics in Weight-Sharing NAS [70.57382341642418]
  We show that simple random search achieves competitive performance to complex state-of-the-art NAS algorithms when the super-net is properly trained. We show that simple random search achieves competitive performance to complex state-of-the-art NAS algorithms when the super-net is properly trained.
  arXiv  Detail & Related papers  (2021-10-04T02:18:44Z)
• K-shot NAS: Learnable Weight-Sharing for NAS with K-shot Supernets [52.983810997539486]
  We introduce $K$-shot supernets and take their weights for each operation as a dictionary. A textitsimplex-net is introduced to produce architecture-customized code for each path. Experiments on benchmark datasets validate that K-shot NAS significantly improves the evaluation accuracy of paths.
  arXiv  Detail & Related papers  (2021-06-11T14:57:36Z)
• Neural Architecture Search with Random Labels [16.18010700582234]
  We investigate a new variant of neural architecture search (NAS) paradigm -- searching with random labels (RLNAS) RLNAS achieves comparable or even better results compared with state-of-the-art NAS methods such as PC-DARTS, Single Path One-Shot.
  arXiv  Detail & Related papers  (2021-01-28T06:41:48Z)
• Few-shot Neural Architecture Search [35.28010196935195]
  We propose few-shot NAS that uses multiple supernetworks, called sub-supernets, each covering different regions of the search space to alleviate the undesired co-adaption. With only up to 7 sub-supernets, few-shot NAS establishes new SoTAs: on ImageNet, it finds models that reach 80.5% top-1 accuracy at 600 MB FLOPS and 77.5% top-1 accuracy at 238 MFLOPS.
  arXiv  Detail & Related papers  (2020-06-11T22:36:01Z)
• GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet [63.96959854429752]
  GreedyNAS is easy-to-follow, and experimental results on ImageNet dataset indicate that it can achieve better Top-1 accuracy under same search space and FLOPs or latency level. By searching on a larger space, our GreedyNAS can also obtain new state-of-the-art architectures.
  arXiv  Detail & Related papers  (2020-03-25T06:54:10Z)
• DSNAS: Direct Neural Architecture Search without Parameter Retraining [112.02966105995641]
  We propose a new problem definition for NAS, task-specific end-to-end, based on this observation. We propose DSNAS, an efficient differentiable NAS framework that simultaneously optimize architecture and parameters with a low-biased Monte Carlo estimate. DSNAS successfully discovers networks with comparable accuracy (74.4%) on ImageNet in 420 GPU hours, reducing the total time by more than 34%.
  arXiv  Detail & Related papers  (2020-02-21T04:41:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.