Scalable NAS with Factorizable Architectural Parameters

• URL: http://arxiv.org/abs/1912.13256v2
• Date: Tue, 22 Sep 2020 18:47:42 GMT
• Title: Scalable NAS with Factorizable Architectural Parameters
• Authors: Lanfei Wang and Lingxi Xie and Tianyi Zhang and Jun Guo and Qi Tian
• Abstract summary: Neural Architecture Search (NAS) is an emerging topic in machine learning and computer vision. This paper presents a scalable algorithm by factorizing a large set of candidate operators into smaller subspaces. With a small increase in search costs and no extra costs in re-training, we find interesting architectures that were not explored before.
• Score: 102.51428615447703
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural Architecture Search (NAS) is an emerging topic in machine learning and computer vision. The fundamental ideology of NAS is using an automatic mechanism to replace manual designs for exploring powerful network architectures. One of the key factors of NAS is to scale-up the search space, e.g., increasing the number of operators, so that more possibilities are covered, but existing search algorithms often get lost in a large number of operators. For avoiding huge computing and competition among similar operators in the same pool, this paper presents a scalable algorithm by factorizing a large set of candidate operators into smaller subspaces. As a practical example, this allows us to search for effective activation functions along with the regular operators including convolution, pooling, skip-connect, etc. With a small increase in search costs and no extra costs in re-training, we find interesting architectures that were not explored before, and achieve state-of-the-art performance on CIFAR10 and ImageNet, two standard image classification benchmarks.

Related papers

• DNA Family: Boosting Weight-Sharing NAS with Block-Wise Supervisions [121.05720140641189]
  We develop a family of models with the distilling neural architecture (DNA) techniques. Our proposed DNA models can rate all architecture candidates, as opposed to previous works that can only access a sub- search space using algorithms. Our models achieve state-of-the-art top-1 accuracy of 78.9% and 83.6% on ImageNet for a mobile convolutional network and a small vision transformer, respectively.
  arXiv  Detail & Related papers  (2024-03-02T22:16:47Z)
• Approximate Neural Architecture Search via Operation Distribution Learning [4.358626952482686]
  We show that given an architectural cell, its performance largely depends on the ratio of used operations. This intuition is to any specific search strategy and can be applied to a diverse set of NAS algorithms.
  arXiv  Detail & Related papers  (2021-11-08T17:38:29Z)
• TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Search [98.22779489340869]
  We propose TransNAS-Bench-101, a benchmark dataset containing network performance across seven vision tasks. We explore two fundamentally different types of search space: cell-level search space and macro-level search space. With 7,352 backbones evaluated on seven tasks, 51,464 trained models with detailed training information are provided.
  arXiv  Detail & Related papers  (2021-05-25T12:15:21Z)
• Landmark Regularization: Ranking Guided Super-Net Training in Neural Architecture Search [70.57382341642418]
  Weight sharing has become a de facto standard in neural architecture search because it enables the search to be done on commodity hardware. Recent works have empirically shown a ranking disorder between the performance of stand-alone architectures and that of the corresponding shared-weight networks. We propose a regularization term that aims to maximize the correlation between the performance rankings of the shared-weight network and that of the standalone architectures.
  arXiv  Detail & Related papers  (2021-04-12T09:32:33Z)
• GNAS: A Generalized Neural Network Architecture Search Framework [0.0]
  In practice, the problems encountered in training NAS (Neural Architecture Search) are not simplex, but a series of combinations of difficulties are often faced. This paper makes reference and improvement to the previous researches which only solve the single problem of NAS, and combines them into a practical technology flow.
  arXiv  Detail & Related papers  (2021-03-19T06:51:22Z)
• MS-RANAS: Multi-Scale Resource-Aware Neural Architecture Search [94.80212602202518]
  We propose Multi-Scale Resource-Aware Neural Architecture Search (MS-RANAS) We employ a one-shot architecture search approach in order to obtain a reduced search cost. We achieve state-of-the-art results in terms of accuracy-speed trade-off.
  arXiv  Detail & Related papers  (2020-09-29T11:56:01Z)
• CATCH: Context-based Meta Reinforcement Learning for Transferrable Architecture Search [102.67142711824748]
  CATCH is a novel Context-bAsed meTa reinforcement learning algorithm for transferrable arChitecture searcH. The combination of meta-learning and RL allows CATCH to efficiently adapt to new tasks while being agnostic to search spaces. It is also capable of handling cross-domain architecture search as competitive networks on ImageNet, COCO, and Cityscapes are identified.
  arXiv  Detail & Related papers  (2020-07-18T09:35:53Z)
• Local Search is a Remarkably Strong Baseline for Neural Architecture Search [0.0]
  We consider, for the first time, a simple Local Search (LS) algorithm for Neural Architecture Search (NAS) We release two benchmark datasets, named MacroNAS-C10 and MacroNAS-C100, containing 200K saved network evaluations for two established image classification tasks.
  arXiv  Detail & Related papers  (2020-04-20T00:08:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.