Automated Search-Space Generation Neural Architecture Search

• URL: http://arxiv.org/abs/2305.18030v3
• Date: Thu, 5 Oct 2023 22:41:01 GMT
• Title: Automated Search-Space Generation Neural Architecture Search
• Authors: Tianyi Chen, Luming Liang, Tianyu Ding, Ilya Zharkov
• Abstract summary: ASGNAS produces high-performing sub-networks in the one shot manner. ASGNAS delivers three noticeable contributions to minimize human efforts. The library will be released at https://github.com/tianyic/tianyic/only_train_once.
• Score: 45.902445271519596
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: To search an optimal sub-network within a general deep neural network (DNN), existing neural architecture search (NAS) methods typically rely on handcrafting a search space beforehand. Such requirements make it challenging to extend them onto general scenarios without significant human expertise and manual intervention. To overcome the limitations, we propose Automated Search-Space Generation Neural Architecture Search (ASGNAS), perhaps the first automated system to train general DNNs that cover all candidate connections and operations and produce high-performing sub-networks in the one shot manner. Technologically, ASGNAS delivers three noticeable contributions to minimize human efforts: (i) automated search space generation for general DNNs; (ii) a Hierarchical Half-Space Projected Gradient (H2SPG) that leverages the hierarchy and dependency within generated search space to ensure the network validity during optimization, and reliably produces a solution with both high performance and hierarchical group sparsity; and (iii) automated sub-network construction upon the H2SPG solution. Numerically, we demonstrate the effectiveness of ASGNAS on a variety of general DNNs, including RegNet, StackedUnets, SuperResNet, and DARTS, over benchmark datasets such as CIFAR10, Fashion-MNIST, ImageNet, STL-10 , and SVNH. The sub-networks computed by ASGNAS achieve competitive even superior performance compared to the starting full DNNs and other state-of-the-arts. The library will be released at https://github.com/tianyic/only_train_once.

Related papers

• G-EvoNAS: Evolutionary Neural Architecture Search Based on Network Growth [6.712149832731174]
  This paper proposes a computationally efficient neural architecture evolutionary search framework based on network growth (G-EvoNAS) The G-EvoNAS is tested on three commonly used image classification datasets, CIFAR10, CIFAR100, and ImageNet. Experimental results demonstrate that G-EvoNAS can find a neural network architecture comparable to state-of-the-art designs in 0.2 GPU days.
  arXiv  Detail & Related papers  (2024-03-05T05:44:38Z)
• OTOv3: Automatic Architecture-Agnostic Neural Network Training and Compression from Structured Pruning to Erasing Operators [57.145175475579315]
  This topic spans various techniques, from structured pruning to neural architecture search, encompassing both pruning and erasing operators perspectives. We introduce the third-generation Only-Train-Once (OTOv3), which first automatically trains and compresses a general DNN through pruning and erasing operations. Our empirical results demonstrate the efficacy of OTOv3 across various benchmarks in structured pruning and neural architecture search.
  arXiv  Detail & Related papers  (2023-12-15T00:22:55Z)
• Search For Deep Graph Neural Networks [4.3002928862077825]
  Current GNN-oriented NAS methods focus on the search for different layer aggregate components with shallow and simple architectures. We propose a GNN generation pipeline with a novel two-stage search space, which aims at automatically generating high-performance. Experiments on real-world datasets show that our generated GNN models outperforms existing manually designed and NAS-based ones.
  arXiv  Detail & Related papers  (2021-09-21T09:24:59Z)
• Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics [117.4281417428145]
  This work targets designing a principled and unified training-free framework for Neural Architecture Search (NAS) NAS has been explosively studied to automate the discovery of top-performer neural networks, but suffers from heavy resource consumption and often incurs search bias due to truncated training or approximations. We present a unified framework to understand and accelerate NAS, by disentangling "TEG" characteristics of searched networks.
  arXiv  Detail & Related papers  (2021-08-26T17:52:07Z)
• Search to aggregate neighborhood for graph neural network [47.47628113034479]
  We propose a framework, which tries to Search to Aggregate NEighborhood (SANE) to automatically design data-specific GNN architectures. By designing a novel and expressive search space, we propose a differentiable search algorithm, which is more efficient than previous reinforcement learning based methods.
  arXiv  Detail & Related papers  (2021-04-14T03:15:19Z)
• Trilevel Neural Architecture Search for Efficient Single Image Super-Resolution [127.92235484598811]
  This paper proposes a trilevel neural architecture search (NAS) method for efficient single image super-resolution (SR) For modeling the discrete search space, we apply a new continuous relaxation on the discrete search spaces to build a hierarchical mixture of network-path, cell-operations, and kernel-width. An efficient search algorithm is proposed to perform optimization in a hierarchical supernet manner.
  arXiv  Detail & Related papers  (2021-01-17T12:19:49Z)
• Continuous Ant-Based Neural Topology Search [62.200941836913586]
  This work introduces a novel, nature-inspired neural architecture search (NAS) algorithm based on ant colony optimization. The Continuous Ant-based Neural Topology Search (CANTS) is strongly inspired by how ants move in the real world.
  arXiv  Detail & Related papers  (2020-11-21T17:49:44Z)
• DDPNAS: Efficient Neural Architecture Search via Dynamic Distribution Pruning [135.27931587381596]
  We propose an efficient and unified NAS framework termed DDPNAS via dynamic distribution pruning. In particular, we first sample architectures from a joint categorical distribution. Then the search space is dynamically pruned and its distribution is updated every few epochs. With the proposed efficient network generation method, we directly obtain the optimal neural architectures on given constraints.
  arXiv  Detail & Related papers  (2019-05-28T06:35:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.