Enabling NAS with Automated Super-Network Generation

• URL: http://arxiv.org/abs/2112.10878v1
• Date: Mon, 20 Dec 2021 21:45:48 GMT
• Title: Enabling NAS with Automated Super-Network Generation
• Authors: J. Pablo Mu\~noz, Nikolay Lyalyushkin, Yash Akhauri, Anastasia Senina, Alexander Kozlov, Nilesh Jain
• Abstract summary: Recent Neural Architecture Search (NAS) solutions have produced impressive results training super-networks and then derivingworks. We present BootstrapNAS, a software framework for automatic generation of super-networks for NAS.
• Score: 60.72821429802335
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent Neural Architecture Search (NAS) solutions have produced impressive results training super-networks and then deriving subnetworks, a.k.a. child models that outperform expert-crafted models from a pre-defined search space. Efficient and robust subnetworks can be selected for resource-constrained edge devices, allowing them to perform well in the wild. However, constructing super-networks for arbitrary architectures is still a challenge that often prevents the adoption of these approaches. To address this challenge, we present BootstrapNAS, a software framework for automatic generation of super-networks for NAS. BootstrapNAS takes a pre-trained model from a popular architecture, e.g., ResNet- 50, or from a valid custom design, and automatically creates a super-network out of it, then uses state-of-the-art NAS techniques to train the super-network, resulting in subnetworks that significantly outperform the given pre-trained model. We demonstrate the solution by generating super-networks from arbitrary model repositories and make available the resulting super-networks for reproducibility of the results.

Related papers

• Auto-Train-Once: Controller Network Guided Automatic Network Pruning from Scratch [72.26822499434446]
  Auto-Train-Once (ATO) is an innovative network pruning algorithm designed to automatically reduce the computational and storage costs of DNNs. We provide a comprehensive convergence analysis as well as extensive experiments, and the results show that our approach achieves state-of-the-art performance across various model architectures.
  arXiv  Detail & Related papers  (2024-03-21T02:33:37Z)
• OTOv3: Automatic Architecture-Agnostic Neural Network Training and Compression from Structured Pruning to Erasing Operators [57.145175475579315]
  This topic spans various techniques, from structured pruning to neural architecture search, encompassing both pruning and erasing operators perspectives. We introduce the third-generation Only-Train-Once (OTOv3), which first automatically trains and compresses a general DNN through pruning and erasing operations. Our empirical results demonstrate the efficacy of OTOv3 across various benchmarks in structured pruning and neural architecture search.
  arXiv  Detail & Related papers  (2023-12-15T00:22:55Z)
• Neural Architecture Transfer 2: A Paradigm for Improving Efficiency in Multi-Objective Neural Architecture Search [7.967995669387532]
  We present NATv2, an extension of Neural Architecture Transfer (NAT) that improves multi-objective search algorithms. NATv2 achieves qualitative improvements in the extractable sub-networks by exploiting the improved super-networks.
  arXiv  Detail & Related papers  (2023-07-03T12:25:09Z)
• Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Experts [55.470959564665705]
  Weight-sharing supernets are crucial for performance estimation in cutting-edge neural search frameworks. The proposed method attains state-of-the-art (SoTA) performance in NAS for fast machine translation models. It excels in NAS for building memory-efficient task-agnostic BERT models.
  arXiv  Detail & Related papers  (2023-06-08T00:35:36Z)
• NASRec: Weight Sharing Neural Architecture Search for Recommender Systems [40.54254555949057]
  We propose NASRec, a paradigm that trains a single supernet and efficiently produces abundant models/sub-architectures by weight sharing. Our results on three Click-Through Rates (CTR) prediction benchmarks show that NASRec can outperform both manually designed models and existing NAS methods.
  arXiv  Detail & Related papers  (2022-07-14T20:15:11Z)
• Evolutionary Neural Cascade Search across Supernetworks [68.8204255655161]
  We introduce ENCAS - Evolutionary Neural Cascade Search. ENCAS can be used to search over multiple pretrained supernetworks. We test ENCAS on common computer vision benchmarks.
  arXiv  Detail & Related papers  (2022-03-08T11:06:01Z)
• A Hardware-Aware System for Accelerating Deep Neural Network Optimization [7.189421078452572]
  We propose a comprehensive system that automatically and efficiently finds sub-networks from a pre-trained super-network. By combining novel search tactics and algorithms with intelligent use of predictors, we significantly decrease the time needed to find optimal sub-networks.
  arXiv  Detail & Related papers  (2022-02-25T20:07:29Z)
• Efficient Transfer Learning via Joint Adaptation of Network Architecture and Weight [66.8543732597723]
  Recent worksin neural architecture search (NAS) can aid transfer learning by establishing sufficient network search space. We propose a novel framework consisting of two modules, the neural architecturesearch module for architecture transfer and the neural weight search module for weight transfer. These two modules conduct search on thetarget task based on a reduced super-networks, so we only need to trainonce on the source task.
  arXiv  Detail & Related papers  (2021-05-19T08:58:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.