Colab NAS: Obtaining lightweight task-specific convolutional neural networks following Occam's razor

• URL: http://arxiv.org/abs/2212.07700v2
• Date: Tue, 29 Aug 2023 09:49:34 GMT
• Title: Colab NAS: Obtaining lightweight task-specific convolutional neural networks following Occam's razor
• Abstract summary: ColabNAS is an affordable hardware-aware neural architecture search (HW NAS) technique for producing task-specific CNNs. Its novel derivative-free search strategy, inspired by Occam's razor, allows to obtain state-of-the-art results on the Visual Wake Word dataset.
• Score: 1.6309682969263486
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The current trend of applying transfer learning from convolutional neural networks (CNNs) trained on large datasets can be an overkill when the target application is a custom and delimited problem, with enough data to train a network from scratch. On the other hand, the training of custom and lighter CNNs requires expertise, in the from-scratch case, and or high-end resources, as in the case of hardware-aware neural architecture search (HW NAS), limiting access to the technology by non-habitual NN developers. For this reason, we present ColabNAS, an affordable HW NAS technique for producing lightweight task-specific CNNs. Its novel derivative-free search strategy, inspired by Occam's razor, allows to obtain state-of-the-art results on the Visual Wake Word dataset, a standard TinyML benchmark, in just 3.1 GPU hours using free online GPU services such as Google Colaboratory and Kaggle Kernel.

Related papers

• An affordable hardware-aware neural architecture search for deploying convolutional neural networks on ultra-low-power computing platforms [3.1945067016153437]
  Hardware-aware neural architecture search (HW-NAS) allows the integration of Convolutional Neural Networks (CNNs) in microcontrollers devices.<n>This work presents a HW-NAS generating tiny CNNs that can run on ultra-low-power microcontrollers.
  arXiv  Detail & Related papers  (2026-06-15T06:51:53Z)
• NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internals [58.83169560132308]
  We introduce NNsight and NDIF, technologies that work in tandem to enable scientific study of the representations and computations learned by very large neural networks.
  arXiv  Detail & Related papers  (2024-07-18T17:59:01Z)
• Insights from the Use of Previously Unseen Neural Architecture Search Datasets [6.239015118429603]
  We present eight new datasets created for a series of NAS Challenges: AddNIST, Language, MultNIST, CIFARTile, Gutenberg, Isabella, GeoClassing, and Chesseract. These datasets and challenges are developed to direct attention to issues in NAS development and to encourage authors to consider how their models will perform on datasets unknown to them at development time.
  arXiv  Detail & Related papers  (2024-04-02T16:48:34Z)
• DNA Family: Boosting Weight-Sharing NAS with Block-Wise Supervisions [121.05720140641189]
  We develop a family of models with the distilling neural architecture (DNA) techniques. Our proposed DNA models can rate all architecture candidates, as opposed to previous works that can only access a sub- search space using algorithms. Our models achieve state-of-the-art top-1 accuracy of 78.9% and 83.6% on ImageNet for a mobile convolutional network and a small vision transformer, respectively.
  arXiv  Detail & Related papers  (2024-03-02T22:16:47Z)
• MatchNAS: Optimizing Edge AI in Sparse-Label Data Contexts via Automating Deep Neural Network Porting for Mobile Deployment [54.77943671991863]
  MatchNAS is a novel scheme for porting Deep Neural Networks to mobile devices. We optimise a large network family using both labelled and unlabelled data. We then automatically search for tailored networks for different hardware platforms.
  arXiv  Detail & Related papers  (2024-02-21T04:43:12Z)
• UnrealNAS: Can We Search Neural Architectures with Unreal Data? [84.78460976605425]
  Neural architecture search (NAS) has shown great success in the automatic design of deep neural networks (DNNs) Previous work has analyzed the necessity of having ground-truth labels in NAS and inspired broad interest. We take a further step to question whether real data is necessary for NAS to be effective.
  arXiv  Detail & Related papers  (2022-05-04T16:30:26Z)
• Evolutionary Neural Cascade Search across Supernetworks [68.8204255655161]
  We introduce ENCAS - Evolutionary Neural Cascade Search. ENCAS can be used to search over multiple pretrained supernetworks. We test ENCAS on common computer vision benchmarks.
  arXiv  Detail & Related papers  (2022-03-08T11:06:01Z)
• CondenseNeXt: An Ultra-Efficient Deep Neural Network for Embedded Systems [0.0]
  A Convolutional Neural Network (CNN) is a class of Deep Neural Network (DNN) widely used in the analysis of visual images captured by an image sensor. In this paper, we propose a neoteric variant of deep convolutional neural network architecture to ameliorate the performance of existing CNN architectures for real-time inference on embedded systems.
  arXiv  Detail & Related papers  (2021-12-01T18:20:52Z)
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective [88.39981851247727]
  We propose a novel framework called training-free neural architecture search (TE-NAS) TE-NAS ranks architectures by analyzing the spectrum of the neural tangent kernel (NTK) and the number of linear regions in the input space. We show that: (1) these two measurements imply the trainability and expressivity of a neural network; (2) they strongly correlate with the network's test accuracy.
  arXiv  Detail & Related papers  (2021-02-23T07:50:44Z)
• MS-RANAS: Multi-Scale Resource-Aware Neural Architecture Search [94.80212602202518]
  We propose Multi-Scale Resource-Aware Neural Architecture Search (MS-RANAS) We employ a one-shot architecture search approach in order to obtain a reduced search cost. We achieve state-of-the-art results in terms of accuracy-speed trade-off.
  arXiv  Detail & Related papers  (2020-09-29T11:56:01Z)
• NASCaps: A Framework for Neural Architecture Search to Optimize the Accuracy and Hardware Efficiency of Convolutional Capsule Networks [10.946374356026679]
  We propose NASCaps, an automated framework for the hardware-aware NAS of different types of Deep Neural Networks (DNNs) We study the efficacy of deploying a multi-objective Genetic Algorithm (e.g., based on the NSGA-II algorithm) Our framework is the first to model and supports the specialized capsule layers and dynamic routing in the NAS-flow.
  arXiv  Detail & Related papers  (2020-08-19T14:29:36Z)
• Neural Architecture Search without Training [8.067283219068832]
  In this work, we examine the overlap of activations between datapoints in untrained networks. We motivate how this can give a measure which is usefully indicative of a network's trained performance. We incorporate this measure into a simple algorithm that allows us to search for powerful networks without any training in a matter of seconds on a single GPU.
  arXiv  Detail & Related papers  (2020-06-08T14:53:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.