FreeREA: Training-Free Evolution-based Architecture Search

• URL: http://arxiv.org/abs/2207.05135v2
• Date: Wed, 10 May 2023 10:04:17 GMT
• Title: FreeREA: Training-Free Evolution-based Architecture Search
• Authors: Niccol\`o Cavagnero, Luca Robbiano, Barbara Caputo and Giuseppe Averta
• Abstract summary: FreeREA is a custom cell-based evolution NAS algorithm that exploits an optimised combination of training-free metrics to rank architectures. Our experiments, carried out on the common benchmarks NAS-Bench-101 and NATS-Bench, demonstrate that i) FreeREA is a fast, efficient, and effective search method for models automatic design.
• Score: 17.202375422110553
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the last decade, most research in Machine Learning contributed to the improvement of existing models, with the aim of increasing the performance of neural networks for the solution of a variety of different tasks. However, such advancements often come at the cost of an increase of model memory and computational requirements. This represents a significant limitation for the deployability of research output in realistic settings, where the cost, the energy consumption, and the complexity of the framework play a crucial role. To solve this issue, the designer should search for models that maximise the performance while limiting its footprint. Typical approaches to reach this goal rely either on manual procedures, which cannot guarantee the optimality of the final design, or upon Neural Architecture Search algorithms to automatise the process, at the expenses of extremely high computational time. This paper provides a solution for the fast identification of a neural network that maximises the model accuracy while preserving size and computational constraints typical of tiny devices. Our approach, named FreeREA, is a custom cell-based evolution NAS algorithm that exploits an optimised combination of training-free metrics to rank architectures during the search, thus without need of model training. Our experiments, carried out on the common benchmarks NAS-Bench-101 and NATS-Bench, demonstrate that i) FreeREA is a fast, efficient, and effective search method for models automatic design; ii) it outperforms State of the Art training-based and training-free techniques in all the datasets and benchmarks considered, and iii) it can easily generalise to constrained scenarios, representing a competitive solution for fast Neural Architecture Search in generic constrained applications. The code is available at \url{https://github.com/NiccoloCavagnero/FreeREA}.

Related papers

• Task-Oriented Real-time Visual Inference for IoVT Systems: A Co-design Framework of Neural Networks and Edge Deployment [61.20689382879937]
  Task-oriented edge computing addresses this by shifting data analysis to the edge. Existing methods struggle to balance high model performance with low resource consumption. We propose a novel co-design framework to optimize neural network architecture.
  arXiv  Detail & Related papers  (2024-10-29T19:02:54Z)
• Mechanistic Design and Scaling of Hybrid Architectures [114.3129802943915]
  We identify and test new hybrid architectures constructed from a variety of computational primitives. We experimentally validate the resulting architectures via an extensive compute-optimal and a new state-optimal scaling law analysis. We find MAD synthetics to correlate with compute-optimal perplexity, enabling accurate evaluation of new architectures.
  arXiv  Detail & Related papers  (2024-03-26T16:33:12Z)
• Auto-Train-Once: Controller Network Guided Automatic Network Pruning from Scratch [72.26822499434446]
  Auto-Train-Once (ATO) is an innovative network pruning algorithm designed to automatically reduce the computational and storage costs of DNNs. We provide a comprehensive convergence analysis as well as extensive experiments, and the results show that our approach achieves state-of-the-art performance across various model architectures.
  arXiv  Detail & Related papers  (2024-03-21T02:33:37Z)
• Neural Architecture Codesign for Fast Bragg Peak Analysis [1.7081438846690533]
  We develop an automated pipeline to streamline neural architecture codesign for fast, real-time Bragg peak analysis in microscopy. Our method employs neural architecture search and AutoML to enhance these models, including hardware costs, leading to the discovery of more hardware-efficient neural architectures.
  arXiv  Detail & Related papers  (2023-12-10T19:42:18Z)
• Entropic Score metric: Decoupling Topology and Size in Training-free NAS [18.804303642485895]
  This paper contributes with a novel training-free metric, named Entropic Score, to estimate model expressivity through the aggregated element-wise entropy of its activations. A proper combination with LogSynflow, to search for model size, yields superior capability to completely design high-performance Hybrid Transformers for edge applications in less than 1 GPU hour.
  arXiv  Detail & Related papers  (2023-10-06T11:49:21Z)
• POPNASv3: a Pareto-Optimal Neural Architecture Search Solution for Image and Time Series Classification [8.190723030003804]
  This article presents the third version of a sequential model-based NAS algorithm targeting different hardware environments and multiple classification tasks. Our method is able to find competitive architectures within large search spaces, while keeping a flexible structure and data processing pipeline to adapt to different tasks. The experiments performed on images and time series classification datasets provide evidence that POPNASv3 can explore a large set of assorted operators and converge to optimal architectures suited for the type of data provided under different scenarios.
  arXiv  Detail & Related papers  (2022-12-13T17:14:14Z)
• Towards Theoretically Inspired Neural Initialization Optimization [66.04735385415427]
  We propose a differentiable quantity, named GradCosine, with theoretical insights to evaluate the initial state of a neural network. We show that both the training and test performance of a network can be improved by maximizing GradCosine under norm constraint. Generalized from the sample-wise analysis into the real batch setting, NIO is able to automatically look for a better initialization with negligible cost.
  arXiv  Detail & Related papers  (2022-10-12T06:49:16Z)
• Design Automation for Fast, Lightweight, and Effective Deep Learning Models: A Survey [53.258091735278875]
  This survey covers studies of design automation techniques for deep learning models targeting edge computing. It offers an overview and comparison of key metrics that are used commonly to quantify the proficiency of models in terms of effectiveness, lightness, and computational costs. The survey proceeds to cover three categories of the state-of-the-art of deep model design automation techniques.
  arXiv  Detail & Related papers  (2022-08-22T12:12:43Z)
• Learning Where To Look -- Generative NAS is Surprisingly Efficient [11.83842808044211]
  We propose a generative model, paired with a surrogate predictor, that iteratively learns to generate samples from increasingly promising latent subspaces. This approach leads to very effective and efficient architecture search, while keeping the query amount low.
  arXiv  Detail & Related papers  (2022-03-16T16:27:11Z)
• Efficient Model Performance Estimation via Feature Histories [27.008927077173553]
  An important step in the task of neural network design is the evaluation of a model's performance. In this work, we use the evolution history of features of a network during the early stages of training to build a proxy classifier. We show that our method can be combined with multiple search algorithms to find better solutions to a wide range of tasks.
  arXiv  Detail & Related papers  (2021-03-07T20:41:57Z)
• MS-RANAS: Multi-Scale Resource-Aware Neural Architecture Search [94.80212602202518]
  We propose Multi-Scale Resource-Aware Neural Architecture Search (MS-RANAS) We employ a one-shot architecture search approach in order to obtain a reduced search cost. We achieve state-of-the-art results in terms of accuracy-speed trade-off.
  arXiv  Detail & Related papers  (2020-09-29T11:56:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.