NACHOS: Neural Architecture Search for Hardware Constrained Early Exit Neural Networks

• URL: http://arxiv.org/abs/2401.13330v1
• Date: Wed, 24 Jan 2024 09:48:12 GMT
• Title: NACHOS: Neural Architecture Search for Hardware Constrained Early Exit Neural Networks
• Authors: Matteo Gambella, Jary Pomponi, Simone Scardapane, and Manuel Roveri
• Abstract summary: Early Exit Neural Networks (EENNs) endow astandard Deep Neural Network (DNN) with Early Exits (EECs) This work presents Neural Architecture Search for Hardware Constrained Early Exit Neural Networks (NACHOS) NACHOS is the first NAS framework for the design of optimal EENNs satisfying constraints on the accuracy and the number of Multiply and Accumulate (MAC) operations performed by the EENNs at inference time.
• Score: 6.279164022876874
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Early Exit Neural Networks (EENNs) endow astandard Deep Neural Network (DNN) with Early Exit Classifiers (EECs), to provide predictions at intermediate points of the processing when enough confidence in classification is achieved. This leads to many benefits in terms of effectiveness and efficiency. Currently, the design of EENNs is carried out manually by experts, a complex and time-consuming task that requires accounting for many aspects, including the correct placement, the thresholding, and the computational overhead of the EECs. For this reason, the research is exploring the use of Neural Architecture Search (NAS) to automatize the design of EENNs. Currently, few comprehensive NAS solutions for EENNs have been proposed in the literature, and a fully automated, joint design strategy taking into consideration both the backbone and the EECs remains an open problem. To this end, this work presents Neural Architecture Search for Hardware Constrained Early Exit Neural Networks (NACHOS), the first NAS framework for the design of optimal EENNs satisfying constraints on the accuracy and the number of Multiply and Accumulate (MAC) operations performed by the EENNs at inference time. In particular, this provides the joint design of backbone and EECs to select a set of admissible (i.e., respecting the constraints) Pareto Optimal Solutions in terms of best tradeoff between the accuracy and number of MACs. The results show that the models designed by NACHOS are competitive with the state-of-the-art EENNs. Additionally, this work investigates the effectiveness of two novel regularization terms designed for the optimization of the auxiliary classifiers of the EENN

Related papers

• LitE-SNN: Designing Lightweight and Efficient Spiking Neural Network through Spatial-Temporal Compressive Network Search and Joint Optimization [48.41286573672824]
  Spiking Neural Networks (SNNs) mimic the information-processing mechanisms of the human brain and are highly energy-efficient. We propose a new approach named LitE-SNN that incorporates both spatial and temporal compression into the automated network design process.
  arXiv  Detail & Related papers  (2024-01-26T05:23:11Z)
• Single Channel Speech Enhancement Using U-Net Spiking Neural Networks [2.436681150766912]
  Speech enhancement (SE) is crucial for reliable communication devices or robust speech recognition systems. We propose a novel approach to SE using a spiking neural network (SNN) based on a U-Net architecture. SNNs are suitable for processing data with a temporal dimension, such as speech, and are known for their energy-efficient implementation on neuromorphic hardware.
  arXiv  Detail & Related papers  (2023-07-26T19:10:29Z)
• Automated Search-Space Generation Neural Architecture Search [45.902445271519596]
  ASGNAS produces high-performing sub-networks in the one shot manner. ASGNAS delivers three noticeable contributions to minimize human efforts. The library will be released at https://github.com/tianyic/tianyic/only_train_once.
  arXiv  Detail & Related papers  (2023-05-25T19:41:40Z)
• DeepAxe: A Framework for Exploration of Approximation and Reliability Trade-offs in DNN Accelerators [0.9556128246747769]
  The role of Deep Neural Networks (DNNs) in safety-critical applications is expanding. DNNs experience massive growth in terms of computation power. It raises the necessity of improving the reliability of DNN accelerators.
  arXiv  Detail & Related papers  (2023-03-14T20:42:38Z)
• RoHNAS: A Neural Architecture Search Framework with Conjoint Optimization for Adversarial Robustness and Hardware Efficiency of Convolutional and Capsule Networks [10.946374356026679]
  RoHNAS is a novel framework that jointly optimize for adversarial-robustness and hardware-efficiency of Deep Neural Network (DNN) For reducing the exploration time, RoHNAS analyzes and selects appropriate values of adversarial perturbation for each dataset to employ in the NAS flow.
  arXiv  Detail & Related papers  (2022-10-11T09:14:56Z)
• On the Intrinsic Structures of Spiking Neural Networks [66.57589494713515]
  Recent years have emerged a surge of interest in SNNs owing to their remarkable potential to handle time-dependent and event-driven data. There has been a dearth of comprehensive studies examining the impact of intrinsic structures within spiking computations. This work delves deep into the intrinsic structures of SNNs, by elucidating their influence on the expressivity of SNNs.
  arXiv  Detail & Related papers  (2022-06-21T09:42:30Z)
• Towards a Design Framework for TNN-Based Neuromorphic Sensory Processing Units [2.419276285404291]
  Temporal Neural Networks (TNNs) are spiking neural networks that exhibit brain-like sensory processing with high energy efficiency. This work presents the ongoing research towards developing a custom design framework for designing efficient application-specific TNN-based Neuromorphic Sensory Processing Units (NSPUs)
  arXiv  Detail & Related papers  (2022-05-27T21:51:05Z)
• Comparative Analysis of Interval Reachability for Robust Implicit and Feedforward Neural Networks [64.23331120621118]
  We use interval reachability analysis to obtain robustness guarantees for implicit neural networks (INNs) INNs are a class of implicit learning models that use implicit equations as layers. We show that our approach performs at least as well as, and generally better than, applying state-of-the-art interval bound propagation methods to INNs.
  arXiv  Detail & Related papers  (2022-04-01T03:31:27Z)
• Search to aggregate neighborhood for graph neural network [47.47628113034479]
  We propose a framework, which tries to Search to Aggregate NEighborhood (SANE) to automatically design data-specific GNN architectures. By designing a novel and expressive search space, we propose a differentiable search algorithm, which is more efficient than previous reinforcement learning based methods.
  arXiv  Detail & Related papers  (2021-04-14T03:15:19Z)
• MS-RANAS: Multi-Scale Resource-Aware Neural Architecture Search [94.80212602202518]
  We propose Multi-Scale Resource-Aware Neural Architecture Search (MS-RANAS) We employ a one-shot architecture search approach in order to obtain a reduced search cost. We achieve state-of-the-art results in terms of accuracy-speed trade-off.
  arXiv  Detail & Related papers  (2020-09-29T11:56:01Z)
• NASCaps: A Framework for Neural Architecture Search to Optimize the Accuracy and Hardware Efficiency of Convolutional Capsule Networks [10.946374356026679]
  We propose NASCaps, an automated framework for the hardware-aware NAS of different types of Deep Neural Networks (DNNs) We study the efficacy of deploying a multi-objective Genetic Algorithm (e.g., based on the NSGA-II algorithm) Our framework is the first to model and supports the specialized capsule layers and dynamic routing in the NAS-flow.
  arXiv  Detail & Related papers  (2020-08-19T14:29:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.