Hardware Aware Evolutionary Neural Architecture Search using Representation Similarity Metric

• URL: http://arxiv.org/abs/2311.03923v1
• Date: Tue, 7 Nov 2023 11:58:40 GMT
• Title: Hardware Aware Evolutionary Neural Architecture Search using Representation Similarity Metric
• Authors: Nilotpal Sinha, Abd El Rahman Shabayek, Anis Kacem, Peyman Rostami, Carl Shneider, Djamila Aouada
• Abstract summary: Hardware-aware Neural Architecture Search (HW-NAS) is a technique used to automatically design the architecture of a neural network for a specific task and target hardware. evaluating the performance of candidate architectures is a key challenge in HW-NAS, as it requires significant computational resources. We propose an efficient hardware-aware evolution-based NAS approach called HW-EvRSNAS.
• Score: 12.52012450501367
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hardware-aware Neural Architecture Search (HW-NAS) is a technique used to automatically design the architecture of a neural network for a specific task and target hardware. However, evaluating the performance of candidate architectures is a key challenge in HW-NAS, as it requires significant computational resources. To address this challenge, we propose an efficient hardware-aware evolution-based NAS approach called HW-EvRSNAS. Our approach re-frames the neural architecture search problem as finding an architecture with performance similar to that of a reference model for a target hardware, while adhering to a cost constraint for that hardware. This is achieved through a representation similarity metric known as Representation Mutual Information (RMI) employed as a proxy performance evaluator. It measures the mutual information between the hidden layer representations of a reference model and those of sampled architectures using a single training batch. We also use a penalty term that penalizes the search process in proportion to how far an architecture's hardware cost is from the desired hardware cost threshold. This resulted in a significantly reduced search time compared to the literature that reached up to 8000x speedups resulting in lower CO2 emissions. The proposed approach is evaluated on two different search spaces while using lower computational resources. Furthermore, our approach is thoroughly examined on six different edge devices under various hardware cost constraints.

Related papers

• EM-DARTS: Hierarchical Differentiable Architecture Search for Eye Movement Recognition [54.99121380536659]
  Eye movement biometrics have received increasing attention thanks to its high secure identification. Deep learning (DL) models have been recently successfully applied for eye movement recognition. DL architecture still is determined by human prior knowledge. We propose EM-DARTS, a hierarchical differentiable architecture search algorithm to automatically design the DL architecture for eye movement recognition.
  arXiv  Detail & Related papers  (2024-09-22T13:11:08Z)
• Multi-Objective Hardware Aware Neural Architecture Search using Hardware Cost Diversity [12.52012450501367]
  Hardware-aware Neural Architecture Search approaches (HW-NAS) automate the design of deep learning architectures tailored specifically to a given target hardware platform. These techniques demand substantial computational resources, primarily due to the expensive process of assessing the performance of identified architectures. We propose a Multi-Objective method to address the HW-NAS problem, called MO-HDNAS, to identify the trade-off set of architectures in a single run with low computational cost.
  arXiv  Detail & Related papers  (2024-04-15T15:32:58Z)
• DNA Family: Boosting Weight-Sharing NAS with Block-Wise Supervisions [121.05720140641189]
  We develop a family of models with the distilling neural architecture (DNA) techniques. Our proposed DNA models can rate all architecture candidates, as opposed to previous works that can only access a sub- search space using algorithms. Our models achieve state-of-the-art top-1 accuracy of 78.9% and 83.6% on ImageNet for a mobile convolutional network and a small vision transformer, respectively.
  arXiv  Detail & Related papers  (2024-03-02T22:16:47Z)
• Search-time Efficient Device Constraints-Aware Neural Architecture Search [6.527454079441765]
  Deep learning techniques like computer vision and natural language processing can be computationally expensive and memory-intensive. We automate the construction of task-specific deep learning architectures optimized for device constraints through Neural Architecture Search (NAS) We present DCA-NAS, a principled method of fast neural network architecture search that incorporates edge-device constraints.
  arXiv  Detail & Related papers  (2023-07-10T09:52:28Z)
• NAS-FCOS: Efficient Search for Object Detection Architectures [113.47766862146389]
  We propose an efficient method to obtain better object detectors by searching for the feature pyramid network (FPN) and the prediction head of a simple anchor-free object detector. With carefully designed search space, search algorithms, and strategies for evaluating network quality, we are able to find top-performing detection architectures within 4 days using 8 V100 GPUs.
  arXiv  Detail & Related papers  (2021-10-24T12:20:04Z)
• ISyNet: Convolutional Neural Networks design for AI accelerator [0.0]
  Current state-of-the-art architectures are found with neural architecture search (NAS) taking model complexity into account. We propose a measure of hardware efficiency of neural architecture search space - matrix efficiency measure (MEM); a search space comprising of hardware-efficient operations; a latency-aware scaling method. We show the advantage of the designed architectures for the NPU devices on ImageNet and the generalization ability for the downstream classification and detection tasks.
  arXiv  Detail & Related papers  (2021-09-04T20:57:05Z)
• Enhanced Gradient for Differentiable Architecture Search [17.431144144044968]
  We propose a neural network architecture search algorithm aiming to simultaneously improve network performance and reduce network complexity. The proposed framework automatically builds the network architecture at two stages: block-level search and network-level search. Experiment results demonstrate that our method outperforms all evaluated hand-crafted networks in image classification.
  arXiv  Detail & Related papers  (2021-03-23T13:27:24Z)
• Generalized Latency Performance Estimation for Once-For-All Neural Architecture Search [0.0]
  We introduce two generalizability strategies which include fine-tuning using a base model trained on a specific hardware and NAS search space. We provide a family of latency prediction models that achieve over 50% lower RMSE loss as compared to ProxylessNAS.
  arXiv  Detail & Related papers  (2021-01-04T00:48:09Z)
• ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding [86.40042104698792]
  We formulate neural architecture search as a sparse coding problem. In experiments, our two-stage method on CIFAR-10 requires only 0.05 GPU-day for search. Our one-stage method produces state-of-the-art performances on both CIFAR-10 and ImageNet at the cost of only evaluation time.
  arXiv  Detail & Related papers  (2020-10-13T04:34:24Z)
• MS-RANAS: Multi-Scale Resource-Aware Neural Architecture Search [94.80212602202518]
  We propose Multi-Scale Resource-Aware Neural Architecture Search (MS-RANAS) We employ a one-shot architecture search approach in order to obtain a reduced search cost. We achieve state-of-the-art results in terms of accuracy-speed trade-off.
  arXiv  Detail & Related papers  (2020-09-29T11:56:01Z)
• Stage-Wise Neural Architecture Search [65.03109178056937]
  Modern convolutional networks such as ResNet and NASNet have achieved state-of-the-art results in many computer vision applications. These networks consist of stages, which are sets of layers that operate on representations in the same resolution. It has been demonstrated that increasing the number of layers in each stage improves the prediction ability of the network. However, the resulting architecture becomes computationally expensive in terms of floating point operations, memory requirements and inference time.
  arXiv  Detail & Related papers  (2020-04-23T14:16:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.