AttendNets: Tiny Deep Image Recognition Neural Networks for the Edge via Visual Attention Condensers

• URL: http://arxiv.org/abs/2009.14385v1
• Date: Wed, 30 Sep 2020 01:53:17 GMT
• Title: AttendNets: Tiny Deep Image Recognition Neural Networks for the Edge via Visual Attention Condensers
• Authors: Alexander Wong, Mahmoud Famouri, and Mohammad Javad Shafiee
• Abstract summary: We introduce AttendNets, low-precision, highly compact deep neural networks tailored for on-device image recognition. AttendNets possess deep self-attention architectures based on visual attention condensers. Results show AttendNets have significantly lower architectural and computational complexity when compared to several deep neural networks.
• Score: 81.17461895644003
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While significant advances in deep learning has resulted in state-of-the-art performance across a large number of complex visual perception tasks, the widespread deployment of deep neural networks for TinyML applications involving on-device, low-power image recognition remains a big challenge given the complexity of deep neural networks. In this study, we introduce AttendNets, low-precision, highly compact deep neural networks tailored for on-device image recognition. More specifically, AttendNets possess deep self-attention architectures based on visual attention condensers, which extends on the recently introduced stand-alone attention condensers to improve spatial-channel selective attention. Furthermore, AttendNets have unique machine-designed macroarchitecture and microarchitecture designs achieved via a machine-driven design exploration strategy. Experimental results on ImageNet$_{50}$ benchmark dataset for the task of on-device image recognition showed that AttendNets have significantly lower architectural and computational complexity when compared to several deep neural networks in research literature designed for efficiency while achieving highest accuracies (with the smallest AttendNet achieving $\sim$7.2% higher accuracy, while requiring $\sim$3$\times$ fewer multiply-add operations, $\sim$4.17$\times$ fewer parameters, and $\sim$16.7$\times$ lower weight memory requirements than MobileNet-V1). Based on these promising results, AttendNets illustrate the effectiveness of visual attention condensers as building blocks for enabling various on-device visual perception tasks for TinyML applications.

Related papers

• Enhancing Small Object Encoding in Deep Neural Networks: Introducing Fast&Focused-Net with Volume-wise Dot Product Layer [0.0]
  We introduce Fast&Focused-Net, a novel deep neural network architecture tailored for encoding small objects into fixed-length feature vectors. Fast&Focused-Net employs a series of our newly proposed layer, the Volume-wise Dot Product (VDP) layer, designed to address several inherent limitations of CNNs. For small object classification tasks, our network outperformed state-of-the-art methods on datasets such as CIFAR-10, CIFAR-100, STL-10, SVHN-Cropped, and Fashion-MNIST. In the context of larger image classification, when combined with a transformer encoder (ViT
  arXiv  Detail & Related papers  (2024-01-18T09:31:25Z)
• Fast GraspNeXt: A Fast Self-Attention Neural Network Architecture for Multi-task Learning in Computer Vision Tasks for Robotic Grasping on the Edge [80.88063189896718]
  High architectural and computational complexity can result in poor suitability for deployment on embedded devices. Fast GraspNeXt is a fast self-attention neural network architecture tailored for embedded multi-task learning in computer vision tasks for robotic grasping.
  arXiv  Detail & Related papers  (2023-04-21T18:07:14Z)
• A Robust and Low Complexity Deep Learning Model for Remote Sensing Image Classification [1.9019295680940274]
  We present a robust and low complexity deep learning model for Remote Sensing Image Classification (RSIC) By conducting extensive experiments on the benchmark datasets NWPU-RESISC45, we achieve a robust and low-complexity model.
  arXiv  Detail & Related papers  (2022-11-05T06:14:30Z)
• Rethinking Performance Gains in Image Dehazing Networks [25.371802581339576]
  We make minimal modifications to popular U-Net to obtain a compact dehazing network. Specifically, we swap out the convolutional blocks in U-Net for residual blocks with the gating mechanism. With a significantly reduced overhead, gUNet is superior to state-of-the-art methods on multiple image dehazing datasets.
  arXiv  Detail & Related papers  (2022-09-23T07:14:48Z)
• Faster Attention Is What You Need: A Fast Self-Attention Neural Network Backbone Architecture for the Edge via Double-Condensing Attention Condensers [71.40595908386477]
  We introduce a new faster attention condenser design called double-condensing attention condensers. The resulting backbone (which we name AttendNeXt) achieves significantly higher inference throughput on an embedded ARM processor. These promising results demonstrate that exploring different efficient architecture designs and self-attention mechanisms can lead to interesting new building blocks for TinyML applications.
  arXiv  Detail & Related papers  (2022-08-15T02:47:33Z)
• AttendSeg: A Tiny Attention Condenser Neural Network for Semantic Segmentation on the Edge [71.80459780697956]
  We introduce textbfAttendSeg, a low-precision, highly compact deep neural network tailored for on-device semantic segmentation. AttendSeg possesses a self-attention network architecture comprising of light-weight attention condensers for improved spatial-channel selective attention.
  arXiv  Detail & Related papers  (2021-04-29T19:19:04Z)
• TinySpeech: Attention Condensers for Deep Speech Recognition Neural Networks on Edge Devices [71.68436132514542]
  We introduce the concept of attention condensers for building low-footprint, highly-efficient deep neural networks for on-device speech recognition on the edge. To illustrate its efficacy, we introduce TinySpeech, low-precision deep neural networks tailored for on-device speech recognition.
  arXiv  Detail & Related papers  (2020-08-10T16:34:52Z)
• EmotionNet Nano: An Efficient Deep Convolutional Neural Network Design for Real-time Facial Expression Recognition [75.74756992992147]
  This study proposes EmotionNet Nano, an efficient deep convolutional neural network created through a human-machine collaborative design strategy. Two different variants of EmotionNet Nano are presented, each with a different trade-off between architectural and computational complexity and accuracy. We demonstrate that the proposed EmotionNet Nano networks achieved real-time inference speeds (e.g. $>25$ FPS and $>70$ FPS at 15W and 30W, respectively) and high energy efficiency.
  arXiv  Detail & Related papers  (2020-06-29T00:48:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.