Efficient Attention Network: Accelerate Attention by Searching Where to Plug

• URL: http://arxiv.org/abs/2011.14058v2
• Date: Sun, 11 Jul 2021 12:44:58 GMT
• Title: Efficient Attention Network: Accelerate Attention by Searching Where to Plug
• Authors: Zhongzhan Huang, Senwei Liang, Mingfu Liang, Wei He, Haizhao Yang
• Abstract summary: We propose a framework called Efficient Attention Network (EAN) to improve the efficiency for the existing attention modules. In EAN, we leverage the sharing mechanism to share the attention module within the backbone and search where to connect the shared attention module via reinforcement learning. Experiments on widely-used benchmarks and popular attention networks show the effectiveness of EAN.
• Score: 11.616720452770322
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, many plug-and-play self-attention modules are proposed to enhance the model generalization by exploiting the internal information of deep convolutional neural networks (CNNs). Previous works lay an emphasis on the design of attention module for specific functionality, e.g., light-weighted or task-oriented attention. However, they ignore the importance of where to plug in the attention module since they connect the modules individually with each block of the entire CNN backbone for granted, leading to incremental computational cost and number of parameters with the growth of network depth. Thus, we propose a framework called Efficient Attention Network (EAN) to improve the efficiency for the existing attention modules. In EAN, we leverage the sharing mechanism (Huang et al. 2020) to share the attention module within the backbone and search where to connect the shared attention module via reinforcement learning. Finally, we obtain the attention network with sparse connections between the backbone and modules, while (1) maintaining accuracy (2) reducing extra parameter increment and (3) accelerating inference. Extensive experiments on widely-used benchmarks and popular attention networks show the effectiveness of EAN. Furthermore, we empirically illustrate that our EAN has the capacity of transferring to other tasks and capturing the informative features. The code is available at https://github.com/gbup-group/EAN-efficient-attention-network.

Related papers

• The Lottery Ticket Hypothesis for Self-attention in Convolutional Neural Network [69.54809052377189]
  Recently many plug-and-play self-attention modules (SAMs) are proposed to enhance the model generalization by exploiting the internal information of deep convolutional neural networks (CNNs) We empirically find and verify some counterintuitive phenomena that: (a) Connecting the SAMs to all the blocks may not always bring the largest performance boost, and connecting to partial blocks would be even better; (b) Adding the SAMs to a CNN may not always bring a performance boost, and instead it may even harm the performance of the original CNN backbone.
  arXiv  Detail & Related papers  (2022-07-16T07:08:59Z)
• Learning Target-aware Representation for Visual Tracking via Informative Interactions [49.552877881662475]
  We introduce a novel backbone architecture to improve target-perception ability of feature representation for tracking. The proposed GIM module and InBN mechanism are general and applicable to different backbone types including CNN and Transformer.
  arXiv  Detail & Related papers  (2022-01-07T16:22:27Z)
• TDAN: Top-Down Attention Networks for Enhanced Feature Selectivity in CNNs [18.24779045808196]
  We propose a lightweight top-down (TD) attention module that iteratively generates a "visual searchlight" to perform top-down channel and spatial modulation of its inputs. Our models are more robust to changes in input resolution during inference and learn to "shift attention" by localizing individual objects or features at each computation step without any explicit supervision.
  arXiv  Detail & Related papers  (2021-11-26T12:35:17Z)
• Focal Attention Networks: optimising attention for biomedical image segmentation [2.5243042477020836]
  We investigate the role of the Focal parameter in modulating attention, revealing a link between attention in loss functions and networks. We achieve optimal performance with fewer number of attention modules on three well-validated biomedical imaging datasets.
  arXiv  Detail & Related papers  (2021-10-31T16:20:22Z)
• An Attention Module for Convolutional Neural Networks [5.333582981327498]
  We propose an attention module for convolutional neural networks by developing an AW-convolution. Experiments on several datasets for image classification and object detection tasks show the effectiveness of our proposed attention module.
  arXiv  Detail & Related papers  (2021-08-18T15:36:18Z)
• DMSANet: Dual Multi Scale Attention Network [0.0]
  We propose a new attention module that not only achieves the best performance but also has lesser parameters compared to most existing models. Our attention module can easily be integrated with other convolutional neural networks because of its lightweight nature.
  arXiv  Detail & Related papers  (2021-06-13T10:31:31Z)
• Online Multiple Object Tracking with Cross-Task Synergy [120.70085565030628]
  We propose a novel unified model with synergy between position prediction and embedding association. The two tasks are linked by temporal-aware target attention and distractor attention, as well as identity-aware memory aggregation model.
  arXiv  Detail & Related papers  (2021-04-01T10:19:40Z)
• Coordinate Attention for Efficient Mobile Network Design [96.40415345942186]
  We propose a novel attention mechanism for mobile networks by embedding positional information into channel attention. Unlike channel attention that transforms a feature tensor to a single feature vector via 2D global pooling, the coordinate attention factorizes channel attention into two 1D feature encoding processes. Our coordinate attention is beneficial to ImageNet classification and behaves better in down-stream tasks, such as object detection and semantic segmentation.
  arXiv  Detail & Related papers  (2021-03-04T09:18:02Z)
• Attention-Guided Network for Iris Presentation Attack Detection [13.875545441867137]
  We propose attention-guided iris presentation attack detection (AG-PAD) to augment CNNs with attention mechanisms. Experiments involving both a JHU-APL proprietary dataset and the benchmark LivDet-Iris-2017 dataset suggest that the proposed method achieves promising results.
  arXiv  Detail & Related papers  (2020-10-23T19:23:51Z)
• Deep Reinforced Attention Learning for Quality-Aware Visual Recognition [73.15276998621582]
  We build upon the weakly-supervised generation mechanism of intermediate attention maps in any convolutional neural networks. We introduce a meta critic network to evaluate the quality of attention maps in the main network.
  arXiv  Detail & Related papers  (2020-07-13T02:44:38Z)
• Knowing What, Where and When to Look: Efficient Video Action Modeling with Attention [84.83632045374155]
  Attentive video modeling is essential for action recognition in unconstrained videos. What-Where-When (W3) video attention module models all three facets of video attention jointly. Experiments show that our attention model brings significant improvements to existing action recognition models.
  arXiv  Detail & Related papers  (2020-04-02T21:48:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.