Global Attention Mechanism: Retain Information to Enhance Channel-Spatial Interactions

• URL: http://arxiv.org/abs/2112.05561v1
• Date: Fri, 10 Dec 2021 14:12:32 GMT
• Title: Global Attention Mechanism: Retain Information to Enhance Channel-Spatial Interactions
• Authors: Yichao Liu, Zongru Shao, Nico Hoffmann
• Abstract summary: We propose a global attention mechanism that boosts the performance of deep neural networks by reducing information reduction and magnifying the global interactive representations. The evaluation of the proposed mechanism for the image classification task on CIFAR-100 and ImageNet-1K indicates that our method stably outperforms several recent attention mechanisms with both ResNet and lightweight MobileNet.
• Score: 1.4438155481047366
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A variety of attention mechanisms have been studied to improve the performance of various computer vision tasks. However, the prior methods overlooked the significance of retaining the information on both channel and spatial aspects to enhance the cross-dimension interactions. Therefore, we propose a global attention mechanism that boosts the performance of deep neural networks by reducing information reduction and magnifying the global interactive representations. We introduce 3D-permutation with multilayer-perceptron for channel attention alongside a convolutional spatial attention submodule. The evaluation of the proposed mechanism for the image classification task on CIFAR-100 and ImageNet-1K indicates that our method stably outperforms several recent attention mechanisms with both ResNet and lightweight MobileNet.

Related papers

• CAT: Learning to Collaborate Channel and Spatial Attention from Multi-Information Fusion [23.72040577828098]
  We propose a plug-and-play attention module, which we term "CAT"-activating the Collaboration between spatial and channel Attentions. Specifically, we represent traits as trainable coefficients (i.e., colla-factors) to adaptively combine contributions of different attention modules. Our CAT outperforms existing state-of-the-art attention mechanisms in object detection, instance segmentation, and image classification.
  arXiv  Detail & Related papers  (2022-12-13T02:34:10Z)
• A Generic Shared Attention Mechanism for Various Backbone Neural Networks [53.36677373145012]
  Self-attention modules (SAMs) produce strongly correlated attention maps across different layers. Dense-and-Implicit Attention (DIA) shares SAMs across layers and employs a long short-term memory module. Our simple yet effective DIA can consistently enhance various network backbones.
  arXiv  Detail & Related papers  (2022-10-27T13:24:08Z)
• Improving Speech Emotion Recognition Through Focus and Calibration Attention Mechanisms [0.5994412766684842]
  We identify misalignments between the attention and the signal amplitude in the existing multi-head self-attention. We propose to use a Focus-Attention (FA) mechanism and a novel-Attention (CA) mechanism in combination with the multi-head self-attention. By employing the CA mechanism, the network can modulate the information flow by assigning different weights to each attention head and improve the utilization of surrounding contexts.
  arXiv  Detail & Related papers  (2022-08-21T08:04:22Z)
• A Discriminative Channel Diversification Network for Image Classification [21.049734250642974]
  We propose a light-weight and effective attention module, called channel diversification block, to enhance the global context. Unlike other channel attention mechanisms, the proposed module focuses on the most discriminative features. Experiments on CIFAR-10, SVHN, and Tiny-ImageNet datasets demonstrate that the proposed module improves the performance of the baseline networks by a margin of 3% on average.
  arXiv  Detail & Related papers  (2021-12-10T23:00:53Z)
• Attention Mechanisms in Computer Vision: A Survey [75.6074182122423]
  We provide a comprehensive review of various attention mechanisms in computer vision. We categorize them according to approach, such as channel attention, spatial attention, temporal attention and branch attention. We suggest future directions for attention mechanism research.
  arXiv  Detail & Related papers  (2021-11-15T09:18:40Z)
• Alignment Attention by Matching Key and Query Distributions [48.93793773929006]
  This paper introduces alignment attention that explicitly encourages self-attention to match the distributions of the key and query within each head. It is simple to convert any models with self-attention, including pre-trained ones, to the proposed alignment attention. On a variety of language understanding tasks, we show the effectiveness of our method in accuracy, uncertainty estimation, generalization across domains, and robustness to adversarial attacks.
  arXiv  Detail & Related papers  (2021-10-25T00:54:57Z)
• Variational Structured Attention Networks for Deep Visual Representation Learning [49.80498066480928]
  We propose a unified deep framework to jointly learn both spatial attention maps and channel attention in a principled manner. Specifically, we integrate the estimation and the interaction of the attentions within a probabilistic representation learning framework. We implement the inference rules within the neural network, thus allowing for end-to-end learning of the probabilistic and the CNN front-end parameters.
  arXiv  Detail & Related papers  (2021-03-05T07:37:24Z)
• Multi-stage Attention ResU-Net for Semantic Segmentation of Fine-Resolution Remote Sensing Images [9.398340832493457]
  We propose a Linear Attention Mechanism (LAM) to address this issue. LAM is approximately equivalent to dot-product attention with computational efficiency. We design a Multi-stage Attention ResU-Net for semantic segmentation from fine-resolution remote sensing images.
  arXiv  Detail & Related papers  (2020-11-29T07:24:21Z)
• Rotate to Attend: Convolutional Triplet Attention Module [21.228370317693244]
  We present triplet attention, a novel method for computing attention weights using a three-branch structure. Our method is simple as well as efficient and can be easily plugged into classic backbone networks as an add-on module. We demonstrate the effectiveness of our method on various challenging tasks including image classification on ImageNet-1k and object detection on MSCOCO and PASCAL VOC datasets.
  arXiv  Detail & Related papers  (2020-10-06T21:31:00Z)
• Single Image Super-Resolution via a Holistic Attention Network [87.42409213909269]
  We propose a new holistic attention network (HAN) to model the holistic interdependencies among layers, channels, and positions. The proposed HAN adaptively emphasizes hierarchical features by considering correlations among layers. Experiments demonstrate that the proposed HAN performs favorably against the state-of-the-art single image super-resolution approaches.
  arXiv  Detail & Related papers  (2020-08-20T04:13:15Z)
• Deep Reinforced Attention Learning for Quality-Aware Visual Recognition [73.15276998621582]
  We build upon the weakly-supervised generation mechanism of intermediate attention maps in any convolutional neural networks. We introduce a meta critic network to evaluate the quality of attention maps in the main network.
  arXiv  Detail & Related papers  (2020-07-13T02:44:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.