An Attention Module for Convolutional Neural Networks
- URL: http://arxiv.org/abs/2108.08205v1
- Date: Wed, 18 Aug 2021 15:36:18 GMT
- Title: An Attention Module for Convolutional Neural Networks
- Authors: Zhu Baozhou and Peter Hofstee and Jinho Lee and Zaid Al-Ars
- Abstract summary: We propose an attention module for convolutional neural networks by developing an AW-convolution.
Experiments on several datasets for image classification and object detection tasks show the effectiveness of our proposed attention module.
- Score: 5.333582981327498
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Attention mechanism has been regarded as an advanced technique to capture
long-range feature interactions and to boost the representation capability for
convolutional neural networks. However, we found two ignored problems in
current attentional activations-based models: the approximation problem and the
insufficient capacity problem of the attention maps. To solve the two problems
together, we initially propose an attention module for convolutional neural
networks by developing an AW-convolution, where the shape of attention maps
matches that of the weights rather than the activations. Our proposed attention
module is a complementary method to previous attention-based schemes, such as
those that apply the attention mechanism to explore the relationship between
channel-wise and spatial features. Experiments on several datasets for image
classification and object detection tasks show the effectiveness of our
proposed attention module. In particular, our proposed attention module
achieves 1.00% Top-1 accuracy improvement on ImageNet classification over a
ResNet101 baseline and 0.63 COCO-style Average Precision improvement on the
COCO object detection on top of a Faster R-CNN baseline with the backbone of
ResNet101-FPN. When integrating with the previous attentional activations-based
models, our proposed attention module can further increase their Top-1 accuracy
on ImageNet classification by up to 0.57% and COCO-style Average Precision on
the COCO object detection by up to 0.45. Code and pre-trained models will be
publicly available.
Related papers
- CCDepth: A Lightweight Self-supervised Depth Estimation Network with Enhanced Interpretability [11.076431337488973]
This study proposes a novel hybrid self-supervised depth estimation network, CCDepth, comprising convolutional neural networks (CNNs) and the white-box CRATE network.
This novel network uses CNNs and the CRATE modules to extract local and global information in images, respectively, thereby boosting learning efficiency and reducing model size.
arXiv Detail & Related papers (2024-09-30T04:19:40Z) - DCNN: Dual Cross-current Neural Networks Realized Using An Interactive Deep Learning Discriminator for Fine-grained Objects [48.65846477275723]
This study proposes novel dual-current neural networks (DCNN) to improve the accuracy of fine-grained image classification.
The main novel design features for constructing a weakly supervised learning backbone model DCNN include (a) extracting heterogeneous data, (b) keeping the feature map resolution unchanged, (c) expanding the receptive field, and (d) fusing global representations and local features.
arXiv Detail & Related papers (2024-05-07T07:51:28Z) - Learning Feature Matching via Matchable Keypoint-Assisted Graph Neural
Network [52.29330138835208]
Accurately matching local features between a pair of images is a challenging computer vision task.
Previous studies typically use attention based graph neural networks (GNNs) with fully-connected graphs over keypoints within/across images.
We propose MaKeGNN, a sparse attention-based GNN architecture which bypasses non-repeatable keypoints and leverages matchable ones to guide message passing.
arXiv Detail & Related papers (2023-07-04T02:50:44Z) - Systematic Architectural Design of Scale Transformed Attention Condenser
DNNs via Multi-Scale Class Representational Response Similarity Analysis [93.0013343535411]
We propose a novel type of analysis called Multi-Scale Class Representational Response Similarity Analysis (ClassRepSim)
We show that adding STAC modules to ResNet style architectures can result in up to a 1.6% increase in top-1 accuracy.
Results from ClassRepSim analysis can be used to select an effective parameterization of the STAC module resulting in competitive performance.
arXiv Detail & Related papers (2023-06-16T18:29:26Z) - A Generic Shared Attention Mechanism for Various Backbone Neural Networks [53.36677373145012]
Self-attention modules (SAMs) produce strongly correlated attention maps across different layers.
Dense-and-Implicit Attention (DIA) shares SAMs across layers and employs a long short-term memory module.
Our simple yet effective DIA can consistently enhance various network backbones.
arXiv Detail & Related papers (2022-10-27T13:24:08Z) - TDAN: Top-Down Attention Networks for Enhanced Feature Selectivity in
CNNs [18.24779045808196]
We propose a lightweight top-down (TD) attention module that iteratively generates a "visual searchlight" to perform top-down channel and spatial modulation of its inputs.
Our models are more robust to changes in input resolution during inference and learn to "shift attention" by localizing individual objects or features at each computation step without any explicit supervision.
arXiv Detail & Related papers (2021-11-26T12:35:17Z) - Bayesian Attention Belief Networks [59.183311769616466]
Attention-based neural networks have achieved state-of-the-art results on a wide range of tasks.
This paper introduces Bayesian attention belief networks, which construct a decoder network by modeling unnormalized attention weights.
We show that our method outperforms deterministic attention and state-of-the-art attention in accuracy, uncertainty estimation, generalization across domains, and adversarial attacks.
arXiv Detail & Related papers (2021-06-09T17:46:22Z) - AGSFCOS: Based on attention mechanism and Scale-Equalizing pyramid
network of object detection [10.824032219531095]
Our model has a certain improvement in accuracy compared with the current popular detection models on the COCO dataset.
Our optimal model can get 39.5% COCO AP under the background of ResNet50.
arXiv Detail & Related papers (2021-05-20T08:41:02Z) - PC-RGNN: Point Cloud Completion and Graph Neural Network for 3D Object
Detection [57.49788100647103]
LiDAR-based 3D object detection is an important task for autonomous driving.
Current approaches suffer from sparse and partial point clouds of distant and occluded objects.
In this paper, we propose a novel two-stage approach, namely PC-RGNN, dealing with such challenges by two specific solutions.
arXiv Detail & Related papers (2020-12-18T18:06:43Z) - Hybrid Multiple Attention Network for Semantic Segmentation in Aerial
Images [24.35779077001839]
We propose a novel attention-based framework named Hybrid Multiple Attention Network (HMANet) to adaptively capture global correlations.
We introduce a simple yet effective region shuffle attention (RSA) module to reduce feature redundant and improve the efficiency of self-attention mechanism.
arXiv Detail & Related papers (2020-01-09T07:47:51Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.