Convolutional Neural Network optimization via Channel Reassessment Attention module

• URL: http://arxiv.org/abs/2010.05605v1
• Date: Mon, 12 Oct 2020 11:27:17 GMT
• Title: Convolutional Neural Network optimization via Channel Reassessment Attention module
• Authors: YuTao Shen and Ying Wen
• Abstract summary: We propose a novel network optimization module called Channel Reassessment (CRA) module. CRA module uses channel attentions with spatial information of feature maps to enhance representational power of networks. Experiments on ImageNet and MS datasets demonstrate that embedding CRA module on various networks effectively improves the performance under different evaluation standards.
• Score: 19.566271646280978
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The performance of convolutional neural networks (CNNs) can be improved by adjusting the interrelationship between channels with attention mechanism. However, attention mechanism in recent advance has not fully utilized spatial information of feature maps, which makes a great difference to the results of generated channel attentions. In this paper, we propose a novel network optimization module called Channel Reassessment Attention (CRA) module which uses channel attentions with spatial information of feature maps to enhance representational power of networks. We employ CRA module to assess channel attentions based on feature maps in different channels, then the final features are refined adaptively by product between channel attentions and feature maps.CRA module is a computational lightweight module and it can be embedded into any architectures of CNNs. The experiments on ImageNet, CIFAR and MS COCO datasets demonstrate that the embedding of CRA module on various networks effectively improves the performance under different evaluation standards.

Related papers

• CSA-Net: Channel-wise Spatially Autocorrelated Attention Networks [19.468704622654357]
  We present a channel-wise spatially autocorrelated (CSA) attention mechanism for deep CNNs. Inspired by geographical analysis, the proposed CSA exploits the spatial relationships between channels of feature maps to produce an effective channel descriptor. We validate the effectiveness of the proposed CSA networks through extensive experiments and analysis on ImageNet, and MS COCO benchmark datasets.
  arXiv  Detail & Related papers  (2024-05-09T13:21:03Z)
• TCCT-Net: Two-Stream Network Architecture for Fast and Efficient Engagement Estimation via Behavioral Feature Signals [58.865901821451295]
  We present a novel two-stream feature fusion "Tensor-Convolution and Convolution-Transformer Network" (TCCT-Net) architecture. To better learn the meaningful patterns in the temporal-spatial domain, we design a "CT" stream that integrates a hybrid convolutional-transformer. In parallel, to efficiently extract rich patterns from the temporal-frequency domain, we introduce a "TC" stream that uses Continuous Wavelet Transform (CWT) to represent information in a 2D tensor form.
  arXiv  Detail & Related papers  (2024-04-15T06:01:48Z)
• TBSN: Transformer-Based Blind-Spot Network for Self-Supervised Image Denoising [94.09442506816724]
  Blind-spot networks (BSN) have been prevalent network architectures in self-supervised image denoising (SSID) We present a transformer-based blind-spot network (TBSN) by analyzing and redesigning the transformer operators that meet the blind-spot requirement. For spatial self-attention, an elaborate mask is applied to the attention matrix to restrict its receptive field, thus mimicking the dilated convolution. For channel self-attention, we observe that it may leak the blind-spot information when the channel number is greater than spatial size in the deep layers of multi-scale architectures.
  arXiv  Detail & Related papers  (2024-04-11T15:39:10Z)
• MCA: Moment Channel Attention Networks [10.780493635885225]
  We investigate the statistical moments of feature maps within a neural network. Our findings highlight the critical role of high-order moments in enhancing model capacity. We propose the Moment Channel Attention (MCA) framework, which efficiently incorporates multiple levels of moment-based information.
  arXiv  Detail & Related papers  (2024-03-04T04:02:59Z)
• An Efficient Speech Separation Network Based on Recurrent Fusion Dilated Convolution and Channel Attention [0.2538209532048866]
  We present an efficient speech separation neural network, ARFDCN, which combines dilated convolutions, multi-scale fusion (MSF), and channel attention. Experimental results indicate that the model achieves a decent balance between performance and computational efficiency.
  arXiv  Detail & Related papers  (2023-06-09T13:30:27Z)
• Joint Channel Estimation and Feedback with Masked Token Transformers in Massive MIMO Systems [74.52117784544758]
  This paper proposes an encoder-decoder based network that unveils the intrinsic frequency-domain correlation within the CSI matrix. The entire encoder-decoder network is utilized for channel compression. Our method outperforms state-of-the-art channel estimation and feedback techniques in joint tasks.
  arXiv  Detail & Related papers  (2023-06-08T06:15:17Z)
• A Generic Shared Attention Mechanism for Various Backbone Neural Networks [53.36677373145012]
  Self-attention modules (SAMs) produce strongly correlated attention maps across different layers. Dense-and-Implicit Attention (DIA) shares SAMs across layers and employs a long short-term memory module. Our simple yet effective DIA can consistently enhance various network backbones.
  arXiv  Detail & Related papers  (2022-10-27T13:24:08Z)
• A Discriminative Channel Diversification Network for Image Classification [21.049734250642974]
  We propose a light-weight and effective attention module, called channel diversification block, to enhance the global context. Unlike other channel attention mechanisms, the proposed module focuses on the most discriminative features. Experiments on CIFAR-10, SVHN, and Tiny-ImageNet datasets demonstrate that the proposed module improves the performance of the baseline networks by a margin of 3% on average.
  arXiv  Detail & Related papers  (2021-12-10T23:00:53Z)
• TDAN: Top-Down Attention Networks for Enhanced Feature Selectivity in CNNs [18.24779045808196]
  We propose a lightweight top-down (TD) attention module that iteratively generates a "visual searchlight" to perform top-down channel and spatial modulation of its inputs. Our models are more robust to changes in input resolution during inference and learn to "shift attention" by localizing individual objects or features at each computation step without any explicit supervision.
  arXiv  Detail & Related papers  (2021-11-26T12:35:17Z)
• Channelized Axial Attention for Semantic Segmentation [70.14921019774793]
  We propose the Channelized Axial Attention (CAA) to seamlessly integratechannel attention and axial attention with reduced computationalcomplexity. Our CAA not onlyrequires much less computation resources compared with otherdual attention models such as DANet, but also outperforms the state-of-the-art ResNet-101-based segmentation models on alltested datasets.
  arXiv  Detail & Related papers  (2021-01-19T03:08:03Z)
• Volumetric Transformer Networks [88.85542905676712]
  We introduce a learnable module, the volumetric transformer network (VTN) VTN predicts channel-wise warping fields so as to reconfigure intermediate CNN features spatially and channel-wisely. Our experiments show that VTN consistently boosts the features' representation power and consequently the networks' accuracy on fine-grained image recognition and instance-level image retrieval.
  arXiv  Detail & Related papers  (2020-07-18T14:00:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.