ConAM: Confidence Attention Module for Convolutional Neural Networks

• URL: http://arxiv.org/abs/2110.14369v1
• Date: Wed, 27 Oct 2021 12:06:31 GMT
• Title: ConAM: Confidence Attention Module for Convolutional Neural Networks
• Authors: Yu Xue, Ziming Yuan and Ferrante Neri
• Abstract summary: We propose a new attention mechanism based on the correlation between local and global contextual information. Our method suppresses useless information while enhancing the informative one with fewer parameters. We implement ConAM with the Python library, Pytorch, and the code and models will be publicly available.
• Score: 1.3571579680845614
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The so-called ``attention'' is an efficient mechanism to improve the performance of convolutional neural networks. It uses contextual information to recalibrate the input to strengthen the propagation of informative features. However, the majority of the attention mechanisms only consider either local or global contextual information, which is singular to extract features. Moreover, many existing mechanisms directly use the contextual information to recalibrate the input, which unilaterally enhances the propagation of the informative features, but does not suppress the useless ones. This paper proposes a new attention mechanism module based on the correlation between local and global contextual information and we name this correlation as confidence. The novel attention mechanism extracts the local and global contextual information simultaneously, and calculates the confidence between them, then uses this confidence to recalibrate the input pixels. The extraction of local and global contextual information increases the diversity of features. The recalibration with confidence suppresses useless information while enhancing the informative one with fewer parameters. We use CIFAR-10 and CIFAR-100 in our experiments and explore the performance of our method's components by sufficient ablation studies. Finally, we compare our method with a various state-of-the-art convolutional neural networks and the results show that our method completely surpasses these models. We implement ConAM with the Python library, Pytorch, and the code and models will be publicly available.

Related papers

• Localized Gaussians as Self-Attention Weights for Point Clouds Correspondence [92.07601770031236]
  We investigate semantically meaningful patterns in the attention heads of an encoder-only Transformer architecture. We find that fixing the attention weights not only accelerates the training process but also enhances the stability of the optimization.
  arXiv  Detail & Related papers  (2024-09-20T07:41:47Z)
• Self-Attention-Based Contextual Modulation Improves Neural System Identification [2.784365807133169]
  Cortical neurons in the primary visual cortex are sensitive to contextual information mediated by horizontal and feedback connections. CNNs integrate global contextual information to model contextual modulation via two mechanisms: successive convolutions and a fully connected readout layer. We find that self-attention can improve neural response predictions over parameter-matched CNNs in two key metrics: tuning curve correlation and peak tuning.
  arXiv  Detail & Related papers  (2024-06-12T03:21:06Z)
• Heterogenous Memory Augmented Neural Networks [84.29338268789684]
  We introduce a novel heterogeneous memory augmentation approach for neural networks. By introducing learnable memory tokens with attention mechanism, we can effectively boost performance without huge computational overhead. We show our approach on various image and graph-based tasks under both in-distribution (ID) and out-of-distribution (OOD) conditions.
  arXiv  Detail & Related papers  (2023-10-17T01:05:28Z)
• Adaptive Local-Component-aware Graph Convolutional Network for One-shot Skeleton-based Action Recognition [54.23513799338309]
  We present an Adaptive Local-Component-aware Graph Convolutional Network for skeleton-based action recognition. Our method provides a stronger representation than the global embedding and helps our model reach state-of-the-art.
  arXiv  Detail & Related papers  (2022-09-21T02:33:07Z)
• HDAM: Heuristic Difference Attention Module for Convolutional Neural Networks [1.1125818448814198]
  The attention mechanism is one of the most important priori knowledge to enhance convolutional neural networks. This article proposes a novel attention mechanism with the difference attention module, HDAM. We implement HDAM with the Python library, Pytorch, and the code and models will be publicly available.
  arXiv  Detail & Related papers  (2022-02-19T09:19:01Z)
• Deep Archimedean Copulas [98.96141706464425]
  ACNet is a novel differentiable neural network architecture that enforces structural properties. We show that ACNet is able to both approximate common Archimedean Copulas and generate new copulas which may provide better fits to data.
  arXiv  Detail & Related papers  (2020-12-05T22:58:37Z)
• Data-Informed Global Sparseness in Attention Mechanisms for Deep Neural Networks [33.07113523598028]
  We propose Attention Pruning (AP), a framework that observes attention patterns in a fixed dataset and generates a global sparseness mask. AP saves 90% of attention computation for language modeling and about 50% for machine translation and GLUE tasks, maintaining result quality.
  arXiv  Detail & Related papers  (2020-11-20T13:58:21Z)
• BayGo: Joint Bayesian Learning and Information-Aware Graph Optimization [48.30183416069897]
  BayGo is a novel fully decentralized joint Bayesian learning and graph optimization framework. We show that our framework achieves faster convergence and higher accuracy compared to fully-connected and star topology graphs.
  arXiv  Detail & Related papers  (2020-11-09T11:16:55Z)
• Focus of Attention Improves Information Transfer in Visual Features [80.22965663534556]
  This paper focuses on unsupervised learning for transferring visual information in a truly online setting. The computation of the entropy terms is carried out by a temporal process which yields online estimation of the entropy terms. In order to better structure the input probability distribution, we use a human-like focus of attention model.
  arXiv  Detail & Related papers  (2020-06-16T15:07:25Z)
• Incorporating Effective Global Information via Adaptive Gate Attention for Text Classification [13.45504908358177]
  We show that simple statistical information can enhance classification performance both efficiently and significantly compared with several baseline models. We propose a classifier with gate mechanism named Adaptive Gate Attention model with Global Information (AGA+GI) in which the adaptive gate mechanism incorporates global statistical features into latent semantic features. Our experiments show that the proposed method can achieve better accuracy than CNN-based and RNN-based approaches without global information on several benchmarks.
  arXiv  Detail & Related papers  (2020-02-22T10:06:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.