HDAM: Heuristic Difference Attention Module for Convolutional Neural Networks

• URL: http://arxiv.org/abs/2202.09556v1
• Date: Sat, 19 Feb 2022 09:19:01 GMT
• Title: HDAM: Heuristic Difference Attention Module for Convolutional Neural Networks
• Authors: Yu Xue and Ziming Yuan
• Abstract summary: The attention mechanism is one of the most important priori knowledge to enhance convolutional neural networks. This article proposes a novel attention mechanism with the difference attention module, HDAM. We implement HDAM with the Python library, Pytorch, and the code and models will be publicly available.
• Score: 1.1125818448814198
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The attention mechanism is one of the most important priori knowledge to enhance convolutional neural networks. Most attention mechanisms are bound to the convolutional layer and use local or global contextual information to recalibrate the input. This is a popular attention strategy design method. Global contextual information helps the network to consider the overall distribution, while local contextual information is more general. The contextual information makes the network pay attention to the mean or maximum value of a particular receptive field. Different from the most attention mechanism, this article proposes a novel attention mechanism with the heuristic difference attention module, HDAM. HDAM's input recalibration is based on the difference between the local and global contextual information instead of the mean and maximum values. At the same time, to make different layers have a more suitable local receptive field size and increase the exibility of the local receptive field design, we use genetic algorithm to heuristically produce local receptive fields. First, HDAM extracts the mean value of the global and local receptive fields as the corresponding contextual information. Then the difference between the global and local contextual information is calculated. Finally HDAM uses this difference to recalibrate the input. In addition, we use the heuristic ability of genetic algorithm to search for the local receptive field size of each layer. Our experiments on CIFAR-10 and CIFAR-100 show that HDAM can use fewer parameters than other attention mechanisms to achieve higher accuracy. We implement HDAM with the Python library, Pytorch, and the code and models will be publicly available.

Related papers

• Deep Homography Estimation for Visual Place Recognition [49.235432979736395]
  We propose a transformer-based deep homography estimation (DHE) network. It takes the dense feature map extracted by a backbone network as input and fits homography for fast and learnable geometric verification. Experiments on benchmark datasets show that our method can outperform several state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-02-25T13:22:17Z)
• Exploring and Exploiting Multi-Granularity Representations for Machine Reading Comprehension [13.191437539419681]
  We propose a novel approach called Adaptive Bidirectional Attention-Capsule Network (ABA-Net) ABA-Net adaptively exploits the source representations of different levels to the predictor. We set the new state-of-the-art performance on the SQuAD 1.0 dataset.
  arXiv  Detail & Related papers  (2022-08-18T10:14:32Z)
• SAR Despeckling Using Overcomplete Convolutional Networks [53.99620005035804]
  despeckling is an important problem in remote sensing as speckle degrades SAR images. Recent studies show that convolutional neural networks(CNNs) outperform classical despeckling methods. This study employs an overcomplete CNN architecture to focus on learning low-level features by restricting the receptive field. We show that the proposed network improves despeckling performance compared to recent despeckling methods on synthetic and real SAR images.
  arXiv  Detail & Related papers  (2022-05-31T15:55:37Z)
• Federated and Generalized Person Re-identification through Domain and Feature Hallucinating [88.77196261300699]
  We study the problem of federated domain generalization (FedDG) for person re-identification (re-ID) We propose a novel method, called "Domain and Feature Hallucinating (DFH)", to produce diverse features for learning generalized local and global models. Our method achieves the state-of-the-art performance for FedDG on four large-scale re-ID benchmarks.
  arXiv  Detail & Related papers  (2022-03-05T09:15:13Z)
• DeHIN: A Decentralized Framework for Embedding Large-scale Heterogeneous Information Networks [64.62314068155997]
  We present textitDecentralized Embedding Framework for Heterogeneous Information Network (DeHIN) in this paper. DeHIN presents a context preserving partition mechanism that innovatively formulates a large HIN as a hypergraph. Our framework then adopts a decentralized strategy to efficiently partition HINs by adopting a tree-like pipeline.
  arXiv  Detail & Related papers  (2022-01-08T04:08:36Z)
• ConAM: Confidence Attention Module for Convolutional Neural Networks [1.3571579680845614]
  We propose a new attention mechanism based on the correlation between local and global contextual information. Our method suppresses useless information while enhancing the informative one with fewer parameters. We implement ConAM with the Python library, Pytorch, and the code and models will be publicly available.
  arXiv  Detail & Related papers  (2021-10-27T12:06:31Z)
• Clustered Federated Learning via Generalized Total Variation Minimization [83.26141667853057]
  We study optimization methods to train local (or personalized) models for local datasets with a decentralized network structure. Our main conceptual contribution is to formulate federated learning as total variation minimization (GTV) Our main algorithmic contribution is a fully decentralized federated learning algorithm.
  arXiv  Detail & Related papers  (2021-05-26T18:07:19Z)
• Global Context Aware RCNN for Object Detection [1.1939762265857436]
  We propose a novel end-to-end trainable framework, called Global Context Aware (GCA) RCNN. The core component of GCA framework is a context aware mechanism, in which both global feature pyramid and attention strategies are used for feature extraction and feature refinement. In the end, we also present a lightweight version of our method, which only slightly increases model complexity and computational burden.
  arXiv  Detail & Related papers  (2020-12-04T14:56:46Z)
• Gait Recognition via Effective Global-Local Feature Representation and Local Temporal Aggregation [28.721376937882958]
  Gait recognition is one of the most important biometric technologies and has been applied in many fields. Recent gait recognition frameworks represent each gait frame by descriptors extracted from either global appearances or local regions of humans. We propose a novel feature extraction and fusion framework to achieve discriminative feature representations for gait recognition.
  arXiv  Detail & Related papers  (2020-11-03T04:07:13Z)
• An Explicit Local and Global Representation Disentanglement Framework with Applications in Deep Clustering and Unsupervised Object Detection [9.609936822226633]
  We propose a framework, called SPLIT, which allows us to disentangle local and global information. Our framework adds generative assumption to the variational autoencoder (VAE) framework. We show that the framework can effectively disentangle local and global information within these models.
  arXiv  Detail & Related papers  (2020-01-24T12:09:20Z)
• Dense Residual Network: Enhancing Global Dense Feature Flow for Character Recognition [75.4027660840568]
  This paper explores how to enhance the local and global dense feature flow by exploiting hierarchical features fully from all the convolution layers. Technically, we propose an efficient and effective CNN framework, i.e., Fast Dense Residual Network (FDRN) for text recognition.
  arXiv  Detail & Related papers  (2020-01-23T06:55:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.