Visual Attention Methods in Deep Learning: An In-Depth Survey

• URL: http://arxiv.org/abs/2204.07756v3
• Date: Sun, 5 May 2024 18:44:14 GMT
• Title: Visual Attention Methods in Deep Learning: An In-Depth Survey
• Authors: Mohammed Hassanin, Saeed Anwar, Ibrahim Radwan, Fahad S Khan, Ajmal Mian,
• Abstract summary: Inspired by the human cognitive system, attention is a mechanism that imitates the human cognitive awareness about specific information. Deep learning has employed attention to boost performance for many applications. The literature lacks a comprehensive survey on attention techniques to guide researchers in employing attention in their deep models.
• Score: 37.18104595529633
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Inspired by the human cognitive system, attention is a mechanism that imitates the human cognitive awareness about specific information, amplifying critical details to focus more on the essential aspects of data. Deep learning has employed attention to boost performance for many applications. Interestingly, the same attention design can suit processing different data modalities and can easily be incorporated into large networks. Furthermore, multiple complementary attention mechanisms can be incorporated into one network. Hence, attention techniques have become extremely attractive. However, the literature lacks a comprehensive survey on attention techniques to guide researchers in employing attention in their deep models. Note that, besides being demanding in terms of training data and computational resources, transformers only cover a single category in self-attention out of the many categories available. We fill this gap and provide an in-depth survey of 50 attention techniques, categorizing them by their most prominent features. We initiate our discussion by introducing the fundamental concepts behind the success of the attention mechanism. Next, we furnish some essentials such as the strengths and limitations of each attention category, describe their fundamental building blocks, basic formulations with primary usage, and applications specifically for computer vision. We also discuss the challenges and general open questions related to attention mechanisms. Finally, we recommend possible future research directions for deep attention. All the information about visual attention methods in deep learning is provided at \href{https://github.com/saeed-anwar/VisualAttention}{https://github.com/saeed-anwar/VisualAttention}

Related papers

• Learning to mask: Towards generalized face forgery detection [10.155873909545198]
  Generalizability to unseen forgery types is crucial for face forgery detectors. Our goal is to reduce the features that are easy to learn in the training phase, so as to reduce the risk of overfitting on specific forgery types. A deep feature mixup strategy is also proposed to synthesize forgeries in the feature domain.
  arXiv  Detail & Related papers  (2022-12-29T13:55:28Z)
• Attention Mechanisms in Computer Vision: A Survey [75.6074182122423]
  We provide a comprehensive review of various attention mechanisms in computer vision. We categorize them according to approach, such as channel attention, spatial attention, temporal attention and branch attention. We suggest future directions for attention mechanism research.
  arXiv  Detail & Related papers  (2021-11-15T09:18:40Z)
• Learning to ignore: rethinking attention in CNNs [87.01305532842878]
  We propose to reformulate the attention mechanism in CNNs to learn to ignore instead of learning to attend. Specifically, we propose to explicitly learn irrelevant information in the scene and suppress it in the produced representation.
  arXiv  Detail & Related papers  (2021-11-10T13:47:37Z)
• Alignment Attention by Matching Key and Query Distributions [48.93793773929006]
  This paper introduces alignment attention that explicitly encourages self-attention to match the distributions of the key and query within each head. It is simple to convert any models with self-attention, including pre-trained ones, to the proposed alignment attention. On a variety of language understanding tasks, we show the effectiveness of our method in accuracy, uncertainty estimation, generalization across domains, and robustness to adversarial attacks.
  arXiv  Detail & Related papers  (2021-10-25T00:54:57Z)
• Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification [101.49122450005869]
  We present a counterfactual attention learning method to learn more effective attention based on causal inference. Specifically, we analyze the effect of the learned visual attention on network prediction. We evaluate our method on a wide range of fine-grained recognition tasks.
  arXiv  Detail & Related papers  (2021-08-19T14:53:40Z)
• Understanding top-down attention using task-oriented ablation design [0.22940141855172028]
  Top-down attention allows neural networks, both artificial and biological, to focus on the information most relevant for a given task. We aim to answer this with a computational experiment based on a general framework called task-oriented ablation design. We compare the performance of two neural networks, one with top-down attention and one without.
  arXiv  Detail & Related papers  (2021-06-08T21:01:47Z)
• Attention, please! A survey of Neural Attention Models in Deep Learning [0.0]
  State-of-the-art in Deep Learning is represented by neural attention models in several application domains. This survey provides a comprehensive overview and analysis of developments in neural attention models.
  arXiv  Detail & Related papers  (2021-03-31T02:42:28Z)
• Deep Reinforced Attention Learning for Quality-Aware Visual Recognition [73.15276998621582]
  We build upon the weakly-supervised generation mechanism of intermediate attention maps in any convolutional neural networks. We introduce a meta critic network to evaluate the quality of attention maps in the main network.
  arXiv  Detail & Related papers  (2020-07-13T02:44:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.