Attention Mechanism in Neural Networks: Where it Comes and Where it Goes

• URL: http://arxiv.org/abs/2204.13154v1
• Date: Wed, 27 Apr 2022 19:29:09 GMT
• Title: Attention Mechanism in Neural Networks: Where it Comes and Where it Goes
• Authors: Derya Soydaner
• Abstract summary: A long time ago in the machine learning literature, the idea of incorporating a mechanism inspired by the human visual system into neural networks was introduced. This study aims to provide a road map for researchers to explore the current development and get inspired for novel approaches beyond the attention.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A long time ago in the machine learning literature, the idea of incorporating a mechanism inspired by the human visual system into neural networks was introduced. This idea is named the attention mechanism, and it has gone through a long development period. Today, many works have been devoted to this idea in a variety of tasks. Remarkable performance has recently been demonstrated. The goal of this paper is to provide an overview from the early work on searching for ways to implement attention idea with neural networks until the recent trends. This review emphasizes the important milestones during this progress regarding different tasks. By this way, this study aims to provide a road map for researchers to explore the current development and get inspired for novel approaches beyond the attention.

Related papers

• Enhancing Efficiency in Vision Transformer Networks: Design Techniques and Insights [5.798431829723857]
  This paper provides a comprehensive exploration of techniques and insights for designing attention mechanisms in Vision Transformer (ViT) networks. We present a systematic taxonomy of various attention mechanisms within ViTs, employing redesigned approaches. The analysis includes an exploration of the novelty, strengths, weaknesses, and an in-depth evaluation of the different proposed strategies.
  arXiv  Detail & Related papers  (2024-03-28T23:31:59Z)
• Brain-inspired learning in artificial neural networks: a review [5.064447369892274]
  We review current brain-inspired learning representations in artificial neural networks. We investigate the integration of more biologically plausible mechanisms, such as synaptic plasticity, to enhance these networks' capabilities.
  arXiv  Detail & Related papers  (2023-05-18T18:34:29Z)
• VQA and Visual Reasoning: An Overview of Recent Datasets, Methods and Challenges [1.565870461096057]
  The integration of vision and language has sparked a lot of attention as a result of this. The tasks have been created in such a way that they properly exemplify the concepts of deep learning.
  arXiv  Detail & Related papers  (2022-12-26T20:56:01Z)
• Deep Learning to See: Towards New Foundations of Computer Vision [88.69805848302266]
  This book criticizes the supposed scientific progress in the field of computer vision. It proposes the investigation of vision within the framework of information-based laws of nature.
  arXiv  Detail & Related papers  (2022-06-30T15:20:36Z)
• Neural Architecture Search for Dense Prediction Tasks in Computer Vision [74.9839082859151]
  Deep learning has led to a rising demand for neural network architecture engineering. neural architecture search (NAS) aims at automatically designing neural network architectures in a data-driven manner rather than manually. NAS has become applicable to a much wider range of problems in computer vision.
  arXiv  Detail & Related papers  (2022-02-15T08:06:50Z)
• Attention Mechanisms in Computer Vision: A Survey [75.6074182122423]
  We provide a comprehensive review of various attention mechanisms in computer vision. We categorize them according to approach, such as channel attention, spatial attention, temporal attention and branch attention. We suggest future directions for attention mechanism research.
  arXiv  Detail & Related papers  (2021-11-15T09:18:40Z)
• Threat of Adversarial Attacks on Deep Learning in Computer Vision: Survey II [86.51135909513047]
  Deep Learning is vulnerable to adversarial attacks that can manipulate its predictions. This article reviews the contributions made by the computer vision community in adversarial attacks on deep learning. It provides definitions of technical terminologies for non-experts in this domain.
  arXiv  Detail & Related papers  (2021-08-01T08:54:47Z)
• Deep Learning for Embodied Vision Navigation: A Survey [108.13766213265069]
  "Embodied visual navigation" problem requires an agent to navigate in a 3D environment mainly rely on its first-person observation. This paper attempts to establish an outline of the current works in the field of embodied visual navigation by providing a comprehensive literature survey.
  arXiv  Detail & Related papers  (2021-07-07T12:09:04Z)
• Attention, please! A survey of Neural Attention Models in Deep Learning [0.0]
  State-of-the-art in Deep Learning is represented by neural attention models in several application domains. This survey provides a comprehensive overview and analysis of developments in neural attention models.
  arXiv  Detail & Related papers  (2021-03-31T02:42:28Z)
• Deep Reinforced Attention Learning for Quality-Aware Visual Recognition [73.15276998621582]
  We build upon the weakly-supervised generation mechanism of intermediate attention maps in any convolutional neural networks. We introduce a meta critic network to evaluate the quality of attention maps in the main network.
  arXiv  Detail & Related papers  (2020-07-13T02:44:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.