Neural Attention Models in Deep Learning: Survey and Taxonomy

• URL: http://arxiv.org/abs/2112.05909v1
• Date: Sat, 11 Dec 2021 03:35:33 GMT
• Title: Neural Attention Models in Deep Learning: Survey and Taxonomy
• Authors: Alana Santana and Esther Colombini
• Abstract summary: Concepts and functions of attention have been studied in philosophy, psychology, neuroscience, and computing. Many different neural attention models are now available and have been a very active research area over the past six years. Here we propose a taxonomy that corroborates with theoretical aspects that predate Deep Learning.
• Score: 0.0
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Attention is a state of arousal capable of dealing with limited processing bottlenecks in human beings by focusing selectively on one piece of information while ignoring other perceptible information. For decades, concepts and functions of attention have been studied in philosophy, psychology, neuroscience, and computing. Currently, this property has been widely explored in deep neural networks. Many different neural attention models are now available and have been a very active research area over the past six years. From the theoretical standpoint of attention, this survey provides a critical analysis of major neural attention models. Here we propose a taxonomy that corroborates with theoretical aspects that predate Deep Learning. Our taxonomy provides an organizational structure that asks new questions and structures the understanding of existing attentional mechanisms. In particular, 17 criteria derived from psychology and neuroscience classic studies are formulated for qualitative comparison and critical analysis on the 51 main models found on a set of more than 650 papers analyzed. Also, we highlight several theoretical issues that have not yet been explored, including discussions about biological plausibility, highlight current research trends, and provide insights for the future.

Related papers

• Brain-Inspired Machine Intelligence: A Survey of Neurobiologically-Plausible Credit Assignment [65.268245109828]
  We examine algorithms for conducting credit assignment in artificial neural networks that are inspired or motivated by neurobiology. We organize the ever-growing set of brain-inspired learning schemes into six general families and consider these in the context of backpropagation of errors. The results of this review are meant to encourage future developments in neuro-mimetic systems and their constituent learning processes.
  arXiv  Detail & Related papers  (2023-12-01T05:20:57Z)
• A Neuro-mimetic Realization of the Common Model of Cognition via Hebbian Learning and Free Energy Minimization [55.11642177631929]
  Large neural generative models are capable of synthesizing semantically rich passages of text or producing complex images. We discuss the COGnitive Neural GENerative system, such an architecture that casts the Common Model of Cognition.
  arXiv  Detail & Related papers  (2023-10-14T23:28:48Z)
• Towards Data-and Knowledge-Driven Artificial Intelligence: A Survey on Neuro-Symbolic Computing [73.0977635031713]
  Neural-symbolic computing (NeSy) has been an active research area of Artificial Intelligence (AI) for many years. NeSy shows promise of reconciling the advantages of reasoning and interpretability of symbolic representation and robust learning in neural networks.
  arXiv  Detail & Related papers  (2022-10-28T04:38:10Z)
• Neural Fields in Visual Computing and Beyond [54.950885364735804]
  Recent advances in machine learning have created increasing interest in solving visual computing problems using coordinate-based neural networks. neural fields have seen successful application in the synthesis of 3D shapes and image, animation of human bodies, 3D reconstruction, and pose estimation. This report provides context, mathematical grounding, and an extensive review of literature on neural fields.
  arXiv  Detail & Related papers  (2021-11-22T18:57:51Z)
• Information theoretic analysis of computational models as a tool to understand the neural basis of behaviors [0.0]
  One of the greatest research challenges of this century is to understand the neural basis for how behavior emerges in brain-body-environment systems. Computational models provide an alternative framework within which one can study model systems. I provide an introduction, a review and discussion to make a case for how information theoretic analysis of computational models is a potent research methodology.
  arXiv  Detail & Related papers  (2021-06-02T02:08:18Z)
• Attention, please! A survey of Neural Attention Models in Deep Learning [0.0]
  State-of-the-art in Deep Learning is represented by neural attention models in several application domains. This survey provides a comprehensive overview and analysis of developments in neural attention models.
  arXiv  Detail & Related papers  (2021-03-31T02:42:28Z)
• Topic Modelling Meets Deep Neural Networks: A Survey [25.950652301810425]
  Topic modelling has been a successful technique for text analysis for almost twenty years. When topic modelling met deep neural networks, there emerged a new and increasingly popular research area, neural topic models. This paper provides a focused yet comprehensive overview of neural topic models for interested researchers in the AI community.
  arXiv  Detail & Related papers  (2021-02-28T12:59:28Z)
• Developing Constrained Neural Units Over Time [81.19349325749037]
  This paper focuses on an alternative way of defining Neural Networks, that is different from the majority of existing approaches. The structure of the neural architecture is defined by means of a special class of constraints that are extended also to the interaction with data. The proposed theory is cast into the time domain, in which data are presented to the network in an ordered manner.
  arXiv  Detail & Related papers  (2020-09-01T09:07:25Z)
• Deep Reinforcement Learning and its Neuroscientific Implications [19.478332877763417]
  The emergence of powerful artificial intelligence is defining new research directions in neuroscience. Deep reinforcement learning (Deep RL) offers a framework for studying the interplay among learning, representation and decision-making. Deep RL offers a new set of research tools and a wide range of novel hypotheses.
  arXiv  Detail & Related papers  (2020-07-07T19:27:54Z)
• Towards a Neural Model for Serial Order in Frontal Cortex: a Brain Theory from Memory Development to Higher-Level Cognition [53.816853325427424]
  We propose that the immature prefrontal cortex (PFC) use its primary functionality of detecting hierarchical patterns in temporal signals. Our hypothesis is that the PFC detects the hierarchical structure in temporal sequences in the form of ordinal patterns and use them to index information hierarchically in different parts of the brain. By doing so, it gives the tools to the language-ready brain for manipulating abstract knowledge and planning temporally ordered information.
  arXiv  Detail & Related papers  (2020-05-22T14:29:51Z)
• On Interpretability of Artificial Neural Networks: A Survey [21.905647127437685]
  We systematically review recent studies in understanding the mechanism of neural networks, describe applications of interpretability especially in medicine. We discuss future directions of interpretability research, such as in relation to fuzzy logic and brain science.
  arXiv  Detail & Related papers  (2020-01-08T13:40:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.