Related papers: From Cognition to Computation: A Comparative Review of Human Attention and Transformer Architectures

From Cognition to Computation: A Comparative Review of Human Attention and Transformer Architectures

URL: http://arxiv.org/abs/2407.01548v1
Date: Thu, 25 Apr 2024 05:13:38 GMT
Title: From Cognition to Computation: A Comparative Review of Human Attention and Transformer Architectures
Authors: Minglu Zhao, Dehong Xu, Tao Gao,
Abstract summary: Recent developments in artificial intelligence like the Transformer architecture incorporate the idea of attention in model designs. Our review aims to provide a comparative analysis of these mechanisms from a cognitive-functional perspective.
Score: 1.5266118210763295
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: Attention is a cornerstone of human cognition that facilitates the efficient extraction of information in everyday life. Recent developments in artificial intelligence like the Transformer architecture also incorporate the idea of attention in model designs. However, despite the shared fundamental principle of selectively attending to information, human attention and the Transformer model display notable differences, particularly in their capacity constraints, attention pathways, and intentional mechanisms. Our review aims to provide a comparative analysis of these mechanisms from a cognitive-functional perspective, thereby shedding light on several open research questions. The exploration encourages interdisciplinary efforts to derive insights from human attention mechanisms in the pursuit of developing more generalized artificial intelligence.

Related papers

EyeTrans: Merging Human and Machine Attention for Neural Code Summarization [16.694601606682046]
We develop a method for incorporating human attention into machine attention to enhance neural code summarization. We conduct comprehensive experiments on two code summarization tasks to demonstrate the effectiveness of incorporating human attention into Transformers.
arXiv Detail & Related papers (2024-02-21T19:45:06Z)
A Neuro-mimetic Realization of the Common Model of Cognition via Hebbian Learning and Free Energy Minimization [55.11642177631929]
Large neural generative models are capable of synthesizing semantically rich passages of text or producing complex images. We discuss the COGnitive Neural GENerative system, such an architecture that casts the Common Model of Cognition.
arXiv Detail & Related papers (2023-10-14T23:28:48Z)
AttentionViz: A Global View of Transformer Attention [60.82904477362676]
We present a new visualization technique designed to help researchers understand the self-attention mechanism in transformers. The main idea behind our method is to visualize a joint embedding of the query and key vectors used by transformer models to compute attention. We create an interactive visualization tool, AttentionViz, based on these joint query-key embeddings.
arXiv Detail & Related papers (2023-05-04T23:46:49Z)
Attention: Marginal Probability is All You Need? [0.0]
We propose an alternative Bayesian foundation for attentional mechanisms. We show how this unifies different attentional architectures in machine learning. We hope this work will guide more sophisticated intuitions into the key properties of attention architectures.
arXiv Detail & Related papers (2023-04-07T14:38:39Z)
Machine Psychology [54.287802134327485]
We argue that a fruitful direction for research is engaging large language models in behavioral experiments inspired by psychology. We highlight theoretical perspectives, experimental paradigms, and computational analysis techniques that this approach brings to the table. It paves the way for a "machine psychology" for generative artificial intelligence (AI) that goes beyond performance benchmarks.
arXiv Detail & Related papers (2023-03-24T13:24:41Z)
Self-attention in Vision Transformers Performs Perceptual Grouping, Not Attention [11.789983276366986]
We show that attention mechanisms in vision transformers exhibit similar effects as those known in human visual attention. Our results suggest that self-attention modules group figures in the stimuli based on similarity in visual features such as color. In a singleton detection experiment, we studied if these models exhibit similar effects as those of feed-forward visual salience mechanisms utilized in human visual attention.
arXiv Detail & Related papers (2023-03-02T19:18:11Z)
Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attention [66.93307963324834]
We propose to augment the transformer architecture with an external attention mechanism to bring external knowledge and context to bear. We find that the proposed external attention mechanism can significantly improve the performance of existing AI systems. The proposed system reaches human parity on the open CommonsenseQA research benchmark with an accuracy of 89.4% in comparison to the human accuracy of 88.9%.
arXiv Detail & Related papers (2021-12-06T18:59:02Z)
Attention Mechanisms in Computer Vision: A Survey [75.6074182122423]
We provide a comprehensive review of various attention mechanisms in computer vision. We categorize them according to approach, such as channel attention, spatial attention, temporal attention and branch attention. We suggest future directions for attention mechanism research.
arXiv Detail & Related papers (2021-11-15T09:18:40Z)
A Review on Intelligent Object Perception Methods Combining Knowledge-based Reasoning and Machine Learning [60.335974351919816]
Object perception is a fundamental sub-field of Computer Vision. Recent works seek ways to integrate knowledge engineering in order to expand the level of intelligence of the visual interpretation of objects.
arXiv Detail & Related papers (2019-12-26T13:26:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.