Delving Deeper Into Astromorphic Transformers

• URL: http://arxiv.org/abs/2312.10925v2
• Date: Tue, 9 Jan 2024 23:27:24 GMT
• Title: Delving Deeper Into Astromorphic Transformers
• Authors: Md Zesun Ahmed Mia, Malyaban Bal, Abhronil Sengupta
• Abstract summary: This paper seeks to delve deeper into various key aspects of neuron-synapse-astrocyte interactions to mimic self-attention mechanisms in Transformers. The cross-layer perspective explored in this work involves bio-plausible modeling of Hebbian and pre-synaptic plasticities in neuron-astrocyte networks. Our analysis on sentiment and image classification tasks on the IMDB and CIFAR10 datasets underscores the importance of constructing Astromorphic Transformers from both accuracy and learning speed improvement perspectives.
• Score: 1.9775291915550175
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Preliminary attempts at incorporating the critical role of astrocytes - cells that constitute more than 50% of human brain cells - in brain-inspired neuromorphic computing remain in infancy. This paper seeks to delve deeper into various key aspects of neuron-synapse-astrocyte interactions to mimic self-attention mechanisms in Transformers. The cross-layer perspective explored in this work involves bio-plausible modeling of Hebbian and pre-synaptic plasticities in neuron-astrocyte networks, incorporating effects of non-linearities and feedback along with algorithmic formulations to map the neuron-astrocyte computations to self-attention mechanism and evaluating the impact of incorporating bio-realistic effects from the machine learning application side. Our analysis on sentiment and image classification tasks on the IMDB and CIFAR10 datasets underscores the importance of constructing Astromorphic Transformers from both accuracy and learning speed improvement perspectives.

Related papers

• A Fuzzy-based Approach to Predict Human Interaction by Functional Near-Infrared Spectroscopy [25.185426359719454]
  The paper introduces a Fuzzy-based Attention (Fuzzy Attention Layer) mechanism, a novel computational approach to interpretability and efficacy of neural models in psychological research. By leveraging fuzzy logic, the Fuzzy Attention Layer is capable of learning and identifying interpretable patterns of neural activity.
  arXiv  Detail & Related papers  (2024-09-26T09:20:12Z)
• Exploring neural oscillations during speech perception via surrogate gradient spiking neural networks [59.38765771221084]
  We present a physiologically inspired speech recognition architecture compatible and scalable with deep learning frameworks. We show end-to-end gradient descent training leads to the emergence of neural oscillations in the central spiking neural network. Our findings highlight the crucial inhibitory role of feedback mechanisms, such as spike frequency adaptation and recurrent connections, in regulating and synchronising neural activity to improve recognition performance.
  arXiv  Detail & Related papers  (2024-04-22T09:40:07Z)
• Brain-Inspired Machine Intelligence: A Survey of Neurobiologically-Plausible Credit Assignment [65.268245109828]
  We examine algorithms for conducting credit assignment in artificial neural networks that are inspired or motivated by neurobiology. We organize the ever-growing set of brain-inspired learning schemes into six general families and consider these in the context of backpropagation of errors. The results of this review are meant to encourage future developments in neuro-mimetic systems and their constituent learning processes.
  arXiv  Detail & Related papers  (2023-12-01T05:20:57Z)
• Astrocytes as a mechanism for meta-plasticity and contextually-guided network function [2.66269503676104]
  Astrocytes are a ubiquitous and enigmatic type of non-neuronal cell. Astrocytes may play a more direct and active role in brain function and neural computation.
  arXiv  Detail & Related papers  (2023-11-06T20:31:01Z)
• Contrastive-Signal-Dependent Plasticity: Self-Supervised Learning in Spiking Neural Circuits [61.94533459151743]
  This work addresses the challenge of designing neurobiologically-motivated schemes for adjusting the synapses of spiking networks. Our experimental simulations demonstrate a consistent advantage over other biologically-plausible approaches when training recurrent spiking networks.
  arXiv  Detail & Related papers  (2023-03-30T02:40:28Z)
• Astromorphic Self-Repair of Neuromorphic Hardware Systems [0.8958368012475248]
  This paper attempts to explore the self-repair role of glial cells, in particular, astrocytes. Hardware-software co-design analysis reveals that bio-morphic astrocytic regulation has the potential to self-repair hardware realistic faults.
  arXiv  Detail & Related papers  (2022-09-15T16:23:45Z)
• Brain Cortical Functional Gradients Predict Cortical Folding Patterns via Attention Mesh Convolution [51.333918985340425]
  We develop a novel attention mesh convolution model to predict cortical gyro-sulcal segmentation maps on individual brains. Experiments show that the prediction performance via our model outperforms other state-of-the-art models.
  arXiv  Detail & Related papers  (2022-05-21T14:08:53Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Overcoming the Domain Gap in Contrastive Learning of Neural Action Representations [60.47807856873544]
  A fundamental goal in neuroscience is to understand the relationship between neural activity and behavior. We generated a new multimodal dataset consisting of the spontaneous behaviors generated by fruit flies. This dataset and our new set of augmentations promise to accelerate the application of self-supervised learning methods in neuroscience.
  arXiv  Detail & Related papers  (2021-11-29T15:27:51Z)
• Increasing Liquid State Machine Performance with Edge-of-Chaos Dynamics Organized by Astrocyte-modulated Plasticity [0.0]
  Liquid state machine (LSM) tunes internal weights without backpropagation of gradients. Recent findings suggest that astrocytes, a long-neglected non-neuronal brain cell, modulate synaptic plasticity and brain dynamics. We propose the neuron-astrocyte liquid state machine (NALSM) that addresses under-performance through self-organized near-critical dynamics.
  arXiv  Detail & Related papers  (2021-10-26T23:04:40Z)
• On the Self-Repair Role of Astrocytes in STDP Enabled Unsupervised SNNs [1.0009912692042526]
  This work goes beyond the focus of current neuromorphic computing architectures on computational models for neuron and synapse. We explore the role of glial cells in fault-tolerant capacity of Spiking Neural Networks trained in an unsupervised fashion using Spike-Timing Dependent Plasticity (STDP) We characterize the degree of self-repair that can be enabled in such networks with varying degree of faults ranging from 50% - 90% and evaluate our proposal on the MNIST and Fashion-MNIST datasets.
  arXiv  Detail & Related papers  (2020-09-08T01:14:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.